ssb archive 📚

i'm playing around with building a static read-only web archive of ssb.

personally, i'm excited for #ppppp and i don't want to lose everything we did here together, the cypher librarian / ssb archaeologist in me wants to continue to link to our past into the future. so the goal is to generate static html pages of everything i care about, complete with full-text search.

initially inspired by %k5Br27/..., i got a proof-of-concept working on top of oasis, but hacky af.

i'm working a new Rust approach, starting from sunrise-choir/ssb-flume-follower-sql: ignore the ssb server entirely and just parse the append-only log directly, generate your own SQL db that can then be queried to generate pages.

now i'm starting to think that generating static read-only web pages from p2p content is actually a good idea, not just for archiving. maybe because i actually like the Jamstack: pre-render dynamic content into static html files hosted on a CDN, cache cache cache.

and for my more controversial opinion: at some point ssb-viewer decided to make "public web hosting" opt-in, rather than opt-out. given that everything posted on Scuttlebutt is "on-the-record", i'm not sure i agree with this decision, many a time i've wished i could reliably permalink to my ssb posts on the wider internet. as Scuttlebutt dies and is re-born, will this social norm stay the same or will this change?

so, how do people feel about a static read-only web archive of ssb?

and, how do people feel if an archive excluded those who opt-out of publicWebHosting? i'm happy to disable search engine indexing.

thinking about how a read-only web archive relates to "small world socials", i realize i already do this:

#art~hack is a small world social event, plus we share our streams with the big world: tube.arthack.nz.

i want to have a small world of meaningful relationships, and share stories of adventures with the big world. but i'm happy if sharing stories to the big world is more one-directional.

I recently shut down #setinhash, which is exactly what you described as a "static read-only web archive of ssb" after running for 3 years. It would be great to see a better one come up! I posted the code on github. It used Hugo and Python.

Regarding the "public web hosting" option I also asked around here: %5U70tGs...

At that time most people replied prefer respect it so I followed. However this cause lots of broken links on the website and I suppose this is one reason google didn't index it well.

There are some people who might get angry about ignoring publicWebHosting opt-in. I don't feel this way but I would want to respect peoples expectations. Having said that, no-one is the boss here. You may just want to consider how to let people opt-out practically if you go ahead with full public.

hmmm... honestly if I were you I would not make it full-public. I think putting it on the web changes the nature of this space too much. There is an expectation of limitation of reach. And making it scrape-able means companies can consume my thoughts and relationships without any accountability, which I think I do feel creeped out by

i really like that it only indexes and surfaces your own posts, which sidesteps a lot of consent concerns for a project whose output is likely to be publically available :)

thanks for the link! for me, an archive of only my own posts is kinda useless, because what matters most is context, especially when heaps of my posts are just links to other posts. my original plan was to archive my posts, and 1 or 2 hops of links from my posts. i want to be able to link back to ssb posts, as i do when i write a post on the current ssb, into the future which might not happen on ssb.

consent concerns

i think the consent question is interesting because we already share content we didn't author on ssb, including with people we don't follow (or in any way, explicitly consent to have our content). to me, the only difference is whether you are talking over muxrpc in an ssb app or over http in a web browser.

for a thought experiment along this line of thinking: imagine a browser-based ssb peer where you could link someone to a url with a message id and a pub address. your browser connects to the pub and requests the post (either out-of-order or by replicating the feed). that seems "okay", by our current set of cultural norms where we don't think much of ssb peers replicating feeds. but hosting an unindexed web page that already includes the post content is not okay?

i also understand i have privilege that leads to me feeling okay with my content being shared online. if possible, i'm keen to understand the reasons behind why ssb content, which is already public to read for anyone who is given access to an ssb pub, shouldn't be on an unindexed web page. is this a mismatch of expectations between what ssb is and what ssb wants to be? is my thinking just cold and heartless, missing the point of our squishy human relationships?

Anyone who wants to read messages here has at least a tiny bit of skin in the game. Reading these messages means you already are participating in the p2p ecosystem.

thanks @Spencer, i like that framing. 💜

ℹ️

• %RyYbXuv...
• %07HuQIb...
• %6ycC56L...

@Spencer

To reduce the immeasurable, infinite complexity of our selves and our multilayered relationships to each other to a …number. That is evil. I still live in our modern society, typing on a computer, so I’m not against numbers, etc. But is there not something core to the idea that our algorithms and judgments should be integrated into the holographic realities within us? I don’t want to flatten my experience by having ML or Silicon Valley heuristics break off pieces of my humanity by digitizing them, all in the name of efficiency or progress.

yeah i agree. technology is best to augment humans, not remove humans.

i too like being weird and messy and awkward and fun. i like to party. i like making up stories about a giraffe. i like being silly with money. i like being silly with power. i like dreaming about a solarpunk future. i like this life thing at the moment. ☀️ 🥔

i'm not a techno-optimist. i think we mostly go in circles, get lost, and rediscover wisdom.

i worry AI will never reach intelligence (before climate change has a word), but still be effective at enforcing our dangerous games (capitalism, etc).

For SSB, what code is anyone ever going to write for a network where we can’t see the entire network how fun @epk is? How do we digitize my feelings when he shares about his dad passing? I share about my dad’s declining mental state and @hoody hibernating says I might be hurting myself by over-empathizing, and I deeply feel that advice, how should we code that so it is accessible to every person on earth instantly? Is that actually good? or is it nonsensical?

for me, what matters about SSB is that something happened here. i'm not sure what exactly, but i know the effect was life-changing for many of us. sure this happened on the internet via mostly text, but there were humans behind those glyphs, and we shared a real experience together. and yes, SSB will continue to happen for as long as there are butts to replicate, but for me i see the tides changing and i'm feeling reflective of my journey so far. 💜

i don't want to optimize SSB, i just don't want to forget what happened. i have a bad memory, so reading old posts is like re-discovering stories about your ancestors, our past selves. brings me great joy to read, even more joy to share.

i also was, in many ways, raised by the internet, so i grew up stumbling through rivers of content. i still love wandering amongst the unfiltered, unprocessed, unsummarized internet. i want the fire hose, good bad and all. the internet is everything and nothing, but amongst the bytes are real happenings by real people. i was born of this byte stuff.

so as i think about how to compost SSB, i want to feed our nutrients back into the soil from where we came, be available for the next wandering mind, the next 15 year old wandering nethead like i was. i think about the Usenet Archives, where you can see the life of an early internet, random posts that mean nothing yet everything. i don't care about being accessible to everyone, as with SSB onboarding, the wanderers will find themselves. i don't worry about our content being out-of-context, what happened here is a story for each person consuming each byte to understand in their own way.

It is important to note that even without the archive, individuals, companies or AI can easily create a crawler that can access the entire SSB network without your knowledge. SSB is as public as WWW.

@Spencer I understand your concern that the culture of SSB may be negatively impacted if an influx of people is brought in by the proposed archive site. Your apprehension is indeed valid. In the recent article I shared %EMCd6pU..., an example is given:

... with the influx of a large number of new netizens, users browsing Tianya began to exhibit a “mixed” knowledge structure and worldview. In the early days of the Chinese Internet, the act of “going online” itself had certain barriers, so users browsing Tianya tended to have a certain degree of homogeneity in terms of knowledge and social strata. In this state, authors could express themselves without hesitation and with high efficiency.
...
However, as a large number of new Internet users poured in, the threshold for the audience was lowered, and the writing cost required for knowledge popularization actually increased. Moreover, many opinionated articles no longer received universal resonance like in the first few years. I remember that around 2008, Tianya and other websites began to see a frequent emergence of “controversial posts,” verbal battles between readers with different views and knowledge structures, which greatly consumed the writing enthusiasm of authors who insisted on niche expressions and were powered by love, causing them to gradually fade from the platform.

Many in SSB value the preservation of high barriers of entry, using them as a means to connect with like-minded individuals. However, compromising the entire user experience of SSB solely to serve as a simple filter, which could likely be achieved through various other methods, is wasteful and may hinder the potential benefits SSB could bring to a broader audience. If you cherish SSB as you would a child, you would want it to grow rather than merely possess it.

Humanity encompasses numerous aspects, with curiosity, exploration, and sharing being equally important.

so here's the code-y bits i've been working on: ahdinosaur/ssb-archive

maybe been a slight obsession, making a Rust-y database for SSB data. 🤪

have most of the foundations set, re-wrote a whole new set of modules for Rust-y SSB data, but other than processing your Patchwork flume log.offset into an sqlite database, and parsing markdown, doesn't really do anything yet.

makes me wonder about making some Rust-y modules for ppppp...

i also notice how many wrong-but-technically-legal-because-there-is-no-schema messages have been published on SSB. but at the same time, it's amazing that everything works, in spite of the chaos. 💜

an SQL schema for SSB data:

note: this does not replace your flume log.offset, but creates a way to search for messages, which you then retrieve from the log.