Ugh, I forgot to mention another relevant counter argument, I really should stop writing these things in one go just before going to sleep...

offchain-content adds another failure mode: it is possible for the server to be aware that there is some message at a certain point in the chain, but to not have its content. That means a breaking change to the ssb-client API, the content entry would suddenly allowed to be undefined. Clients can currently rely on it always being an object (with at the very least a type key). This is a big one, forcing all application developers to adapt.

This also mean that replication rpcs need to be able to account for this situation, but that's just an implementation detail to keep in mind, not really an issue.

@moid

Messages aren't that large on average, for the most part folks aren't writing lengthy multi-part posts.

In the larger picture, ssb is an arbitrary database, not just a social network. I can imagine lots of use-cases where all of negligable size limit, deduplication, and indexability are extremely helpful, e.g. storing the dependency graph of a decentralized package manager, or messages that represent directories in a file system.

Constantly having to worry whether your messages hit an arbitrary limit can be a showstopper for certain kinds of applications, especially those that interact more with machines then with humans (who are able to split up their output on their own). There'd probably come a point where different applications develop different, inefficient and ad-hoc ways to get around the message limit.

but I can't see lots of regular messages that would be duplicated.

Again, that's just if you consider social networks.

[...] you'll pay a lot for it in terms of greatly increasing the number of calls to fetch content.

As I stressed in the original post, this would not be implemented via blobs, but rather as specialized rpcs. There would be no additional roundtrip cost whatsoever.

@cel

I think we could implement deleting messages with current messages/feeds

That is an interesting proposal, but it does rely on fully trusting your replication peers, something that both Dominic and I are reluctant to do.

About hashing message content without metadata: wouldn't it mean a message id would not point to a unique message?

Nope, I guess I was unclear on that one. The offchain-content would the compute the hash of a message as hash(concat(all_the_currently_hashed_stuff_except_the_content_field, hash(content))).

@frankiebee

Why not just write a client that implements many different protocalls rather then change ssb to fit a large use case.

I honestly didn't see this proposal as expanding the scope of ssb. It doesn't add new functionality to its API, it just improves upon things that are already there. And I won't stop trying to improve ssb details just because they are already kinda working.

in any distributed system selctive deletion sounds like a lie.

I am well aware of that. But this is not about deleting data on other machines, it is about deleting data from your own machine, while still being able to verify messages from the author.

@Dominic

there are some indexes where the order of messages is significant

True, but the order this depends on is the order of the sigchain, not some arbitrary replication order. If the current implementation relies on those to match, than this will mean additional work for sbot, but it isn't a conceptual problem in general.

There are lots of other complications this would introduce, that would need to be discussed,

Ack, this post was only an introduction to the concept itself, not about actual implementation and roll-out.

If you posted a response but I didn't @mention you back, then you are outside of my replication range and I didn't get the response.