Forking another thread:

I've been reading a lot of posts on the issue of JSON validation and it seems intuitive to me that contant as blobs (see this thread) would resolve most of the issue as we'd only have to deal with top-level props rather than arbitrarily nested JSON in the content property. Anyone have thoughts on whether my intuition is correct, or whether these are unrelated problems?

I'm going to try and track/wrangle this idea as #blob-content.

cc: @dominic @Ajoscha @cel

Thanks for the fork suggestion @mix.

@christianbundy Message content as blobs is in the pipeline, can't do everything at once =)

The idea is one that is pops up regularly btw, the earliest mention I found was in an ssb github issue from the early design stages.

This is fairly orthogonal to the json problems. We plan on completely moving away from json anyways, but there are other good reasons for only signing a hash of the message content (that is what it all boils down to). I planned a whole post on this topic, so I hope it is ok if I postpone a more throughout response until the issue comes up when the discussion about general metadata design choices has moved to that point.

Oh, and a quick warning: Don't get too attached to the idea that content would use the regular blob mechanism. It might end up that way, or it might not. In particular it would be inefficient to do additional round-trips for fetching content-blobs after receiving messages, instead it might be better to have specialized rpcs that send the content directly, by simply concatenating it on the metadata. Or something like that, I didn't spend a lot of time yet on concrete designs.

Specific desgns are also tied to message size limit considerations (we could increase the content limit, but we also need to keep in mind that fetching larger messages will slow down transmission of metadata), as well as back-pressure for parallel data streams.

@Aljoscha

Thanks for those links, those are great resources. I've seen it pop up a few times, and I'll look forward to reading your post on the topic. No rush on that, of course, I hope my excitement for working on blob content didn't come off as pushy.

I'm privy to the basics of blobs but that's about it, so no attachment at all. I've often wondered whether managing blobs with IPFS/Dat might be an option, but again, I'll look forward to hearing your post on this if it's something you're already working on. Thanks for the quick response!