Proposal towards resolving Markdown ambiguity & supporting plurality

As discussed in %VG8tIpq... up to %tDOfQSu..., messages from latest Patchbay and Patchwork use a different Markdown variant than other SSB messages. At present, it is ambiguous which variant a recent post is written in, and this results in messages being rendered not as they were intended.

RFC 7763 and RFC 7764 discuss this as the problem of missing context for Markdown. They offer the solution that markdown data should have associated with it an identifier for which variant it uses, to provide the missing context, such as in the filename or content-type. They created a registry of some Markdown Variants and their ids. If SSB messages that use markdown include an appropriate variant id, we can resolve the ambiguity for new messages.

Latest Patchbay and Patchwork use ssb-markdown v4.0.0 which implements what is closest to CommonMark, while previous versions of ssb-markdown used ssb-marked which implements what is closest to GFM. Manyverse uses remark with plugins, with default configuration of GFM. Both CommonMark and GFM are in IANA's markdown variants registry. These SSB markdown implementations also have special handling of SSB ids, so I propose using the IANA identifiers prefixed by "SSB-" to indicate the variant has SSB extensions.

May I propose this change for Patchcore, Patchbay and Patchwork:

When publishing a message of type post, add to the content object a property "variant": "SSB-CommonMark".
When rendering messages, take the msg.value.content.variant value and pass it in as an option named variant in calls to ssb-markdown.

ssb-markdown at present would discard the variant option, but in future versions it could use it.

Would this be acceptable?

@mixmix @mmckegg @christianbundy #patchbay-dev #patchwork-dev #ssb-markdown

I think I spoke with @christianbundy ... we could add an markdown-variant to fields, but this would proliferate to all sorts of fields like stuff in gatherings.

Personally I think we could just write up the expectations / spec of our most common variant somewhere and that would be enough for the most part.

@cel

At this point I'm leaning toward scrapping my implementation and use something with GFM as a base. I'd love to specify both the media type and variant metadata, but I don't have the time or funds to support that work. I'd also love to support multiple Markdown variants, but I think that work depends on us including the above metadata for how it should be parsed.

@andré

Is your remark implementation a drop-in solution that we could use instead? I understand that you have some React-specific code, but the previous ssb-markdown is outdated and the current ssb-markdown is the wrong Markdown variant. I'm wondering whether your implementation might be the best option we have. What do you think?

As far as I'm aware, the expectation is:

• GFM
• emoji shortcodes
• block() method
  • hashtags
  • auto-linked sigil links
  • auto linked @mentions from msg.value.content.mentions
  • auto-linked images (I'd love to drop this)
  • audio/video like ![audio:description](&...)
• inline() method (for message previews)
  • newline removed
  • links removed
  • images removed

@mixmix @bundy okay, makes sense. I wonder, what would it take to bring the previous ssb-markdown up to date?

@cel I think christian looked and there have been major version bumps, I remember him saying "it was gonna be a major re-write anyway, so I looked for libraries with plugins and that's looked good"

@christianbundy awesome to see the spec of what our markdown might look like all in one place. Jeez, so succinct. I can attest those inline message preview methods sounds really useful. I've been hacking around to do that sort of thing.

@mix

Yeah, the only tricky bit is handling all of the options:

• toUrl: optional function which takes references as input and turns them into client-specific URIs
• imageLink: optional function which takes image src as input and returns a href to link-wrap with
• emoji: optional function which takes emoji shortcode as input and returns HTML for an image
• protocols: optional array of protocols to linkify, defaults to HTTP(S) + FTP + Dat
  • Dat support is broken in any URI parser because Dat identifiers aren't URIs

They're fine on their own, but often a bit of a hassle when interfacing with each other and different plugins and the various moving pieces that try to do URI encoding on our sigil link URLs. I tried to spike a quick implementation with Remark, but it ended up producing:

<a href="&#x26;nBmGtTlQew5bSiy5CwKA7Qz6Qfj6UYzmf9gYgcNg+90=.sha256">&#x26;nBmGtTlQew5bSiy5CwKA7Qz6Qfj6UYzmf9gYgcNg+90=.sha256</a></p>

🥴

Update: I've opened an issue upstream to see whether there's a simple way to make this newline change.

Another update: I've opened a pull request to resolve the newline issue.

+1 to that PR @christianbundy

@cel

Merged and published as 4.0.1. 👍

Regarding the future, it would be really nice to have a way to specify metadata about the post. For example, here's how you'd do a "post" with ActivityStreams:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "summary": "A simple note",
  "type": "Note",
  "mediaType": "text/markdown",
  "content": "## A simple note\nA simple markdown `note`"
}

This is actually a pretty neat spec, it seems like they've put a lot of effort into specifying exactly how all of this should work together: https://www.w3.org/ns/activitystreams#class-definitions

@christianbundy as long as the @context is a hash not a http url. http can easily return something different the next time, the context should be a hash.