@dinoƧ𝔸Ⓤᖇ
I seem to be running into this weird quirk on my ssb-pub:

2019-07-01T07:49:47.698406542Z ssb-friends: stream legacy api used
2019-07-01T07:49:49.042806447Z /home/node/.npm-global/lib/node_modules/ssb-server/node_modules/ssb-msgs/index.js:75
2019-07-01T07:49:49.042842233Z     var r = (typeof obj == 'string') ? obj : obj.link
2019-07-01T07:49:49.042846314Z                                                  ^
2019-07-01T07:49:49.042849070Z
2019-07-01T07:49:49.042851609Z TypeError: Cannot read property 'link' of null
2019-07-01T07:49:49.042854226Z     at /home/node/.npm-global/lib/node_modules/ssb-server/node_modules/ssb-msgs/index.js:75:50
2019-07-01T07:49:49.042859275Z     at /home/node/.npm-global/lib/node_modules/ssb-server/node_modules/ssb-msgs/index.js:50:9
2019-07-01T07:49:49.042861865Z     at Array.forEach (<anonymous>)
2019-07-01T07:49:49.042864665Z     at traverse (/home/node/.npm-global/lib/node_modules/ssb-server/node_modules/ssb-msgs/index.js:49:11)
2019-07-01T07:49:49.042867148Z     at Object.exports.indexLinks (/home/node/.npm-global/lib/node_modules/ssb-server/node_modules/ssb-msgs/index.js:72:3)
2019-07-01T07:49:49.042869622Z     at module.exports (/home/node/.npm-global/lib/node_modules/ssb-server/node_modules/ssb-links/links.js:20:8)
2019-07-01T07:49:49.042872137Z     at /home/node/.npm-global/lib/node_modules/ssb-server/node_modules/flumeview-query/links.js:37:5
2019-07-01T07:49:49.042874546Z     at reduce (/home/node/.npm-global/lib/node_modules/ssb-server/node_modules/flumeview-level/index.js:90:25)
2019-07-01T07:49:49.042877063Z     at /home/node/.npm-global/lib/node_modules/ssb-server/node_modules/pull-write/index.js:35:23
2019-07-01T07:49:49.042879506Z     at /home/node/.npm-global/lib/node_modules/ssb-server/node_modules/pull-write/index.js:41:13

The line in question that is failing here seems to be ssb-msgs/index.js:

var r = (typeof obj == 'string') ? obj : obj.link

Could it be a corrupt message of some kind that managed to sneak in and can I do something to fix it myself?

i believe this is the corrupt message mentioned in %NlMjeM4....

@SoapDog pull requested a fix here: https://github.com/ssbc/ssb-msgs/pull/11

once this is merged into ssb-msgs and then ssb-server is updated, #somebodyshould make a new version of ssb-pub to use the latest ssb-server (this requires manually testing that everything still works on a fresh server, since in my experience changelogs are undocumented and new versions may break without warning).

aha, just found this https://github.com/ssbc/ssb-msgs/pull/11
seems to be the problem

cool! you cross-posted the same findings.
I've been posting invites to my pub to friends and family that want to follow me on my Ghana-trip, but I can go dat-blog only for now for them

@Hendrik Peter I'm really happy that this might be fixing your problem. I don't believe it is the same post as sbot crashes before gossiping and until you fix it, it won't start.

Did this happened just after you posted a message? Was it from which client? I've patched it in the current patchfox version (the one that is available from the store) and the update is automatic. If this happened after using some other client then we need to investigate. It is caused by a broken mentions array.

hey @SoapDog, It's happening on my pub server, it happened after it fetched a message from someone else. I nuked my flume, but during the re-creation of links (by pulling messages from the network) the problem resurfaces.

@Hendrik Peter well, then that PR is now more useful rsrsrsrs. Can you try it on your pub?

I'm sitting in the office right now, I would need to rebuild ssb-server, ssb-pub and then the pub docker image to test this which will take some time (and I'll probably run into other issues caused by things that got updated). I'll try it in a few hours!

@Hendrik Peter please send feedback to @SoapDog when you do. I'm curious <3 and thanks a lot for testing it out.

I'm looking at how to test this without essentially shooting myself and others in the legs. If I were to patch my pub (and get it back up running) it would start sending the "broken" message forward to everyone connected. are #patchwork, #patchbay and #manyverse (among others) able to deal with the same flume issues it in their current state?

@SoapDog thoughts?

@Hendrik Peter I think we need people with more knowledge than myself to correctly answer this. The first thing I'd do it nuke flume. I know it is a pub server and it is supposed to keep and spread messages but if there is a broken message inside and we don't yet have a fix that is widespread, I'd rather nuke the database.

@SoapDog @SoapDog (Manyverse) @andrestaltz @mixmix @dinoƧ𝔸Ⓤᖇ what do you folks think about this?

If you transmit the message and it breaks erryones stuff ... then we'll be forced to upgrade.
That there's a message which can break setups means our code was sloppy. It need to be fixed RIP OFF THE PLASTER (future mix will probably curse me)

But I'm serious, do whatever works. You've communicated some stuff clearly, you aren't responsible for fixing everyones everything @Hendrik Peter, you're doing a great job <3

Okidoki. yeah sure then I'll test around in a bit.
I guess nothing will be official or rollable to other pubs until pull 11 does its merge.

Other related question for you @SoapDog (since you're the author of ssb-msgs) and @mixmix (since you're quite knowledgeable on the subject).

Is there some kind of "spec" of how messages should look? something that checks the content of a messages before it goes further into the ssb-msgs system (and ssb for that sake)?

I could see an abuse case here where you could essentially send a malformed object-type in a (private) message to someone (no-one on the way would be able to see that the message was malformed in case of private) and then have targeted peoples ssb-clients go belly up through runtime errors, "banning" them from using their favorite ssb-client.

The fact that a malformed mentions object popped up might be a nice pointer into the direction that we might wanna have specifications of how different message-types should look or take a look at them.

@Hendrik Peter I'm not the author of ssb-msgs, I just sent a PR to it because I was having the same problem and could narrow it to those lines. I usually don't dive that deep into our stack rsrsrsrs.

I think I've seen a JSON Schema validation thingy/module for messages in the past but I can't remember the name now. What I do right now is not the best way. I basically use Patchfox and go to a message that is the same type as the one I want to implement, then I use the show raw message menu item and check it out. I do this a couple times to understand how the props work and then proceed to mimic them. Not the best way but I'd rather copy message structures that work than be hurt by deprecated modules or outdated documentation.

On the topic of documentation, the people in #village-tracker are doing an amazing job and most of the current advances in Patchfox can be attributed to the recent improvements in the docs.

@Hendrik Peter we've been there in the past ! It happens occassionally. Repeating what I've said, I'm a big believer in it being the responsibility of the listener to listen carefully.

essentially JSON-Schemas : https://github.com/ssbc?q=schema

I personally think ssb-msgs needs deprecating. I don't think it's much used, and if it is then I think it needs to be split out into different domains, rather than the all-encompassing "msgs"

So the entire error seems to be coming from a test post by @SoapDog's surface go.

I've posted my full details here:
https://github.com/ssbc/ssb-msgs/pull/11/files#r299367419

So somehow one of the mentions is actually just a null object.
Since this is a pub server, what is the best way of reseting my database and how can i make sure this broken message doesn't end up back in my DB?

@Hendrik Peter

I could swear it didn't gossip! auhahuah well, now we kinda need the patch to be merged.

A way is to block my profile, that would solve for now. Do you have the key for that message? I can't find it in any of my identities.

I think that @Christian Bundy has a tool/script that deletes a feed from a database. You could use it to delete mine. I can't find that message here so if you delete the feed maybe it won't come back.

I'm trying to find that script now. got any pointer on where I can find it?

@Hendrik Peter I think there is an early version of the script here: %6wBVeuu...

It has a hardcoded feed id, you'll need to replace it with mine. I'd advise you to delete all my identities.

IIRC @Christian Bundy made a more flexible version afterwards that you could pass an id to it but this version should work I think.

I just found this one by searching around in patchwork:
https://github.com/fraction/flumedb-delete/#usage

Should do the trick. I guess the feedId is your user id

@Hendrik Peter yes! Delete @SoapDog, @SoapDog (Macbook) and @SoapDog (Manyverse) just to be sure.

@Hendrik Peter thanks for taking the time to sort this out with me.

Hendrik Peter in a way, it is funny that my tiny 10'' machine can destroy the SSB stack across the world by shipping a broken mentions array. There are probably more cases of broken messages like this that can wreck things up.

We're making a network for the long haul, we kinda need to make it more resilient to broken messages and easier to prune them.

sure thing. only problem is, that this message is now out in the wild and could randomly kill any client that comes in contact with it. But i'll remove it on my side for now

or as I said earlier, to make sure that there is a standard and testing a message that comes in if it's supported by the running instance of ssb. if not skip. I think this message object shouldn't be stopped from being saved in the database (as I could just have an outdated version of ssb), but it should be filtered out of any front-end as early in the stack as possible.

There is a chance that message is not spreading as it breaks clients very fast. Still, we need to make the ecosystem more reslient.

I'm thinking about making a large post calling for help on it. The patch is there, we need review and testing by the client authors though.

@SoapDog (Macbook) @Hendrik Peter

Yep!

https://github.com/ssbc/ssb-db/blob/master/api.md#del-async