%U13iMbN...

👆 IMHO the recommended way to block. As soon as I merge that PR for patchwork, people will be able to see why you blocked someone.

@Denis M Goddard yes. If you also set the port config as mentioned, then it can run at the same time as your main instance. If you also set the caps keys as mentioned, it exists in a separate/isolated network. (But note, as @mixmix mentioned in %gOTdlhx..., the sign key should not be changed mid-feed.)

For consistency I would suggest only use ssb_appname=ssb-test for testing/development purposes. Other identities/ssb-servers could use other appnames, e.g. ssb-foo.

There's also ssb-identities for the Node.js stack. That is meant to allow having/using multiple ssb identities in one ssb-server (without needing multiple copies of the database). I don't think I have seen any SSB clients using it though - except Feedless which uses a forked version.

Using a separate ssb-server for testing

@RavteCyber

Export environmental variable ssb_appname=ssb-test. That makes ssb-server and ssb-client use ~/.ssb-test instead of ~/.ssb, including automatically making a new secret file in ~/.ssb-test if needed. Then don't follow anyone or use invites from the main network from the test id, or vice versa.
To run it alongside normal ssb, make it use different ports. Put in ~/.ssb-test/config:

{
  "port": 8007,
  "ws": {
    "port": 8988
  }
}

For good measure, set sign and shs caps, as mentioned by @aig in %y00L8pC... and by me in thread %tQakYap..., also referencing @josiah's guide %tWGy2kn.... e.g.:

{
  "port": 8007,
  "ws": {
    "port": 8988
  },
  "caps": {
    "shs": "o9dkRr3BsTYxRJ1dLDXlk0FjPZ0LsP6Tti8PlYUEMU0=",
    "sign": "K84etuJU/BQ5R8eYb0U0I4pUVZfyEcLVlIiKxDg1RAE="
  }
}

I'm not using ssb-server directyly, rather secret-stack + ssb-db and other plugins. Have not seen this bug yet, wonder where it exists.
Does anyone have an idea for a test we could write to stimulate this bug?

@Denis M Goddard
Glad it's working.
It shouldn't be necessary to have host set in both the config and the CLI option. But if it is, that is good to know.

@Denis M Goddard
I don't think the ssb-server/sbot CLI can connect using the unix socket currently. (sbotc would.) It is probably trying to connect to your public IP. What you could do is change the listener host in the ssb config from 127.0.0.1 to 0.0.0.0 or ::. But then it will be listening publicly and I'm not sure if you want that. In that case, you may be able to set "host": "127.0.0.1" in the sbot config (in the top-level config object, not in connections) - or as a CLI option, like ssb-server --host 127.0.0.1 whoami - to make it connect to 127.0.0.1. That host option has to match the host you have set in incoming.net[0] or you will get an error about conflicting connection settings.

Hi @Denis M Goddard,

The hidden service is connecting to 127.0.0.1:8008; maybe the ssb-server is not listening on that. You could try ss -ltnp|grep 8008 or netstat -ltnp|grep 8008 to see what it is listening on. You can change the listen host by adding a "host": "..." property in the connections object alongside the "port" property.
e.g. "net": [{ "port": 8008, "host": "::", "scope": "public", "transform": "shs" }]
Or maybe you could change the torrc line, e.g. to [::1]:8080 to make it connect using IPv6

Added more context to the issue.

Stuck indexing and not responding to incoming connections

ssb-server as of v16 can get into an unresponsive state. In this state, it rejects incoming connections and does not index the flumelog.

When a secret-handshake client tries to connect to the server - even a local one - the server rejects the connection a few seconds after sending the first response packet. Example transcript from trying to connect to the server:

$ strace sbotc -4 whoami
[...]
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
getrandom("\316\f%\243\333\220\214\203e\357\312;P\270\211\30", 16, 0) = 16
getrandom("\320\366\27\177:\234]{:\222\245\363Ow\0232\302\315\276\240C\235\344\220\10\365\tj\373\raT", 32, 0) = 32
write(3, "$\213\202\nVI\222wb\354Z\307\\\3*~\257R\376F\23\3710\377\0.J\25\366\322\1\216"..., 64) = 64
read(3, "\205\364n\264\322\17;\342\22\v^\271\310\271@\215\253=\345@\334o\233\320\345}\307#\341OaH"..., 64) = 64
write(3, "\212N\261/\v\301\2061_\36\367\364P\265\314\313\36\201\25\21\341\301\250=;\2p\214m\303\201\20"..., 112) = 112
read(3, [A few seconds passes here...] "", 80)                         = 0
write(2, "sbotc: ", 7sbotc: )                  = 7
write(2, "hello not accepted", 18hello not accepted)      = 18
write(2, ": ", 2: )                       = 2
write(2, "Broken pipe\n", 12Broken pipe
)           = 12

If the server is configured to use the unix noauth socket, sbotc can connect using that. (The -4 option used above forces it to use TCP and thus SHS instead of the noauth socket). Connecting with sbotc using the noauth socket reveals that ssb-server is not indexing its flumelog, as this command shows the progress stuck:

$ while sleep 1; do sbotc progress | jq -c .indexes.target-.indexes.current; done
366939
366939
366939
366939
366939
^C

It looks to me that ssb-server v16 gets into this state when it starts while the flumelog is not caught up. In previous versions, doing this would not be problem. ssb-server should allow appending data to the flumelog while it is offline and then indexing and catching up when it starts. Running ssb-server v15 causes it to catch up indexing, and then switching back to ssb-server v16 works again.

Opened https://github.com/ssbc/ssb-server/issues/734

I noticed that the server I get when starting Patchwork behaves differently than the one I get when I execute ssb-server start. I assume this is because the latter doesn't loads all the modules I need. How do I load modules when starting ssb-server on the command line?

Advisory: ssb-server crash fixed

A bug in ssb-links causes ssb-server to crash when handling certain messages recently published.

The main error message on crash is: Error: can only encode arrays.

This bug is fixed in ssb-links v3.0.10. If you encounter this crash, please reinstall/upgrade your ssb-server to get the updated ssb-links module.

It is possible a similar issue could exist with other #ssb plugins using flumeview-links, in which case we will update this thread.

This issue primarily affects pub operators and users who run ssb-server directly. Users of apps that bundle ssb-server (e.g. Patchwork) are not affected.

There are no known security implications.

Context: %X7e37FY...

@farewellutopia
You're welcome.

If that error is accompanied with a few seconds delay on the client side before the connection fails, it could be that the server is busy/overloaded and doesn't respond to the connection request in time. (Or the client could be busy/overloaded, but that is less common).

Thanks @cel-desktop for this very helpful example! It helps me learn about the sbot ass well as the pull-stream API.

The client often fails to connect to the server (Error: could not connect to sbot), on the server I see this log:

server error, from net:127.0.0.1:47778~shs:
Error: stream ended with:0 but wanted:64
    at drain (/usr/local/lib/node_modules/ssb-server/node_modules/pull-reader/index.js:43:26)
    at /usr/local/lib/node_modules/ssb-server/node_modules/pull-reader/index.js:63:18
    at /usr/local/lib/node_modules/ssb-server/node_modules/pull-reader/index.js:20:7
    at drain (/usr/local/lib/node_modules/ssb-server/node_modules/stream-to-pull-stream/index.js:126:18)
    at Socket.<anonymous> (/usr/local/lib/node_modules/ssb-server/node_modules/stream-to-pull-stream/index.js:147:5)
    at Socket.emit (events.js:228:7)
    at TCP.<anonymous> (net.js:664:12)

I understand that I'm connecting unnecessarily often and that I could instead just connect once and keep a reference to the server. I'm still curious about what exactly is causing this error.

@farewellutopia
Here is an example function to get the latest vote by a user for a target id.
This will use the most efficient RPC API if available, and fall back to others. (TODO: support go-sbot [%7QnBSs0...]) For the sake of simplicity, it uses pull.filter/take/map in all cases, even though that could be omitted or simplified if using the ssb-query or ssb-backlinks method. Public domain.

// Get latest vote by feed for target.
// Calls back:
//   > 0 (usually 1): upvote
//   0: unvote (retracted/reset vote)
//   null: no vote
//   < 0 (usually -1): downvote (rare)
//   other: invalid/unknown vote
// Note: vote expression is omitted
function getVote(sbot, feed, target, cb) {
  pull(
    // most efficient: use ssb-backlinks, returning one message
    sbot.backlinks ? sbot.backlinks.read({
      reverse: true,
      limit: 1,
      query: [{$filter: {
        dest: target,
        value: {
          author: feed,
          content: {
            type: 'vote',
            vote: {
              link: target
            }
          }
        }
      }}]
    }) :
    // sbot.links: returns all the user's votes for the target.
    // usually the user has few votes, so this is okay.
    sbot.links ? sbot.links({
      source: feed,
      dest: target,
      rel: 'vote',
      values: true,
      reverse: true,
    }) :
    // ssb-query: scan through user's vote messages
    sbot.query ? sbot.query.read({
      reverse: true,
      limit: 1,
      query: [{$filter: {
        value: {
          author: feed,
          content: {
            type: 'vote',
            vote: {
              link: target
            }
          }
        }
      }}]
    }) :
    // scan through user's feed
    sbot.createUserStream ? sbot.createUserStream({
      id: feed,
      reverse: true
    }) : pull.error('Unable to get votes'),
    // verify vote message schema
    // needed for use with sbot.links and createUserStream above
    pull.filter(function (msg) {
      var c = msg && msg.value && msg.value.content
      return c && c.type === 'vote'
        && c.vote && c.vote.link === target
    }),
    pull.take(1),
    pull.map(function (msg) {
      var c = msg && msg.value && msg.value.content
      return c && c.vote && c.vote.value
    }),
    pull.collect(function (err, msgs) {
      if (err) return cb(err)
      if (!msgs.length) return cb(null, null)
      cb(null, msgs[0])
    })
  )
}

Thanks, @cel-desktop . I didn't realize that createUserStream needs the user id as well. I'm trying to get the last vote the current user submitted for a particular message. As the whoami+caching approach seems to be the more generic, I'll use that.

@farewellutopia
What query are you trying to make? To get messages by the current user, with either createUserStream or server.query.read you will have to specify the user id. You can make a whoami call at the start of your application and keep the id in memory. Or you can use the sbot.id value which is the id of the ssb-client's key (usually the same as the server's id but may be different if you are using a different master key to connect, or possibly connecting via noauth socket.

I was wondering if there is a method combining createUserStream and server.query.read? I want to retrieve particular messages by the current user and it feels somehow awkward to first request the FeedID with whoami just to send it back immediately as part of the query.

Is #ssb-server the right channel to ask API questions?

Unfortunately, the problem re-occured. Calls to the db like sbot.get block. Reinstalled v15 and it works again.

@Christian Bundy
From bin.js. But I just re-installed again now and it works fine. bin.js requires index.js ('./') so maybe that PR fixed it. Thanks!

Actually I forgot about this issue %lvLBrXS... - you might need to add an additional "dummy" argument to the plugins.install command. Like this:

sbot plugins.install ssb-private --for-real

@ben@grim @Attilastroport

After upgrading my pub to ssb-server v16.0.1, it no longer responds to incoming peer connections. The secret-handshake times out after 5 seconds. example.txt. That is with no existing peer connections, no indexing taking place, no high CPU usage, no plugins, and either the old config or no config. Starting up a ssb-server on a separate ssb_appname worked (it would respond to clients). So I tried to import the old flumelog into a new ssb-server directory. But ssb-server would not pick up the new flumelog data. I started making a script using flumecat and sbotc to read the old flumelog and add the messages one by one. That was working, except it got invalid signature on a message (probably some encoding issue) and I did not investigate further, since it would take a very long time to import messages this way, and I want to keep the old received timestamps also. So I reverted back to ssb-server v15.2.0 and that is working well.

Using ssb-server v16.0.1 on my local system and one other sbot is working. (I have not checked whether it picks up data added to log.offset while offline)

It's a bug, introduced in commit 275666a468097d004f7cabaf29d2167150a92bd6

@masukomi @diplowom I think for anyone. One of my friends assigned a photo of a large green angry person to @wmnteCW... and that's the photo I (and I suspect anyone within hop-distance of my friend) will see.

@dangerousbeans

See: https://github.com/ssbc/ssb-server/issues/634

Cannot read property 'apply' of undefined

How can I best fix this?

ssb@secretislands:~$ ssb-server friends
Cannot read property 'apply' of undefined
ssb@secretislands:~$ ssb-server friends.following 
Command not found: friends.following
Cannot read property 'apply' of undefined
ssb@secretislands:~$ ssb-server following
Command not found: following
Cannot read property 'apply' of undefined
ssb@secretislands:~$ ssb-server 
Cannot read property 'apply' of undefined

If you have more than one pub, you could let them befriend each other. Otherwise I'd say no.

I mean on wherever you are trying to set a description.

$ jq .connections < .ssb/config

{
  "incoming": {
    "net": [
      {
        "external": [
          "ssb.bornmesh.dk"
        ],
        "host": "",
        "port": 8008,
        "scope": "public",
        "transform": "shs"
      }
    ],
    "unix": [
      {
        "scope": "local",
        "server": true,
        "transform": "noauth"
      }
    ]
  },
  "outgoing": {
    "net": [
      {
        "transform": "shs"
      }
    ]
  }
}

PS: Do you have a connections item in your [optional] ~/.ssb/config? Perhaps that's the missing part?

/cc #help

You could downgrade?

$ sbot version
"13.0.3"

What's not to love about nodejs?

Well, the first one made nodejs go in its pants

Can you be more specific? I use that syntax all the time (except I use sbotc, not sbot, because the former is much faster).

the second went through, but it isn't reflecting.

Perhaps somebody else knows whether that syntax is still supported.

Come to think about it, I believe you can also do:

$ sbot publish --about "$PUBKEY" --description "$DESCRIPTION" --type about

$ sbot publish '{"about": "@pubkey=.ed25519", "description": "Ich bin ein Bornholmer", "type": "about"}'

@jfr thanks a lot, I will write that version number down and in case this one doesn't work for me, I will downgrade to the one you mention.

@ChristianBundy, just upgraded to that version and am trying it out. So far it is working even though I get a ton of deprecation errors when I start it:

PS C:\Users\andre\scoop\apps\alacritty\current> ssb-server start
ssb-server 14.1.6 C:\Users\andre\.ssb logging.level:notice
my key ID: <just my public key>
WARNING-DEPRECATION: ssb-links not installed as a plugin. If you are using git-ssb, ssb-npm or patc
hfoo please consider installing it
ssb-friends: stream legacy api used
[npm-registry] Listening on http://127.0.0.1:8043. (indexes:0%)
Trace: deprecated api used: ssb-ref.parseAddress.. (indexes:57.0%)
    at Object.parseAddress (C:\Users\andre\scoop\persist\nodejs-lts\bin\node_modules\ssb-server\nod
e_modules\ssb-ref\index.js:96:15)
    at Socket.<anonymous> (C:\Users\andre\scoop\persist\nodejs-lts\bin\node_modules\ssb-server\plug
ins\local.js:55:22)
    at Socket.emit (events.js:189:13)
    at Socket.<anonymous> (C:\Users\andre\scoop\persist\nodejs-lts\bin\node_modules\ssb-server\node
_modules\broadcast-stream\index.js:51:12)
    at Socket.emit (events.js:189:13)
    at UDP.onMessage [as onmessage] (dgram.js:628:8)
************************************************** (ebt:99.8%)

The instabilities I was facing were more of a feeling of strangeness where my ssb-server would spend too long without peering or I would spend too long without seeing new messages. Sometimes it would do hard crashes when viewing some message, or trying to retrieve something. It was a bid odd. It still worked, I just felt I couldn't trust it very much.

Thanks very much for pointing me to that version. I also look forward to trying the rust and go stuff too.

hey @SoapDog @regular and I are running ~11.3 because we ran into some issues, but I haven't checked back in a while. I wish there was an LTS though!

@SoapDog

If you can, I'd upgrade to 14.1.6. What sort of instability are you facing? I know that @ev and @regular (and maybe @jfr?) run a version from ~6 months ago, but I wouldn't personally recommend it.

Friends, I am running ssb-server 14.0.4 and I think I may be facing instability. Is this a good version to run? I've heard that people are using earlier versions, what are you fine people using?

@Rajeshkannan,

Theoretically all you need is ~/.ssb/secret and ~/.ssb/gossip.json in an otherwise clean ~/.ssb/ directory, then sbot/ssb-server will pull in everything else from the network.

@dominic

Sure!

• aligned-block-file: add write() that writes to a specific position rather than appending
• flumelog-offset: add del() that overwrites an item with null bytes, filter null items from get() and stream()
• flumeview-level: maybe bugfix, also implement del() so it doesn't have to rebuild (later)
• flumeview-reduce: fix bug where view doesn't start rebuild until server restart (todo)
• flumedb: add del() that triggers flumelog.del() and flumeview.del() if available, falling back to rebuild if it's unavailable
• ssb-db: fix a bug in create.js, export flumedb.del() and flumedb.stream()
• ssb-server: depend on new version of ssb-db (trivial)

@christianbundy can you give a brief description of what changes are needed in each of those modules?

I had a brief conversation with @dominic where he walked me through how he'd go about implementing deletion. My implementation was mostly on the right track, but the gist is that I was messing around with this sort of nonsense instead of just filtering out deleted items before returning.

As far as I can tell it's working, but it requires changes to:

• aligned-block-file
• flumelog-offset
• flumeview-level
• flumeview-reduce (TODO)
• flumedb
• ssb-db
• ssb-server

I'm thinking that the best way to go about this is probably starting with the leafs of the dependency tree, bubbling my way up from there:

Most of this work took a few iterations so I'm sure I'll need some cleanup, but I think we're close. Once this is done then we can teach flumeviews to delete single items so that we can avoid resource-intensive rebuilds, but that's still on the horizon.

I was reading this more along the lines of "webtorrent might become a second channel to spread blobs."
similar to the way manyverse can use wifi, internet, and bluetooth for gossip.

From the scuttlebutt FAQ:

Replication of larger files could be implemented by integrating bittorrent (or even better, webtorrent).

Is anyone working on that? All I know is there's #blobs

webtorrent was used for ferment but made problems IIRC. how much integration are you looking for? adding info-hashes to content is quite easy, as are DAT or IPFS links.

do you think that would remain backward-compatible with the current flumelog, or would that require a migration between the old format and the new format?

it would definitely require a migration.

return a message without validating that it also has the same length appended after the message ends.

this is a bug which can be easily fixed.

@keks

You could hack flumelog-offset to add a field of bits per message (e.g. 1-8 bytes) and use one of these bits to say "this message was deleted".

Do you think that would remain backward-compatible with the current flumelog, or would that require a migration between the old format and the new format? I was a bit surprised by some behavior that I found which I think is the current implementation, where if you request an incorrect offset (e.g. 42, if the valid ones are 30 and 48) then it will return a message without validating that it also has the same length appended after the message ends.

I'm having the same feelings about the fact that a list of operations rather than data might be useful. It's true that Scuttlebutt has that semantic, where each message is a type, but it might be really nice for flumedb items to have this capability. I'm still bummed that we don't have rebuilds on all flumeviews, but I suppose that will come with time.

Just one more thought: another problem we're going to have is that you can have views loaded in one client but not another, so while appends are added whenever the view is loaded that isn't true for the rest of the API.

For example, if you start Patchbay and do a flumedb.append() followed by flumedb.rebuild() then all of the active views will append and then self-destruct. If you switch to Patchwork and it has different views, then they'll see the append but not the rebuild, because from the perspective of flumedb that rebuild has already been completed.

I suppose my concerns are that:

1. We're attempting full rebuilds on level views that support del() natively.
2. Some/many views don't support rebuilds in the first place.
3. Rebuilds are not respected by inactive views.

Replacement

As discussed above, replacing the item with a fake message should work, although it may be creating errors. The biggest problem right now is that not all flumeviews support rebuild(), so we don't have a way to force rebuilds without restarting ssb-server. I've pasted some of the view errors I'm seeing, so maybe we should be outputing warnings when a flumeview only has partial API support?

Merge delete

Assuming we have JSON items A, B, and C, one way to delete B would be to add a bunch of whitespace at the end of A so that it overwrites B. I was having problems where my writes weren't being respected, maybe due to the multiple layers of caching, but this seems possible.

Unfortunately it's hacky as hell and only works if you can pad the previous item with whitespace, plus you have to do the opposite with the first item since there isn't a previous item to merge into. Another downside is that if you request B then it seems to just output a corrupted string -- again, I imagine this is a cache thing but I wasn't able to circumvent it while working on the problem yesterday.

Skip delete

When we get/stream an item we're using a getMeta() function that works with pull-cursor to iterate through the file. This function returns the value of the requested item plus the sequence numbers of the items before and after the current item. One hacky and less-performant idea was to read the before/after values, and skip them if they're null.

For example, assuming A - F are arbitrary messages, we delete C and then request D. The function should return the current value of D, notice that C is an empty buffer, and return B and E as the before/after sequences. This is fine, but doesn't work at the beginning or end of the file. It also means that we need to read at least 3x as much data on each read, plus buffer comparisons, which I'd imagine is prohibitively slow (unless we add another layer of caching).

Misc

Again, I'm super impressed by the speed of flumelog-offset, and it's way cool that it can write at 13.8 MB /s, but I've been watching the file size and I've never seen it increase faster than 800 KiB/s because the flumelog isn't the bottleneck -- the network and indexes are.

The offset db is incredibly fast and efficient, but it feels like we've traded away mutability for some speed gains that we don't actually use. I'm sure it's possible to add mutability to flumedb, but when working on it I feel like I'm really cutting across the grain. This might just be another incantation of boring Scuttlebutt, but since we're nowhere close to the theoretical speed of flumelog-offset then I'm wondering whether it'd make sense to use something more... boring.

cc: @cel

@christianbundy good point that the message length cannot be increased. So adding "deleted": true at the flumelog layer would not work. "content": {} would not be valid in a feed according to ssb-feed because it needs a string type of length ≥ 3, ≤ 52 - but if it was box-encrypted it could in fact be shorter, e.g. "0.box", which would not leave room to replace with an object with a type property. How about use that, or even a shorter string, like "content": "X" to mark a deleted message? Client code should treat it as a private message because the content is a string, and fail to decrypt just as if it was encrypted to some other recipients. Or, I think it would be okay to borrow bytes from the signature if need be. Zero'ing the author field (or any of the other metadata fields) I think would be fine; we will see if it breaks anything.

Another situation would be deleting messages from ssb-ooo. ssb-ooo also uses a flumelog to store messages, but it does not validate them, except by hash. So I think it would be possible ssb-ooo could store a message value that is just {} - although I don't think this would happen now unless there was a broken or malicious implementation. I think it could be fixed in ssb-ooo by at just making it ensure that e.g. a message has a content property, which is not equal to the value used to mark deletion, but is at least as long as it in bytes.

That's odd about those stream errors. It reminds me, when replicating feeds, ssb-server should probably abort sending a feed if it encounters a deleted message in the feed - since the peer will not be able to validate it or any of the subsequent messages and will therefore discard them. This could be done for legacy gossip by hooking the createHistoryStream function, like how it is done for blocks in ssb-friends (formerly in scuttlebot/plugins/block.js) ("break off this feed if they suddenly block the recipient"). For ssb-ebt, I think hooking sbot.getAtSequence would be the way to do it. I don't think getting invalid messages would cause a peer to break the stream though, so those errors might be from something else. Maybe it could be local errors in replication causing the stream to be aborted locally. Do local RPC methods still work when they would output a deleted message or do they end with an error? like sbot.getAtSequence([feedId, seq]) for a feed id + seqnum of a deleted message, or createHistoryStream({feed: feedId, seq: seq-1, limit: 3}) (to output a deleted message with its previous and next messages)

@cel

Yep, although the message must be the lesser than or equal in length to the message you're deleting, so if the message is "content": {} then we can't add a type unless we remove characters from the signature or something. Also we don't want to depend on that type because others could add similar messages, so we'd need another lookup mechanism (e.g. ensure msg.value.author === "@00000000000000000000000000000000000000000000.ed25519") anyway.

The errors are all "Error replicating with @${X}: Error: parent stream is closing" or "Error: stream ended with:${Y} but wanted:${Z}". It's possible that they're just your run-of-the-mill network errors on dropped connections and such, but they seemed to get worse after I've done deletion.

I'll post an update once I'm at a good stopping place with the new code and be sure to cc you for feedback. Thank you!

@christianbundy what about set the content to an object of a specific type, like {"type":"deleted"} and add a property to the msg object like "deleted":true. Then it should not choke up client code, but would still be distinguishable as a deleted message. What do you mean by peering errors?

@pub_cel

The problem with 0, null, {}, etc., is that all of our code expects a well-formed message, and tends to throw errors when you don't have one. We can use a fake or modified message, which is what I'm doing now, but I've seen an abundance of peering errors that may suggest that the gossip code is viewing it anyway.

@moid

The way you've done it in filter_log() works, and that's how I'm currently doing feed deletion. The problem is that it's incredibly slow, especially on mobile devices, and so the next step is to do deletion without rebuilding the log or indexes when possible. Anything built on leveldb can do quick and easy deletes with db.del(key), so as long as our log supports deletion then it's simple to say "if deletion is supported, delete, otherwise rebuild".

I like flumelog-offset and suggest delete messages by e.g. replacing their value or content in the log with null. -cel

@moid

Thanks! I've deleted my log many times today and am currently in the process of downloading messages, so I'll check out that link once it's available. Right now it's just throwing "not found".

The compaction method is how the current flumedb deletion prototype works, streaming from one log to the other while filtering out the deleted content. The updated code makes deletion a first-class citizen in flumelog-offset and views that support it (only my experimental one), but it relies on replacing the deleted content with a fake message, which causes problems down the line.

Really looking forward to checking out your link, I appreciate the new information.

Also, I found this PR where I think leveldb was replaced with flumedb + flumelog-offset, but I can't find any context for the change. If there are any threads I should read or history I should know, I'd love a pointer!

cc: @mikey 💁

Deleting feeds and comparing flumelogs

In the past few months I've been increasingly interested in feed deletion, which gives users the ability to delete content from folks that they've blocked. Being unable to delete feeds from your computer is disempowering, and means that bad people can post bad things that have to stay on your computer forever.

That's bad.

I've been working on feed deletion with #verse and I've been making some great progress adding deletion to the following JavaScript modules:

• aligned-block-file
• flumelog-offset
• flumedb
• ssb-db
• ssb-server

The gist is pretty simple: flumelog-offsets stores all messages in one big file, and deletion works by replacing the target message with something else. My first approach was to replace it with a bunch of 0s, but ssb-db always expects a message and tends to blow up when you hand it empty buffers.

Currently my approach is to add a fake message, which is fine, but I'm feeling unsatisfied by the amount of hackiness required to delete items from the log.

Over time I've been wondering: why flumelog-offset? It's impressively fast, but being double the speed of leveldb doesn't matter unless that's the bottleneck. The benchmarks show flumelog-offset writing at 13 MB/s, but in reality I've never seen it write faster than ~800 KiB/s because the bottleneck
is our peering bandwidth, not the file write speed.

If that's the case, does it matter that flumelog-level only writes at 8 MB/s? I'm not completely sold on leveldb, but it seems to have a lot of positives:

1. Already-implemented read, write, update, and deletion.
2. Streaming as a first-class citizen.
3. Loads of compatibility, from a Raspberry Pi to a browser.
4. Heaps of documentation, support, and smart folks working on it.
5. We're already using it for many/most of our flumeviews.

Don't get me wrong, I think flumelog-offset is incredibly impressive and unexpectedly fast as an append-only database, but I'm wondering which trade-offs we're making when we optimize for appends and ignore mutability.

What do you all think? I'd love some feedback on your experiences with flumelog-offset, leveldb, and thoughts on what we should look for in a log.

cc: @dominic @arj @mix @regular #flumedb ( #flume ) #javascript #scuttlebot

In case it's useful, I've consolidated everything that I know about the server option here: https://github.com/ssbc/multiserver/issues/33

I had a 6 hour marathon of ssb-bluetooth coding last night, so I'm going to step back from scuttlebutt stuff for a couple of days and focus on day job stuff =p. Do you have the time to look into it @Dom ? It doesn't ~ seem ~ related to the changes I made, right?

I hit this error recently. here's some stuff

Here's the PR where I hit it, I think the solution was something like adding server: true to the config starting the serrver
https://github.com/ssbc/patchbay/pull/299

Here's an issue that was raised earlier which might have more info
https://github.com/ssbc/ssb-server/issues/577

I think ssb-server start probably has some config which is now not working. I've been working to document ssb-config more ... le sigh. cc @Gordon @dominic ?

Nah, if you are already running sbot ssb-server, probably better to install patchwork from source and run via npm.

@nanomonkey The current release of patchwork 3.11.x isn't very friendly with scuttlebot@13

There's some info about how to fix it if you hit problems here: %cPyWHus...

The current master version of patchwork (which will be 3.12.0 when I finally release it), is much better.

@nanomonkey

What version of Patchwork?