After upgrading my pub to ssb-server v16.0.1, it no longer responds to incoming peer connections. The secret-handshake times out after 5 seconds. example.txt. That is with no existing peer connections, no indexing taking place, no high CPU usage, no plugins, and either the old config or no config. Starting up a ssb-server on a separate ssb_appname worked (it would respond to clients). So I tried to import the old flumelog into a new ssb-server directory. But ssb-server would not pick up the new flumelog data. I started making a script using flumecat and sbotc to read the old flumelog and add the messages one by one. That was working, except it got invalid signature on a message (probably some encoding issue) and I did not investigate further, since it would take a very long time to import messages this way, and I want to keep the old received timestamps also. So I reverted back to ssb-server v15.2.0 and that is working well.

Using ssb-server v16.0.1 on my local system and one other sbot is working. (I have not checked whether it picks up data added to log.offset while offline)

@cel-desktop

👀

@cel-desktop

Are you starting SSB-Server from the command-line (bin.js) or from Node.js (index.js)? There was a regression introduced right after 15.2.0 that I've recently found + solved here: https://github.com/ssbc/ssb-server/pull/727

@cel-desktop

Oops, to be explicit: this problem only happens when you run from index.js, the bin.js script is immune.

@Christian Bundy
From bin.js. But I just re-installed again now and it works fine. bin.js requires index.js ('./') so maybe that PR fixed it. Thanks!

Unfortunately, the problem re-occured. Calls to the db like sbot.get block. Reinstalled v15 and it works again.

@cel-desktop

I think that call blocks because the index isn't up-to-date, which might be caused by an deadlock that Mix and I found a couple of days ago in SSB-DB. I've written a test that [I think] reproduces the problem, I'll comment here once the fix makes it into a new release of SSB-Server.

Opened https://github.com/ssbc/ssb-server/issues/734

Added more context to the issue.

Stuck indexing and not responding to incoming connections

ssb-server as of v16 can get into an unresponsive state. In this state, it rejects incoming connections and does not index the flumelog.

When a secret-handshake client tries to connect to the server - even a local one - the server rejects the connection a few seconds after sending the first response packet. Example transcript from trying to connect to the server:

$ strace sbotc -4 whoami
[...]
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(8008), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
getrandom("\316\f%\243\333\220\214\203e\357\312;P\270\211\30", 16, 0) = 16
getrandom("\320\366\27\177:\234]{:\222\245\363Ow\0232\302\315\276\240C\235\344\220\10\365\tj\373\raT", 32, 0) = 32
write(3, "$\213\202\nVI\222wb\354Z\307\\\3*~\257R\376F\23\3710\377\0.J\25\366\322\1\216"..., 64) = 64
read(3, "\205\364n\264\322\17;\342\22\v^\271\310\271@\215\253=\345@\334o\233\320\345}\307#\341OaH"..., 64) = 64
write(3, "\212N\261/\v\301\2061_\36\367\364P\265\314\313\36\201\25\21\341\301\250=;\2p\214m\303\201\20"..., 112) = 112
read(3, [A few seconds passes here...] "", 80)                         = 0
write(2, "sbotc: ", 7sbotc: )                  = 7
write(2, "hello not accepted", 18hello not accepted)      = 18
write(2, ": ", 2: )                       = 2
write(2, "Broken pipe\n", 12Broken pipe
)           = 12

If the server is configured to use the unix noauth socket, sbotc can connect using that. (The -4 option used above forces it to use TCP and thus SHS instead of the noauth socket). Connecting with sbotc using the noauth socket reveals that ssb-server is not indexing its flumelog, as this command shows the progress stuck:

$ while sleep 1; do sbotc progress | jq -c .indexes.target-.indexes.current; done
366939
366939
366939
366939
366939
^C

It looks to me that ssb-server v16 gets into this state when it starts while the flumelog is not caught up. In previous versions, doing this would not be problem. ssb-server should allow appending data to the flumelog while it is offline and then indexing and catching up when it starts. Running ssb-server v15 causes it to catch up indexing, and then switching back to ssb-server v16 works again.