ssbc / go-ssb Goto Github PK

View Code? Open in Web Editor NEW

160.0 9.0 26.0 14.67 MB

Go implementation of ssb (work in progress!)

Home Page: https://scuttlebutt.nz

Go 98.62% JavaScript 0.95% Shell 0.10% Makefile 0.31% Nix 0.02%

go golang secure-scuttlebutt ssb

go-ssb's Issues

ssb/consistency error: message sequence missmatch for feed

This might happen because of a local duplication bug (some messages are append twice) in the verify logic.

It is a fatal error because it cripples replicatio. Tthe system thinks a feed has more messages then it actually does.

The workaround is to start go-sbot with -fsck sequences -repair which will check all the messages in a log, delete the affected feeds. On the next sync/connection it will reload them.

AFAICT this only happens for external messages, not the ones signed by the bot itself.

Data Race

Produced by running go test -race -count 100 in package go.cryptoscpe.co/ssb/sbot

==================
WARNING: DATA RACE
Write at 0x00c00be9a5f1 by goroutine 581:
  runtime.slicecopy()
      /nix/store/9kd1prwr5kkn36x9gw2q62ma24m142db-go-1.12.1/share/go/src/runtime/slice.go:197 +0x0
  go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/margaret/multilog/badger.(*sublog).Get()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/margaret/multilog/badger/sublog.go:27 +0x1f9
  go.cryptoscope.co/ssb/multilogs.publishLog.Append()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/multilogs/publish.go:70 +0x3af
  go.cryptoscope.co/ssb/multilogs.(*publishLog).Append()
      <autogenerated>:1 +0xfd
  go.cryptoscope.co/ssb/sbot.TestFeedsOneByOne()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/sbot/feeds_test.go:89 +0x1005
  testing.tRunner()
      /nix/store/9kd1prwr5kkn36x9gw2q62ma24m142db-go-1.12.1/share/go/src/testing/testing.go:865 +0x163

Previous write at 0x00c00be9a5f1 by goroutine 582:
  runtime.slicecopy()
      /nix/store/9kd1prwr5kkn36x9gw2q62ma24m142db-go-1.12.1/share/go/src/runtime/slice.go:197 +0x0
  go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/margaret/multilog/badger.(*sublog).Get()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/margaret/multilog/badger/sublog.go:27 +0x1f9
  go.cryptoscope.co/ssb/plugins/gossip.(*handler).fetchFeed()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/plugins/gossip/fetch.go:135 +0x1a7b
  go.cryptoscope.co/ssb/plugins/gossip.(*handler).fetchAll()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/plugins/gossip/fetch.go:71 +0x10a
  go.cryptoscope.co/ssb/plugins/gossip.(*handler).fetchAllLib()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/plugins/gossip/fetch.go:38 +0x21c
  go.cryptoscope.co/ssb/plugins/gossip.(*handler).HandleConnect()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/plugins/gossip/handler.go:118 +0xa03
  go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/muxrpc.(*HandlerMux).HandleConnect.func1()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/muxrpc/handler.go:53 +0x66

Goroutine 581 (running) created at:
  testing.(*T).Run()
      /nix/store/9kd1prwr5kkn36x9gw2q62ma24m142db-go-1.12.1/share/go/src/testing/testing.go:916 +0x65a
  testing.runTests.func1()
      /nix/store/9kd1prwr5kkn36x9gw2q62ma24m142db-go-1.12.1/share/go/src/testing/testing.go:1157 +0xa8
  testing.tRunner()
      /nix/store/9kd1prwr5kkn36x9gw2q62ma24m142db-go-1.12.1/share/go/src/testing/testing.go:865 +0x163
  testing.runTests()
      /nix/store/9kd1prwr5kkn36x9gw2q62ma24m142db-go-1.12.1/share/go/src/testing/testing.go:1155 +0x523
  testing.(*M).Run()
      /nix/store/9kd1prwr5kkn36x9gw2q62ma24m142db-go-1.12.1/share/go/src/testing/testing.go:1072 +0x2eb
  main.main()
      _testmain.go:44 +0x222

Goroutine 582 (finished) created at:
  go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/muxrpc.(*HandlerMux).HandleConnect()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/muxrpc/handler.go:51 +0x1d3
  go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/muxrpc.handle.func2()
      /home/cryptix/go/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/muxrpc/rpc.go:82 +0x68
==================

sbot should fetch feeds in parallel instead of one-by-one

Fetching data for feeds is slow because peers are connected to sequentially. To make it faster, it can be done concurrently.

SSB Room

Add support to host SSB Rooms.

SSB room act as a proxy to for user to connect an discover each other but does not have access to user data.

Load sbot configuration flags from file

Allow loading of configuration flags from file

go-sbot configuration flags are currently set via the command line, with default values defined in the code. I'd like to propose adding a --config argument which would allow for flag values to be optionally read from file.

How it might work:

The path for the --config argument defaults to ~/.ssb-go/config:

go-sbot --config

An alternate path and configuration file may be specified:

go-sbot --config /home/glyph/go-sbot-conf

When the --config argument is passed, any flag values found in the file override the defaults set in code. Command line arguments override defaults and values found in file. This allows a mixture of default values, values from file and values from the command line. Hard-coded -> config file -> CLI arguments.

We could also do away with the default file idea and require that a path and filename be specified.

Example contents of the file:

{
  "hops": 2,
  "ebtEnabled": true,
  "localAdv": true,
  "localDiscov": true,
  "wsPort": ":7777"
}

I'm happy to create a PR for this if it will be a welcome addition? I'd like this for the purposes of PeachCloud and think it might be useful for others.

@cryptix

Are there any reasons why this is a bad idea or an undesirable addition? If not, do you have any suggested changes to the behaviour I've outlined?

Follow up: timestamp sorting

See #61 (comment). There were small tests for the functionality and it worked but nothing comprehensive about failure recovery. I had to disable it ( ace3e87 ) to not hinder development more.

A better way of going about this in general would be to have integration tests with ssb-fixtures to sync medium to huge networks and make sure the indexes don't trip.

Support for private messages

Re-index decrypted messages for content data to support client side private messages.

sbotcli connect warning after install

hey @cryptix, I started trying to run the go-sbot, following the README here.

I was able to install go-sbot and sbotcli on my laptop (running mac OS) and on the pi.

When I run go-sbot in either place, it appears to start up, ~/.go-ssb gets populated,

log messages:

level=info t=66.427057ms event="waiting for indexes to catch up"
level=info t=67.688541ms event="repo open" feeds=0 msgs=-1
level=info t=67.717838ms event=serving ID="@rQn3+8VJCO4scVkpzDxVee6cgCp8aCldvspXXAT3EXU=.ed25519" addr=:8008 version=snapshot build=

in another terminal, when I run

sbotcli connect "net:wx.larpa.net:8008~shs:DTNmX+4SjsgZ7xyDh5xxmNtFqa6pWi5Qtw7cE8aR9TQ="
I get the warning:

level=warn event="connect command failed" err="muxrpc: didnt get length of next body (frames:0): EOF"
level=error run-failure="connect: async call failed.: muxrpc: didnt get length of next body (frames:0): EOF"

Above, I'm using the address of your pub which I found in the README. I've also tried a few other pub addresses from my gossip.json generated by patchwork.

also tried,
sbotcli connect "net:ssb.learningsocieties.org:8008~shs:uMiN0TRVMGVNTQUb6KCbiOi/8UQYcyojiA83rCghxGo="

and get this warning,

level=warn event="connect command failed" err="muxrpc: error decoding json from request source: invalid character 'c' looking for beginning of value"
level=error run-failure="connect: async call failed.: muxrpc: didnt get length of next body (frames:0): EOF"

do you know the cause? or another test I should I do?

deadlocks

With current master, after a while go-sbot might still accept connections but doesn't serve messages or blobs. Here is a go routine profile after having triggerd shutdown:

goroutine profile: total 1091
413 @ 0x9a4545 0x9b4fa5 0x9b4f8e 0x9d5c07 0x9e49e5 0x1022e5b 0x10225ae 0x11d3c69 0x11d1a76 0x11d10f8 0xd34677 0xd3f2e2 0x9d9b41
#	0x9d5c06	sync.runtime_SemacquireMutex+0x46						/home/cryptix/go-src/src/runtime/sema.go:71
#	0x9e49e4	sync.(*Mutex).lockSlow+0x104							/home/cryptix/go-src/src/sync/mutex.go:138
#	0x1022e5a	sync.(*Mutex).Lock+0x8fa							/home/cryptix/go-src/src/sync/mutex.go:81
#	0x10225ad	go.cryptoscope.co/ssb/blobstore.(*wantManager).CreateWants+0x4d			/home/cryptix/go-ssb/blobstore/wants.go:276
#	0x11d3c68	go.cryptoscope.co/ssb/plugins/blobs.(*createWantsHandler).HandleSource+0x1e8	/home/cryptix/go-ssb/plugins/blobs/createWants.go:88
#	0x11d1a75	go.cryptoscope.co/muxrpc/v2/typemux.sourceStub.HandleCall+0x95			/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/typemux/source.go:35
#	0x11d10f7	go.cryptoscope.co/muxrpc/v2/typemux.(*HandlerMux).HandleCall+0xf7		/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/typemux/new_mux.go:45
#	0xd34676	go.cryptoscope.co/muxrpc/v2.(*HandlerMux).HandleCall+0xf6			/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/handler.go:61
#	0xd3f2e1	go.cryptoscope.co/muxrpc/v2.(*rpc).fetchRequest.func1+0x61			/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/rpc_server.go:225

350 @ 0x9a4545 0x9b4fa5 0x9b4f8e 0x9d5c07 0x9e49e5 0x11e4886 0x11e4504 0x120e130 0x9d9b41
#	0x9d5c06	sync.runtime_SemacquireMutex+0x46					/home/cryptix/go-src/src/runtime/sema.go:71
#	0x9e49e4	sync.(*Mutex).lockSlow+0x104						/home/cryptix/go-src/src/sync/mutex.go:138
#	0x11e4885	sync.(*Mutex).Lock+0x3e5						/home/cryptix/go-src/src/sync/mutex.go:81
#	0x11e4503	go.cryptoscope.co/ssb/plugins/ebt.(*Sessions).WaitFor+0x63		/home/cryptix/go-ssb/plugins/ebt/session.go:104
#	0x120e12f	go.cryptoscope.co/ssb/sbot.replicateNegotiator.HandleConnect+0xef	/home/cryptix/go-ssb/sbot/replicate_negotiation.go:39

272 @ 0x9a4545 0x9b4fa5 0x9b4f8e 0x9d5c07 0x9e49e5 0x11e4368 0x11e417b 0x11e312b 0x11e2b07 0xd34677 0xd3f2e2 0x9d9b41
#	0x9d5c06	sync.runtime_SemacquireMutex+0x46					/home/cryptix/go-src/src/runtime/sema.go:71
#	0x9e49e4	sync.(*Mutex).lockSlow+0x104						/home/cryptix/go-src/src/sync/mutex.go:138
#	0x11e4367	sync.(*Mutex).Lock+0x227						/home/cryptix/go-src/src/sync/mutex.go:81
#	0x11e417a	go.cryptoscope.co/ssb/plugins/ebt.(*Sessions).Started+0x3a		/home/cryptix/go-ssb/plugins/ebt/session.go:69
#	0x11e312a	go.cryptoscope.co/ssb/plugins/ebt.(*MUXRPCHandler).Loop+0x8a		/home/cryptix/go-ssb/plugins/ebt/handler.go:132
#	0x11e2b06	go.cryptoscope.co/ssb/plugins/ebt.(*MUXRPCHandler).HandleCall+0x486	/home/cryptix/go-ssb/plugins/ebt/handler.go:104
#	0xd34676	go.cryptoscope.co/muxrpc/v2.(*HandlerMux).HandleCall+0xf6		/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/handler.go:61
#	0xd3f2e1	go.cryptoscope.co/muxrpc/v2.(*rpc).fetchRequest.func1+0x61		/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/rpc_server.go:225

8 @ 0x9a4545 0x9b45ef 0xf80bba 0x9d9b41
#	0xf80bb9	github.com/dgraph-io/badger/y.(*WaterMark).process+0x2d9	/home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]+incompatible/y/watermark.go:207

8 @ 0x9a4545 0x9b45ef 0xfa6325 0x9d9b41
#	0xfa6324	github.com/dgraph-io/badger.(*levelsController).runWorker+0x224	/home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]+incompatible/levels.go:348

4 @ 0x9a4545 0x970e4f 0x970a8b 0xfc42c5 0x9d9b41
#	0xfc42c4	github.com/dgraph-io/badger.(*valueLog).waitOnGC+0x64	/home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]+incompatible/value.go:1337

4 @ 0x9a4545 0x970e4f 0x970acb 0xf99357 0xfc6d97 0x9d9b41
#	0xf99356	github.com/dgraph-io/badger.(*DB).flushMemtable+0xb6	/home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]+incompatible/db.go:912
#	0xfc6d96	github.com/dgraph-io/badger.Open.func4+0x36		/home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]+incompatible/db.go:298

4 @ 0x9a4545 0x9b45ef 0xdc12c6 0x9d9b41
#	0xdc12c5	go.cryptoscope.co/margaret/multilog/roaring.(*MultiLog).writeBatches+0xc5	/home/cryptix/go/pkg/mod/go.cryptoscope.co/[email protected]/multilog/roaring/multilog.go:44

4 @ 0x9a4545 0x9b45ef 0xf97c14 0x9d9b41
#	0xf97c13	github.com/dgraph-io/badger.(*DB).doWrites+0x413	/home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]+incompatible/db.go:705

4 @ 0x9a4545 0x9b45ef 0xf99a15 0x9d9b41
#	0xf99a14	github.com/dgraph-io/badger.(*DB).updateSize+0x134	/home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]+incompatible/db.go:998

4 @ 0x9a4545 0x9b45ef 0xfb1c48 0x9d9b41
#	0xfb1c47	github.com/dgraph-io/badger.(*publisher).listenForUpdates+0x167	/home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]+incompatible/publisher.go:65

3 @ 0x9a4545 0x9b45ef 0x103d165 0x9d9b41
#	0x103d164	go.cryptoscope.co/librarian/badger.(*index).writeBatches+0x104	/home/cryptix/go/pkg/mod/go.cryptoscope.co/[email protected]/badger/index.go:156

3 @ 0x9a4545 0x9b4fa5 0x9b4f8e 0x9d5c07 0x9e49e5 0x11e446b 0x11e43cf 0x11e4aa5 0x11e3b87 0x11e2b07 0xd34677 0xd3f2e2 0x9d9b41
#	0x9d5c06	sync.runtime_SemacquireMutex+0x46					/home/cryptix/go-src/src/runtime/sema.go:71
#	0x9e49e4	sync.(*Mutex).lockSlow+0x104						/home/cryptix/go-src/src/sync/mutex.go:138
#	0x11e446a	sync.(*Mutex).Lock+0xca							/home/cryptix/go-src/src/sync/mutex.go:81
#	0x11e43ce	go.cryptoscope.co/ssb/plugins/ebt.(*Sessions).Ended+0x2e		/home/cryptix/go-ssb/plugins/ebt/session.go:89
#	0x11e4aa4	go.cryptoscope.co/ssb/plugins/ebt.(*MUXRPCHandler).Loop.func1+0x64	/home/cryptix/go-ssb/plugins/ebt/handler.go:143
#	0x11e3b86	go.cryptoscope.co/ssb/plugins/ebt.(*MUXRPCHandler).Loop+0xae6		/home/cryptix/go-ssb/plugins/ebt/handler.go:262
#	0x11e2b06	go.cryptoscope.co/ssb/plugins/ebt.(*MUXRPCHandler).HandleCall+0x486	/home/cryptix/go-ssb/plugins/ebt/handler.go:104
#	0xd34676	go.cryptoscope.co/muxrpc/v2.(*HandlerMux).HandleCall+0xf6		/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/handler.go:61
#	0xd3f2e1	go.cryptoscope.co/muxrpc/v2.(*rpc).fetchRequest.func1+0x61		/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/rpc_server.go:225

1 @ 0x976f94 0x9d62fd 0xad8865 0x9d9b41
#	0x9d62fc	os/signal.signal_recv+0x9c	/home/cryptix/go-src/src/runtime/sigqueue.go:147
#	0xad8864	os/signal.loop+0x24		/home/cryptix/go-src/src/os/signal/signal_unix.go:23

1 @ 0x9a4545 0x97016a 0x96ff15 0x1025008 0x11d5aa3 0xd3d722 0x11d3d6f 0x11d1a76 0x11d10f8 0xd34677 0xd3f2e2 0x9d9b41
#	0x1025007	go.cryptoscope.co/ssb/blobstore.(*wantProc).Pour+0xa27					/home/cryptix/go-ssb/blobstore/wants.go:498
#	0x11d5aa2	go.cryptoscope.co/ssb/plugins/blobs.(*createWantsHandler).HandleSource.func1+0x142	/home/cryptix/go-ssb/plugins/blobs/createWants.go:100
#	0xd3d721	go.cryptoscope.co/muxrpc/v2.(*ByteSource).Reader+0x81					/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/stream2_source.go:143
#	0x11d3d6e	go.cryptoscope.co/ssb/plugins/blobs.(*createWantsHandler).HandleSource+0x2ee		/home/cryptix/go-ssb/plugins/blobs/createWants.go:94
#	0x11d1a75	go.cryptoscope.co/muxrpc/v2/typemux.sourceStub.HandleCall+0x95				/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/typemux/source.go:35
#	0x11d10f7	go.cryptoscope.co/muxrpc/v2/typemux.(*HandlerMux).HandleCall+0xf7			/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/typemux/new_mux.go:45
#	0xd34676	go.cryptoscope.co/muxrpc/v2.(*HandlerMux).HandleCall+0xf6				/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/handler.go:61
#	0xd3f2e1	go.cryptoscope.co/muxrpc/v2.(*rpc).fetchRequest.func1+0x61				/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/rpc_server.go:225

1 @ 0x9a4545 0x99d25b 0x9d3c15 0xa4f9e5 0xa50a25 0xa50a03 0xaa974f 0xabca2e 0xcd92b8 0x9d9b41
#	0x9d3c14	internal/poll.runtime_pollWait+0x54		/home/cryptix/go-src/src/runtime/netpoll.go:222
#	0xa4f9e4	internal/poll.(*pollDesc).wait+0x44		/home/cryptix/go-src/src/internal/poll/fd_poll_runtime.go:87
#	0xa50a24	internal/poll.(*pollDesc).waitRead+0x1a4	/home/cryptix/go-src/src/internal/poll/fd_poll_runtime.go:92
#	0xa50a02	internal/poll.(*FD).Read+0x182			/home/cryptix/go-src/src/internal/poll/fd_unix.go:159
#	0xaa974e	net.(*netFD).Read+0x4e				/home/cryptix/go-src/src/net/fd_posix.go:55
#	0xabca2d	net.(*conn).Read+0x8d				/home/cryptix/go-src/src/net/net.go:182
#	0xcd92b7	net/http.(*connReader).backgroundRead+0x57	/home/cryptix/go-src/src/net/http/server.go:690

1 @ 0x9a4545 0x99d25b 0x9d3c15 0xa4f9e5 0xa525dc 0xa525be 0xaaacc5 0xac6af2 0xac5925 0xce40e6 0xce3e17 0x12594b4 0x125945d 0x9d9b41
#	0x9d3c14	internal/poll.runtime_pollWait+0x54		/home/cryptix/go-src/src/runtime/netpoll.go:222
#	0xa4f9e4	internal/poll.(*pollDesc).wait+0x44		/home/cryptix/go-src/src/internal/poll/fd_poll_runtime.go:87
#	0xa525db	internal/poll.(*pollDesc).waitRead+0x1fb	/home/cryptix/go-src/src/internal/poll/fd_poll_runtime.go:92
#	0xa525bd	internal/poll.(*FD).Accept+0x1dd		/home/cryptix/go-src/src/internal/poll/fd_unix.go:394
#	0xaaacc4	net.(*netFD).accept+0x44			/home/cryptix/go-src/src/net/fd_unix.go:172
#	0xac6af1	net.(*TCPListener).accept+0x31			/home/cryptix/go-src/src/net/tcpsock_posix.go:139
#	0xac5924	net.(*TCPListener).Accept+0x64			/home/cryptix/go-src/src/net/tcpsock.go:261
#	0xce40e5	net/http.(*Server).Serve+0x265			/home/cryptix/go-src/src/net/http/server.go:2937
#	0xce3e16	net/http.(*Server).ListenAndServe+0xb6		/home/cryptix/go-src/src/net/http/server.go:2866
#	0x12594b3	net/http.ListenAndServe+0x173			/home/cryptix/go-src/src/net/http/server.go:3120
#	0x125945c	main.startDebug.func1+0x11c			/home/cryptix/go-ssb/cmd/go-sbot/metrics.go:86

1 @ 0x9a4545 0x99d25b 0x9d3c15 0xa4f9e5 0xa525dc 0xa525be 0xaaacc5 0xaccc92 0xacb385 0x1213749 0x9d9b41
#	0x9d3c14	internal/poll.runtime_pollWait+0x54			/home/cryptix/go-src/src/runtime/netpoll.go:222
#	0xa4f9e4	internal/poll.(*pollDesc).wait+0x44			/home/cryptix/go-src/src/internal/poll/fd_poll_runtime.go:87
#	0xa525db	internal/poll.(*pollDesc).waitRead+0x1fb		/home/cryptix/go-src/src/internal/poll/fd_poll_runtime.go:92
#	0xa525bd	internal/poll.(*FD).Accept+0x1dd			/home/cryptix/go-src/src/internal/poll/fd_unix.go:394
#	0xaaacc4	net.(*netFD).accept+0x44				/home/cryptix/go-src/src/net/fd_unix.go:172
#	0xaccc91	net.(*UnixListener).accept+0x31				/home/cryptix/go-src/src/net/unixsock_posix.go:162
#	0xacb384	net.(*UnixListener).Accept+0x64				/home/cryptix/go-src/src/net/unixsock.go:260
#	0x1213748	go.cryptoscope.co/ssb/sbot.WithUNIXSocket.func1.1+0x48	/home/cryptix/go-ssb/sbot/options.go:245

1 @ 0x9a4545 0x9b4fa5 0x9b4f8e 0x9d5c07 0x9e49e5 0x1025c46 0x1025965 0xd24756 0xd24268 0x101f5fc 0x1020b4d 0x1025fa5 0x9d9b41
#	0x9d5c06	sync.runtime_SemacquireMutex+0x46				/home/cryptix/go-src/src/runtime/sema.go:71
#	0x9e49e4	sync.(*Mutex).lockSlow+0x104					/home/cryptix/go-src/src/sync/mutex.go:138
#	0x1025c45	sync.(*Mutex).Lock+0x345					/home/cryptix/go-src/src/sync/mutex.go:81
#	0x1025964	go.cryptoscope.co/ssb/blobstore.NewWantManager.func1+0x64	/home/cryptix/go-ssb/blobstore/wants.go:62
#	0xd24755	go.cryptoscope.co/luigi.FuncSink.Pour+0x55			/home/cryptix/go/pkg/mod/go.cryptoscope.co/[email protected]/func.go:14
#	0xd24267	go.cryptoscope.co/luigi.(*broadcastSink).Pour+0x2c7		/home/cryptix/go/pkg/mod/go.cryptoscope.co/[email protected]/broadcast.go:62
#	0x101f5fb	go.cryptoscope.co/ssb/blobstore.(*blobStore).Put+0xa9b		/home/cryptix/go-ssb/blobstore/store.go:152
#	0x1020b4c	go.cryptoscope.co/ssb/blobstore.(*wantManager).getBlob+0x5ac	/home/cryptix/go-ssb/blobstore/wants.go:168
#	0x1025fa4	go.cryptoscope.co/ssb/blobstore.NewWantManager.func2+0x2a4	/home/cryptix/go-ssb/blobstore/wants.go:105

1 @ 0x9a4545 0x9b4fa5 0x9b4f8e 0x9d5c07 0x9e49e5 0x1205906 0x12051cb 0x12592e5 0x9d9b41
#	0x9d5c06	sync.runtime_SemacquireMutex+0x46		/home/cryptix/go-src/src/runtime/sema.go:71
#	0x9e49e4	sync.(*Mutex).lockSlow+0x104			/home/cryptix/go-src/src/sync/mutex.go:138
#	0x1205905	sync.(*Mutex).Lock+0x785			/home/cryptix/go-src/src/sync/mutex.go:81
#	0x12051ca	go.cryptoscope.co/ssb/sbot.(*Sbot).Close+0x4a	/home/cryptix/go-ssb/sbot/new.go:106
#	0x12592e4	main.runSbot.func3+0x264			/home/cryptix/go-ssb/cmd/go-sbot/main.go:232

1 @ 0x9a4545 0x9b4fa5 0x9b4f8e 0x9d5c07 0x9e49e5 0xd3d005 0xd3cf0f 0xd3ac09 0x11cd3a3 0x120568b 0x12571d8 0x1258326 0x9a4149 0x9d9b41
#	0x9d5c06	sync.runtime_SemacquireMutex+0x46			/home/cryptix/go-src/src/runtime/sema.go:71
#	0x9e49e4	sync.(*Mutex).lockSlow+0x104				/home/cryptix/go-src/src/sync/mutex.go:138
#	0xd3d004	sync.(*Mutex).Lock+0x124				/home/cryptix/go-src/src/sync/mutex.go:81
#	0xd3cf0e	go.cryptoscope.co/muxrpc/v2.(*ByteSource).Cancel+0x2e	/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/stream2_source.go:65
#	0xd3ac08	go.cryptoscope.co/muxrpc/v2.(*rpc).Terminate+0x168	/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/rpc_server.go:468
#	0x11cd3a2	go.cryptoscope.co/ssb/network.(*node).Close+0x222	/home/cryptix/go-ssb/network/new.go:513
#	0x120568a	go.cryptoscope.co/ssb/sbot.(*Sbot).Close+0x50a		/home/cryptix/go-ssb/sbot/new.go:117
#	0x12571d7	main.runSbot+0x2597					/home/cryptix/go-ssb/cmd/go-sbot/main.go:389
#	0x1258325	main.main+0x25						/home/cryptix/go-ssb/cmd/go-sbot/main.go:397
#	0x9a4148	runtime.main+0x208					/home/cryptix/go-src/src/runtime/proc.go:204

1 @ 0x9a4545 0x9b4fa5 0x9b4f8e 0x9d5c07 0x9e49e5 0xd3de45 0xd3dbe8 0xd3dac5 0xd39e2d 0x11cb6f9 0x9d9b41
#	0x9d5c06	sync.runtime_SemacquireMutex+0x46				/home/cryptix/go-src/src/runtime/sema.go:71
#	0x9e49e4	sync.(*Mutex).lockSlow+0x104					/home/cryptix/go-src/src/sync/mutex.go:138
#	0xd3de44	sync.(*Mutex).Lock+0x2a4					/home/cryptix/go-src/src/sync/mutex.go:81
#	0xd3dbe7	go.cryptoscope.co/muxrpc/v2.(*frameBuffer).copyBody+0x47	/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/stream2_source.go:201
#	0xd3dac4	go.cryptoscope.co/muxrpc/v2.(*ByteSource).consume+0x144		/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/stream2_source.go:170
#	0xd39e2c	go.cryptoscope.co/muxrpc/v2.(*rpc).Serve+0x3cc			/home/cryptix/go/pkg/mod/go.cryptoscope.co/muxrpc/[email protected]/rpc_server.go:421
#	0x11cb6f8	go.cryptoscope.co/ssb/network.(*node).handleConnection+0xa98	/home/cryptix/go-ssb/network/new.go:329

1 @ 0x9d37fd 0xd01b82 0xd01945 0xcfe532 0xd0c1c5 0xd0daa5 0xce0724 0xce264d 0xce3d23 0xcdf52d 0x9d9b41
#	0x9d37fc	runtime/pprof.runtime_goroutineProfileWithLabels+0x5c	/home/cryptix/go-src/src/runtime/mprof.go:716
#	0xd01b81	runtime/pprof.writeRuntimeProfile+0xe1			/home/cryptix/go-src/src/runtime/pprof/pprof.go:724
#	0xd01944	runtime/pprof.writeGoroutine+0xa4			/home/cryptix/go-src/src/runtime/pprof/pprof.go:684
#	0xcfe531	runtime/pprof.(*Profile).WriteTo+0x3f1			/home/cryptix/go-src/src/runtime/pprof/pprof.go:331
#	0xd0c1c4	net/http/pprof.handler.ServeHTTP+0x384			/home/cryptix/go-src/src/net/http/pprof/pprof.go:256
#	0xd0daa4	net/http/pprof.Index+0x944				/home/cryptix/go-src/src/net/http/pprof/pprof.go:367
#	0xce0723	net/http.HandlerFunc.ServeHTTP+0x43			/home/cryptix/go-src/src/net/http/server.go:2042
#	0xce264c	net/http.(*ServeMux).ServeHTTP+0x1ac			/home/cryptix/go-src/src/net/http/server.go:2417
#	0xce3d22	net/http.serverHandler.ServeHTTP+0xa2			/home/cryptix/go-src/src/net/http/server.go:2843
#	0xcdf52c	net/http.(*conn).serve+0x8ac				/home/cryptix/go-src/src/net/http/server.go:1925

Most suspicious are the sync.Lock() lines around EBT and BlobsWant.

Memory exhaustion when running on 32bits

Launching go-sbot.exe compiled for windows/arm (32bits ARMv7) causes a memory exhaustion error on Nmap:

PS C:\Users\andre\ssbc\go-ssb\cmd\go-sbot> .\go-sbot.exe
time=2020-05-13T17:43:43.2457731+01:00 caller=metrics.go:87 module=sbot starting=metrics addr=localhost:6078
time=2020-05-13T17:43:43.2897736+01:00 caller=stdlib.go:89 module=stdlib ts="2020/05/13 17:43:43" msg="saved identity @R3i0Wa6YIBKapV/1Z3heGggoW+1UJVkm88/Us0Tk2WA=.ed25519 to C:\\Users\\andre\\.ssb-go\\secret"
level=warn time=2020-05-13T17:43:43.3117725+01:00 caller=new.go:120 module=sbot event="bot init" msg="loading default idx" idx=userFeeds
badger 2020/05/13 17:43:43 INFO: All 0 tables opened in 1ms
time=2020-05-13T17:43:43.4347706+01:00 caller=log.go:28 check=fatal err="sbot: OpenContacts failed: error getting contacts index: db/idx: badger failed to open: Mmap value log file. Path=C:\\Users\\andre\\.ssb-go\\indexes\\contacts\\db\\000000.vlog. Error=MapViewOfFile: Not enough memory resources are available to process this command."
time=2020-05-13T17:43:43.4447724+01:00 caller=recovery.go:63 event=panic location=CheckFatal panicLog=panics\CheckFatal517618535
panicWithStack: wrote panics\CheckFatal517618535
PS C:\Users\andre\ssbc\go-ssb\cmd\go-sbot>

Panic on bad gossip (?) response

The go-sbot panics when an error is returned without header flag 'isEndOrError' set to true.

level=debug time=2020-04-24T13:53:51.770526929+02:00 caller=new.go:218 module=sbot invite="not for us"
level=warn time=2020-04-24T13:53:51.770591321+02:00 caller=auth.go:38 module=sbot module=graph module=graph msg="authbypass - trust on first use"
time=2020-04-24T13:53:51.770782005+02:00 caller=handler.go:68 module=sbot plugin=gossip remote=<@zwWh.ed25519> event=gossiprx handleConnect="oops - dont have my own feed. requesting..."
level=debug time=2020-04-24T13:53:51.772856289+02:00 caller=fetch.go:201 module=sbot plugin=gossip event=gossiprx fr=<@Seaq.ed25519> latest=0 received=1 took=2.046944ms
panic: runtime error: slice bounds out of range [:15] with capacity 8

goroutine 81 [running]:
go.cryptoscope.co/ssb/message/legacy.Verify(0xc000120b18, 0x4, 0x8, 0x0, 0xc00009ac00, 0x0, 0xc000204b28, 0xc00010f7a6)
        /home/freight/src/ssb/message/legacy/verify.go:43 +0xaf9
go.cryptoscope.co/ssb/message.legacyVerify.Verify(0x0, 0xce4ca0, 0xc000199b20, 0x7b8a00, 0xc00010f808, 0xc00010f7a0, 0x3)
        /home/freight/src/ssb/message/drains.go:55 +0x8f
go.cryptoscope.co/ssb/message.(*streamDrain).Pour(0xc000080ac0, 0xf3fbe0, 0xc000204960, 0xce4ca0, 0xc000199b20, 0xc000199b20, 0x0)
        /home/freight/src/ssb/message/drains.go:116 +0x65
go.cryptoscope.co/luigi/mfr.(*sinkMap).Pour(0xc000199a00, 0xf3fbe0, 0xc000204960, 0xce4ca0, 0xc000199b20, 0x0, 0x0)
        /home/freight/src/go/pkg/mod/go.cryptoscope.co/[email protected]/mfr/map.go:36 +0xba
go.cryptoscope.co/luigi.Pump(0xf3fbe0, 0xc000204960, 0xf36320, 0xc000199a00, 0xf2cec0, 0xc0000f29c0, 0x1, 0x1)
        /home/freight/src/go/pkg/mod/go.cryptoscope.co/[email protected]/stream.go:65 +0x11f
go.cryptoscope.co/ssb/plugins/gossip.(*handler).fetchFeed(0xc00022a0d0, 0xf3fba0, 0xc00003a120, 0xc000104ae0, 0xf46e20, 0xc0000f2510, 0xbfa0d2b3edf194aa, 0x91b90d36, 0x16d40a0, 0x0, ...)
        /home/freight/src/ssb/plugins/gossip/fetch.go:240 +0x955
go.cryptoscope.co/ssb/plugins/gossip.(*handler).HandleConnect(0xc00022a0d0, 0xf3fba0, 0xc00003a120, 0xf46e20, 0xc0000f2510)
        /home/freight/src/ssb/plugins/gossip/handler.go:69 +0xdae
go.cryptoscope.co/muxrpc.(*HandlerMux).HandleConnect.func1(0xf3fba0, 0xc00003a120, 0xf46e20, 0xc0000f2510, 0xc0001204c0, 0xf366e0, 0xc00022a0d0)
        /home/freight/src/go/pkg/mod/go.cryptoscope.co/[email protected]/handler.go:58 +0x59
created by go.cryptoscope.co/muxrpc.(*HandlerMux).HandleConnect
        /home/freight/src/go/pkg/mod/go.cryptoscope.co/[email protected]/handler.go:56 +0x127

refactor network code

Right now we can only listen on ONE network address for incoming connections. It would be rad to have multiple ones with different connTrackers (localhost with no limits, public with only one connection per pubkey, for instance)

Another limiting factor is that for mobile applications, the listeners might need to be suspended and resumed, also with changing addresses.

sbotcli call invite.create

When I run sbotcli call invite.create or sbotcli call invite.create 10

I get the error:

level=error run-failure="invite.create: call failed.: muxrpc: error decoding json from request source: invalid character ':' looking for beginning of value"

make sbot an interface

Right now there are random indicies hanging of the type Sbot struct from go.cryptoscope.co/ssb/sbot. Some applications don't need them and they waste ram and disk space.

It would be nice to configure which indicies should be present on the bot at creation time and replace the hardcoded struct fields with a method to get the different indicies. Maybe something like:

type Sbot interface{
  GetMultilog(name string) (multilog.Multilog, error)
}

and an option to create them like:

func MountIndex(name string, logCreate ...) Option {
  return func(sbot *sbot) error {
    // hook up index to sbot
  }
}

go-sbot build failure

go.cryptoscope.co/ssb/cmd/go-sbot

cmd/go-sbot/main.go:172:15: first argument to append must be slice; have map[string]*ssb.KeyPair
cmd/go-sbot/main.go:174:51: cannot use kps (type map[string]*ssb.KeyPair) as type []*ssb.KeyPair in argument to multilogs.NewPrivateRea
d

Build error due to bad dependency

go install ./cmd/go-sbot/

go: go.cryptoscope.co/[email protected]: parsing /home/cryptix/go/src/go.cryptoscope.co/muxrpc/go.mod: open /home/cryptix/go/src/go.cryptoscope.co/muxrpc/go.mod: no such file or directory

Advertisements sent on addresses not listened too

When running:

go-sbot -localadv -l :8001

The sbot will advertise on both IPv4, and IPv6 subnets, though only IPv6 is being listened on.

message verify failed (sbot: invalid signature)

There are still some edge cases around message verification:

caller=handler.go:112 module=sbot plugin=gossip handleConnect="fetchFeed hops failed" 
err="fetchFeed(@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519:4505):
message verify failed:
ssb Verify(@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519:4506):
could not verify message: sbot: invalid signature"

If you have the feed in question you can do this to reproduce the problem and get an extended diff:

$ cd ssb/message
$ npm i
$ node encode_test.js '@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519'
$ go test

2019/01/21 23:07:25 loaded 5366 messages from testdata.zip
--- FAIL: TestSignatureVerify (1.05s)
    signature_test.go:26: strings are not identical: 
        {
          "previous": "%N4UCoD60og80twWAv6cliXOAPfBcg2yU1bFxnMz4ElM=.sha256",
          "author": "@/02iw6SFEPIHl8nMkYSwcCgRWxiG6VP547Wcp1NW8Bo=.ed25519",
          "sequence": 4506,
          "timestamp": 1462418380624,
          "hash": "sha256",
          "content": {
            "type": "post",
            "text": "- http://research.microsoft.com/en-us/um/people/simonpj/Papers/financial-contracts/contracts-icfp.pdf\n\nopens with haskell and one-letter variable names. great start.\n\n> At this point, any red-blooded funtional programmer should start to foam at the mouth, yelling \"build a combinator library\". And indeed, that turns out to be not only possible, but tremendously bene\fu000cficial.\n\nok agree, way better than that powerpoint\n\n> `c7 = scaleK 100 (get (truncate t1 (one GBP)))`\n\nso the author says these are the four functions that make a contract that pays 100 GBP at time t1.\n\n> Why did we go to the trouble of de\fu000cfining zb in terms of four combinators, rather than making it primitive? Beause it turns out that `scaleK`, `get`, `truncate`, and `one` are all independently useful.\n\n\"it turns out\" instantiates a contract too, i think.\n\n> The temperature in Los Angeles can be objectively measured; but the value to me of insuring my house is subjective, and is not an observable. Observables are thus a diff\u000berent \"kind of thing\" from contracts, so we give them a di\u000bfferent type:\n\nnot sure i buy that syllogism...\n\n![legalese.png](&wWf1aNvgqvtWnu7fywNff2UALjcrsmKxK4RgePUixm4=.sha256)\n\nok sure\n\n![gibberish.png](&6/6P/avhWbg9GNVVZ8qqBatjMwwPEHil1mRm3uzuUxc=.sha256)\n\n:\\\n\n> Much of the subtlety in \fu000cfinancial contracts arises because the participants can exercise choices. We enapsulate choice in two primitive combinators, or and anytime. The former allows one to choose which of two contracts to acquire (this section), while the latter allows one to choose when to acquire it (Section 3.5).\n\nsure, why not\n\nthere is a second section about evaluation, which hopefully answers the burning questions this section raises (next msg)",
            "root": "%9NYn+97xLhIqI0oPdiWxOckPODa9X267553iFtEDH74=.sha256",
            "branch": "%N4UCoD60og80twWAv6cliXOAPfBcg2yU1bFxnMz4ElM=.sha256",
            "mentions": [
              {
                "link": "&wWf1aNvgqvtWnu7fywNff2UALjcrsmKxK4RgePUixm4=.sha256",
                "name": "legalese.png",
                "size": 135799,
                "type": "image/png"
              },
              {
                "link": "&6/6P/avhWbg9GNVVZ8qqBatjMwwPEHil1mRm3uzuUxc=.sha256",
                "name": "gibberish.png",
                "size": 53361,
                "type": "image/png"
              }
            ],
            "channel": "networks"
          }
        }
    require.go:794: 
                Error Trace:    signature_test.go:29
                Error:          Received unexpected error:
                                sbot: invalid signature
                                go.cryptoscope.co/ssb/message.Signature.Verify
                                        /home/cryptix/go/src/go.cryptoscope.co/ssb/message/signature.go:56
                                go.cryptoscope.co/ssb/message.TestSignatureVerify
                                        /home/cryptix/go/src/go.cryptoscope.co/ssb/message/signature_test.go:28
                                testing.tRunner
                                        /nix/store/1205i5sksik4dqlr85pphrrllmic9a9n-go-1.11.4/share/go/src/testing/testing.go:827
                                runtime.goexit
                                        /nix/store/1205i5sksik4dqlr85pphrrllmic9a9n-go-1.11.4/share/go/src/runtime/asm_amd64.s:1333
                Test:           TestSignatureVerify
                Messages:       verify failed
FAIL
exit status 1
FAIL    go.cryptoscope.co/ssb/message   1.241s

(The paste stripped the color encoding. The gist of the diff is should have \f but has \u000c)

There is another one of these involving float encoding documented in this thread: http://viewer.scuttlebot.io/%25ej5uaph%2BvBGQmX%2B9lFywOLzyZMbpBvZ5khm26ERdZbo%3D.sha256

[EBT] empty state file (unexpected EOF)

I've seen a couple of cases on my pub where the EBT session couldn't begin because the stored state file was empty (zero bytes).

The only case I could think if is that the server crashed between creating and writing the file, here:

https://github.com/cryptoscope/ssb/blob/f8b517c21b5d9b8fb05f923f29e59dda9a1730e5/internal/statematrix/ql.go#L149

This should be changed to an atomic write where it writes to new tmpfile and moves it over the existing one once it is done.

Test blocking

Right now I just set the connection between peers to zero but I fear it might just route around in that case.

Resource leak

We got much better at this but there still is slow leakage in memory and/or goroutines.

Fileutil error on android

I'm getting this error when running the sbot on android:

E/Go: panic: open /proc/sys/kernel/osrelease: permission denied
E/Go: goroutine 1 [running]:
E/Go: github.com/pietgeursen/gobotexample/vendor/modernc.org/fileutil.init.0()
E/Go: 	/home/piet/Documents/projects/go/src/github.com/pietgeursen/gobotexample/vendor/modernc.org/fileutil/fileutil_linux.go:34 +0x300

Using code from this commit

A second sbotcli connect results in fail

Running sbotcli connect more than once results in the following error

caller=main.go:95 module=cli runErr="init: base64 decode of --remoteKey failed: error dialing: error wrapping connection: secrethandshake: other side not authenticated"

"legacy SSB messages"

The README mentions:

This currently constructs legacy SSB messages, that still have the signature inside the signed value

And then later:

a new, canonical format is badly needed.

And in the protocol reference it explains how the signature is inside the signed message. I take it that is how SSB currently does things.

Is there something I'm missing as to why you referred to it as a "legacy" format in the README? Has a newer format been proposed somewhere?

[Compilation error] go: error loading module requirements

sudo apt install golang-1.14
export PATH=$PATH:/usr/local/go/bin
git clone https://github.com/cryptoscope/ssb.git go-sbot
cd go-sbot/cmd/go-sbot/

go build -v -i
go: finding github.com/catherinejones/testdiff v0.0.0-20180525195050-ae148f75f077
go: finding github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d
Fetching https://go.cryptoscope.co/secretstream?go-get=1
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding github.com/prometheus/client_golang v0.9.1
Fetching https://golang.org/x/sys?go-get=1
go: finding github.com/gogo/protobuf v1.1.1
go: finding github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc
go: finding github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: finding github.com/AndreasBriese/bbloom v0.0.0-20180913140656-343706a395b7
go: finding github.com/ugorji/go/codec v0.0.0-20190126102652-8fd0f8d918c8
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys?go-get=1
go: finding golang.org/x/sys v0.0.0-20190124100055-b90733256f2e
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
get "golang.org/x/net": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net?go-get=1
go: finding golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3
go: github.com/catherinejones/[email protected]: git fetch -f https://github.com/catherinejones/testdiff refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/fred/go/pkg/mod/cache/vcs/a6ef7cffc03906c1219d9d15bc4776afe3583e713a02c67cba3b7602fe594b66: exit status 128:
	fatal: could not read Username for 'https://github.com': terminal prompts disabled
go: finding github.com/shurcooL/go v0.0.0-20190121191506-3fef8c783dec
Fetching https://go.cryptoscope.co/librarian?go-get=1
go: finding github.com/dgryski/go-farm v0.0.0-20190104051053-3adb47b1fb0f
Parsing meta tags from https://go.cryptoscope.co/secretstream?go-get=1 (status code 200)
get "go.cryptoscope.co/secretstream": found meta tag get.metaImport{Prefix:"go.cryptoscope.co/secretstream", VCS:"git", RepoRoot:"https://github.com/cryptoscope/secretstream"} at https://go.cryptoscope.co/secretstream?go-get=1
go: finding go.cryptoscope.co/secretstream v1.1.4-0.20190911135450-b2c157275a98
Parsing meta tags from https://go.cryptoscope.co/librarian?go-get=1 (status code 200)
get "go.cryptoscope.co/librarian": found meta tag get.metaImport{Prefix:"go.cryptoscope.co/librarian", VCS:"git", RepoRoot:"https://github.com/cryptoscope/librarian"} at https://go.cryptoscope.co/librarian?go-get=1
go: finding go.cryptoscope.co/librarian v0.1.1
go: go.cryptoscope.co/[email protected]: unknown revision b2c157275a98
go: error loading module requirements

uname -a
Linux 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux

handle feed consistency errors more gracefully

Even though the gossip plugin has strict locking to only fetch each feed once at a time it seems like there is still a problem with this...

What ever the cause may be, if the gossip scheduler throws consistency error: wrong stored message sequence for .. we need to handle that gracefully.

first reported here: sunrise-choir/sunrise-social-android-app#2

doesn't build on go1.10

libp2p/go-reuseport states the requirement but I didn't see it.

# go.cryptoscope.co/ssb/vendor/github.com/libp2p/go-reuseport
vendor/github.com/libp2p/go-reuseport/interface.go:33:20: undefined: net.ListenConfig
vendor/github.com/libp2p/go-reuseport/interface.go:41:2: not enough arguments to return
vendor/github.com/libp2p/go-reuseport/interface.go:48:2: not enough arguments to return
vendor/github.com/libp2p/go-reuseport/interface.go:60:10: unknown field 'Control' in struct literal of type net.Dialer

I'm currently leaning towards just dropping go1.10 all together.

cc @keks @PragmaticCypher do you have objections?

badger shutdown issues

In some cases badger (the current key-value store backing our indexes) freaks out if you try to close it with open queries:

panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                                    
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x85bdb7]                                                                                                                                                    
                                                                                                                                                                                                                           
goroutine 29520910 [running]:                                                                                                                                                                                              
github.com/dgraph-io/badger/skl.(*Skiplist).IncrRef(...)                                                                                                                                                                   
        /home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]/skl/skl.go:86                                                                                                                                          
github.com/dgraph-io/badger.(*DB).getMemTables(0xc0001e4000, 0x0, 0x0, 0x0, 0x0)                                                                                                                                           
        /home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]/db.go:475 +0xd7                                                                                                                                        
github.com/dgraph-io/badger.(*DB).get(0xc0001e4000, 0xc00135a380, 0x31, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)                                                                                                           
        /home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]/db.go:499 +0x63                                                                                                                                        
github.com/dgraph-io/badger.(*Txn).Get(0xc0015e8000, 0xc01b5f46c0, 0x29, 0x30, 0xc01b6574e8, 0x42c68f, 0x8)                                                                                                                
        /home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]/transaction.go:409 +0xe7                                                                                                                               
go.cryptoscope.co/margaret/multilog/badger.(*sublog).Get.func1(0xc0015e8000, 0xd82af8, 0xc0015e8000)                                                                                                                       
        /home/cryptix/go/pkg/mod/go.cryptoscope.co/[email protected]/multilog/badger/sublog.go:30 +0x5c                                                                                                                      
github.com/dgraph-io/badger.(*DB).View(0xc0001e4000, 0xc01b6575a0, 0x0, 0x0)                                                                                                                                               
        /home/cryptix/go/pkg/mod/github.com/dgraph-io/[email protected]/transaction.go:587 +0x7e                                                                                                                               
go.cryptoscope.co/margaret/multilog/badger.(*sublog).Get(0xc01a2b0960, 0xe87d20, 0xc01c414048, 0x0, 0x0, 0x0, 0x0)                                                                                                        
        /home/cryptix/go/pkg/mod/go.cryptoscope.co/[email protected]/multilog/badger/sublog.go:29 +0xfd                                                                                                                     
go.cryptoscope.co/ssb/plugins/gossip.(*handler).fetchFeed(0xc0001305a0, 0xe99700, 0xc0215db780, 0xc002ad8180, 0xea5b80, 0xc000c72ea0, 0x0, 0x0)                                                                           
        /home/cryptix/go/src/go.cryptoscope.co/ssb/plugins/gossip/fetch.go:68 +0x17f6                                                                                                                                     
go.cryptoscope.co/ssb/plugins/gossip.(*handler).fetchAll(0xc0001305a0, 0xe99700, 0xc0215db780, 0xea5b80, 0xc000c72ea0, 0xe9b480, 0xc000156da8, 0xc019d0f8c0, 0x20)                                                                /home/cryptix/go/src/go.cryptoscope.co/ssb/plugins/gossip/handler.go:151 +0xc2
go.cryptoscope.co/ssb/plugins/gossip.(*handler).fetchAllLib(0xc0001305a0, 0xe99700, 0xc0215db780, 0xea5b80, 0xc000c72ea0, 0xc0134f8000, 0x7ea, 0xa00, 0x8013deb28, 0x0)
        /home/cryptix/go/src/go.cryptoscope.co/ssb/plugins/gossip/handler.go:128 +0x148
go.cryptoscope.co/ssb/plugins/gossip.(*handler).HandleConnect(0xc0001305a0, 0xe99700, 0xc0215db780, 0xea5b80, 0xc000c72ea0)
        /home/cryptix/go/src/go.cryptoscope.co/ssb/plugins/gossip/handler.go:112 +0x677
go.cryptoscope.co/muxrpc.(*HandlerMux).HandleConnect.func1(0xe99700, 0xc0215db780, 0xea5b80, 0xc000c72ea0, 0xc01ce46da0, 0xe916c0, 0xc0001305a0)                                                                          
        /home/cryptix/go/pkg/mod/go.cryptoscope.co/[email protected]/handler.go:53 +0x59
created by go.cryptoscope.co/muxrpc.(*HandlerMux).HandleConnect
        /home/cryptix/go/pkg/mod/go.cryptoscope.co/[email protected]/handler.go:51 +0x125

The dropFeed branch has overhauled closing behavior. I hope that will improve things.

I'd actually like to have a database that just closes it's transactions without panicing when I call Close() on it, though...

major refactors

Wanted to document this for a while. This is one of the core reason why go-ssb is so wasteful with RAM.

turn luigi into a pattern

We use the luigi package for a lot data processing between the offset-database (margaret) and our muxrpc implementation. At it's core it offers async read and write (with context cancellation) and an observable pattern.

It is functional but wastes a lot of memory since every single message get sent through a channel inside interface{}. This means that the messages all end up on the heap as single-use things and then need to be cleaned up again.

One further complication is that we use luigi streams as data transformations (i.e. bytes from the connection > muxrpc frame > validation process), these also involve (un)packing empty interfaces (and escaping to the heap) even though most of these processes could be tight loops and re-using buffers

Follow up: EBT improvments

It would be really good to cover more of the unchecked tests in #72 (comment)

but mostly I'd love to see this stuff:

Update the vector clock / want list not just once after connecting
Currently the sets of feeds we want to receive is just send once when the connection is established.
The JS documentation is a bit lacking about when to actually re-send those and thus confirm received messages.
But also it doesn't send a clock update once we call sbot.Replicate(feed) - which means reception is delayed until a reconnect... at least this a part should be fixed before merging this.
Implement a more scalable solution to canceling individual feed subscriptions
Right now there is a context per feed. This works but is inefficient. Probably need to re-work the multisink so that they can be unsubscribed directly.
Partition sets of feeds over multiple connections
One of the advantages of ebt, other then not asking about feeds that didn't change, is receiving feeds from the best known source. For this we need a bit of heuristics, like ping to the peer and who has messages sooner.

Preliminary testing results: functional but a couple of glitches.

rebuilding of the own/self network frontier after changes on the replication lister (graph walk)
The TODO above about this acknowledged a slight variation of this but it's actually worse. Right now, it just builds the network frontier for ebt when it doesn't have one.. any (un)follows after the first ebt connection are without effect.
Followgraph walk finishes to soon
Tangential and slightly sure it was the case before these changes. The graph/builder walk along the follow messages sometimes collapses and returns to few results.
Untangle circular dependency between state matrix and graph builder (See PR #97)
Before the graph builder was hooked up after the indexing system.
The way it's setup now, the combined index needs the state matrix to update it with new received messages... which also needs the graph now for the first fill.
This circle needs to be broken so that the server setup doesn't race.

Data race

happening in ssb/sbot

From https://travis-ci.org/cryptoscope/ssb/jobs/535724332#L770

==================
WARNING: DATA RACE
Write at 0x00c0000a2531 by goroutine 10:
  runtime.slicecopy()
      /home/travis/.gimme/versions/go1.12.5.linux.amd64/src/runtime/slice.go:197 +0x0
  go.cryptoscope.co/margaret/multilog/badger.(*sublog).Get()
      /home/travis/gopath/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/margaret/multilog/badger/sublog.go:27 +0x1f9
  go.cryptoscope.co/ssb/multilogs.publishLog.Append()
      /home/travis/gopath/src/go.cryptoscope.co/ssb/multilogs/publish.go:70 +0x3af
  go.cryptoscope.co/ssb/multilogs.(*publishLog).Append()
      <autogenerated>:1 +0xfd
  go.cryptoscope.co/ssb/sbot.TestFeedsOneByOne()
      /home/travis/gopath/src/go.cryptoscope.co/ssb/sbot/feeds_test.go:89 +0x1005
  testing.tRunner()
      /home/travis/.gimme/versions/go1.12.5.linux.amd64/src/testing/testing.go:865 +0x163
Previous read at 0x00c0000a2530 by goroutine 156:
  runtime.slicecopy()
      /home/travis/.gimme/versions/go1.12.5.linux.amd64/src/runtime/slice.go:197 +0x0
  github.com/dgraph-io/badger/y.KeyWithTs()
      /home/travis/gopath/src/go.cryptoscope.co/ssb/vendor/github.com/dgraph-io/badger/y/y.go:106 +0xa2
  github.com/dgraph-io/badger.(*Txn).Get()
      /home/travis/gopath/src/go.cryptoscope.co/ssb/vendor/github.com/dgraph-io/badger/transaction.go:408 +0x104
  go.cryptoscope.co/margaret/multilog/badger.(*sublog).Get.func1()
      /home/travis/gopath/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/margaret/multilog/badger/sublog.go:30 +0x90
  github.com/dgraph-io/badger.(*DB).View()
    sting/testing.go:1072 +0x2eb
  main.main()
      _testmain.go:44 +0x222
Goroutine 156 (finished) created at:
  go.cryptoscope.co/muxrpc.(*HandlerMux).HandleConnect()
      /home/travis/gopath/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/muxrpc/handler.go:51 +0x1d3
  go.cryptoscope.co/muxrpc.handle.func2()
      /home/travis/gopath/src/go.cryptoscope.co/ssb/vendor/go.cryptoscope.co/muxrpc/rpc.go:82 +0x68
==================

Unable to run on 32bit Raspberry Pi

I followed the instructions to compile on

Raspberry Pi 4
Raspbian Lite, 32bit OS
Go 1.14
Compiled on the pi itself

When I attempt to run go-sbot as follows

$GOPATH/bin/go-sbot

I get the following error. Any ideas? I am new to scuttlebutt, so please forgive stupid mistakes.

t=612.141µs starting=metrics addr=localhost:6078
2021/02/07 11:50:19 dropping empty lockflile /home/pi/.ssb-go/indexes/groups/keys/mkv/.11eaa770cd0c84e0bf7a8f6c7e51040b6ea1ac4e
2021/02/07 11:50:19 dropping empty lockflile /home/pi/.ssb-go/indexes/groups/keys/mkv/.342aaf809784f0cdd40211c942c053729a833a0a
go-sbot: failed to instantiate ssb server: sbot: failed to open sublog for add-member messages: mlog/badger: badger failed to open: Map log file. Path=/home/pi/.ssb-go/sublogs/group-member-helper/badger/000000.vlog. Error=cannot allocate memory

Decoding 9 bytes of bad CBOR data can exhaust memory (message/multimsg/multimsg.go)

Decoding 9-10 bytes of malformed CBOR data can cause "fatal error: out of memory" and "runtime: out of memory" errors. Only 1 decode attempt is required to cause the error.

cc @cryptix

Relevant Code

message/multimsg/multimsg.go

import (
...	
	"github.com/ugorji/go/codec"
...
)
...
func (mm *MultiMessage) UnmarshalBinary(data []byte) error {
...
	var mh codec.CborHandle
	mh.StructToArray = true
	dec := codec.NewDecoderBytes(data[1:], &mh)
...
	switch mm.tipe {
	case Legacy:
		var msg legacy.StoredMessage
		err := dec.Decode(&msg)
...
	case Gabby:
		var msg ggWithMetadata
		err := dec.Decode(&msg)

🔥 Error (fatal error: out of memory)

For info about CBOR and security, see Section 8 of RFC 7049 (Security Considerations).

For more comparisons, see fxamacker/cbor.

Description and Minimal Reproduction

In October 2013, RFC 7049 Section 8 (CBOR Security Considerations) warned that malformed CBOR data can be used to exhaust system resources.

Resource exhaustion attacks might attempt to lure a decoder into
allocating very big data items (strings, arrays, maps) or exhaust the
stack depth by setting up deeply nested items. Decoders need to have
appropriate resource management to mitigate these attacks.

In September 2019, oasislabs/oasis-core discovered out of memory errors can be caused by tiny CBOR data and traced the problem to ugorji/go (same CBOR library used by cryptoscope/ssb). They fixed the problem by switching to a more secure CBOR library.

In February 2020, smartcontractkit/chainlink reported an issue on pivitoltracker (I don't have login access) and fixed it in a GitHub PR titled "Switch to more secure CBOR library". They were also using same CBOR library as cryptoscope/ssb.

To reproduce, decode 9-byte or 10-byte malformed CBOR data described in Section 8 of RFC 7049.

Examples of malformed CBOR data to achieve this can be found on GitHub since September 2019 (or possibly earlier if you look beyond Go projects).

sbotc <-> go-sbot bad dialog report

When executing this commands, it seems some index searching are missing?

sbotc backlinks.read '{"query":[{"$filter":{"dest":"#zenytv","value":{"content":{"type":"post"}},"timestamp":{"$gt":'"1599682818301"'}}}]}'
{"name":"Error","message":"no such command: backlinks.read","stack":""}

self_name=$(sbotc query.read '{"query":[{"$filter":{"value":{"author": "'"$self"'", "content":{"type":"about", "about": "'"$self"'"}}}}]}' | jq -r .value?.content?.name | grep -v null | tail -n 1)
{"name":"Error","message":"no such command: query.read","stack":""}

Is it normal?
Is there a CLI commands listing?

thanks

Properly handle use of Port 0

When port 0 is specified as a listening port, it requests the OS for a random eligible port. This may be good for multi-user setups.

Port 0 is incorrectly handled because:

when passed as ':0", a random port is listened on, but the advertised port is the default '8008'.
when passed w/ a listening IP address, the service incorrectly advertises port '0', i.e. "net:192.168.0.3:0~shs:some-key="

How to convert between curves with x/crypto

I have a feeling #43 will be with us for a while. Opening this issue to collect things.

See also ssbc/go-secretstream#15 for more.

is go-ssb data store compatibile with oasis or other ssb clients?

in the past with debugging the node ssb server,
I would use oasis to take a look at what posts I could see,
make posts,
see my profile,
by pointing oasis to use the same ~/.ssb folder as the pub I was using

is this possible with go-ssb or definitely out of scope?

not a big deal if not possible, but just wondering

or alternatively, would be nice to create some documentation on the best way to test that things are working,
and I'd be happy to help with this

Launching go-sbot.exe on windows fails due to permission error on secret file

When launching go-sbot.exe on Windows (ARMv7 32bits binaries) the application exits complaining about secret having the wrong permissions:

PS C:\Users\andre\ssbc\go-ssb\cmd\go-sbot> .\go-sbot.exe
time=2020-05-13T17:47:19.5137291+01:00 caller=metrics.go:87 module=sbot starting=metrics addr=localhost:6078
time=2020-05-13T17:47:19.5157276+01:00 caller=log.go:28 check=fatal err="sbot: failed to get keypair: repo: error opening key pair: ssb.LoadKeyPair: expected key file permissions -rw-------, but got -rw-rw-rw-"
time=2020-05-13T17:47:19.5307296+01:00 caller=recovery.go:63 event=panic location=CheckFatal panicLog=panics\CheckFatal939319243
panicWithStack: wrote panics\CheckFatal939319243
PS C:\Users\andre\ssbc\go-ssb\cmd\go-sbot>

This .ssb-go was created by the same application on its first run. This happens after the first run every time you run it. I suspect that whatever code is used to create secret is passing the wrong perms on windows.

sync: off-by-three

whoops.. that's weird. There need to be more than 2 new messages otherwise the source just plays empty.

--- FAIL: TestFeedsOneByOne (1.16s)
    feeds_test.go:84: runniung connect 0
    logger.go:31: test=bob plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    feeds_test.go:97:
        	Error Trace:	feeds_test.go:97
        	Error:      	Not equal:
        	            	expected: 0
        	            	actual  : -1
        	Test:       	TestFeedsOneByOne
        	Messages:   	check run 0
    feeds_test.go:84: runniung connect 1
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=bob plugin=gossip msg="fetchHops done" hops=2 stored=1
    feeds_test.go:97:
        	Error Trace:	feeds_test.go:97
        	Error:      	Not equal:
        	            	expected: 1
        	            	actual  : -1
        	Test:       	TestFeedsOneByOne
        	Messages:   	check run 1
    feeds_test.go:84: runniung connect 2
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=ali plugin=gossip/hist event=gossiptx n=3
    logger.go:31: test=bob plugin=gossip fr="@34wNGCIqZEi61rtkJzsrWGoczIvW1pbj2DKHZmLnYQs=.ed25519" latest=0 event=gossiprx new=3 took=3.947179ms
    logger.go:31: test=bob plugin=gossip msg="fetchHops done" hops=2 stored=1
    feeds_test.go:84: runniung connect 3
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=bob plugin=gossip msg="fetchHops done" hops=2 stored=2
    feeds_test.go:97:
        	Error Trace:	feeds_test.go:97
        	Error:      	Not equal:
        	            	expected: 3
        	            	actual  : 2
        	Test:       	TestFeedsOneByOne
        	Messages:   	check run 3
    feeds_test.go:84: runniung connect 4
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=bob plugin=gossip msg="fetchHops done" hops=2 stored=2
    feeds_test.go:97:
        	Error Trace:	feeds_test.go:97
        	Error:      	Not equal:
        	            	expected: 4
        	            	actual  : 2
        	Test:       	TestFeedsOneByOne
        	Messages:   	check run 4
    feeds_test.go:84: runniung connect 5
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=ali plugin=gossip/hist event=gossiptx n=3
    logger.go:31: test=bob plugin=gossip fr="@34wNGCIqZEi61rtkJzsrWGoczIvW1pbj2DKHZmLnYQs=.ed25519" latest=3 event=gossiprx new=3 took=3.767502ms
    logger.go:31: test=bob plugin=gossip msg="fetchHops done" hops=2 stored=2
    feeds_test.go:84: runniung connect 6
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=bob plugin=gossip msg="fetchHops done" hops=2 stored=2
    feeds_test.go:97:
        	Error Trace:	feeds_test.go:97
        	Error:      	Not equal:
        	            	expected: 6
        	            	actual  : 5
        	Test:       	TestFeedsOneByOne
        	Messages:   	check run 6
    feeds_test.go:84: runniung connect 7
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=bob plugin=gossip msg="fetchHops done" hops=2 stored=2
    feeds_test.go:97:
        	Error Trace:	feeds_test.go:97
        	Error:      	Not equal:
        	            	expected: 7
        	            	actual  : 5
        	Test:       	TestFeedsOneByOne
        	Messages:   	check run 7
    feeds_test.go:84: runniung connect 8
    logger.go:31: test=ali plugin=gossip msg="fetchHops done" hops=2 stored=1
    logger.go:31: test=ali plugin=gossip/hist event=gossiptx n=3
    logger.go:31: test=bob plugin=gossip fr="@34wNGCIqZEi61rtkJzsrWGoczIvW1pbj2DKHZmLnYQs=.ed25519" latest=6 event=gossiprx new=3 took=3.202215ms

Runtime error: out of memory (legacy replication on arm64 Pi 3B+)

Context

I'm running go-sbot on my Pi 3B+ with 64-bit Debian. The SSB account currently follows 3 accounts and is followed by 1. The sbot is running with default configuration.

Error

The sbot is stable upon start-up. However, once I connect to an instance of Patchwork on my laptop (sbotcli connect ...) the memory usage quickly climbs and the sbot crashes after approximately 60 seconds: fatal error: runtime: out of memory. It spits out approximately 3,000 lines of stack trace :)

I'm not familiar enough with Go to give great analysis of the error output but it seems that the final code reference is: /go-ssb/plugins/gossip/fetch.go:61.

I'd be happy to provide the stack-trace if that's helpful.

The sbot is far more stable when I run it with EBT (-enable-ebt true).

muxrpc CallError: Error - no such command: gossip.connect

sbotcli connect "net:scuttle.space:8008:~shs:skBzPazHliOXCWLwloGvHYki0wPLOUeJpvW10U7MOJ4="
level=error run-failure="connect: async call failed.: error reading response from request source: muxrpc: error reading from packet source: muxrpc CallError: Error - no such command: gossip.connect"

This can happen because of a fallback to work around a missing feature in go-muxrpc, namely individual handler authorization based on the keypair of the session. That’s why there are two muxers in the sbot package (public and master), like with ssb-server you only get elevated access to master if you have the same keypair as the running server.

Therefore, you can't have special privileged gossip.connect if you have other handlers on gossip.* already, like ping. That’s why I introduced ctrl.* for privileged calls.

Blobs is a better example: you have blobs.add and blobs.rm but also blobs.has and blobs.get. You don't want add and rm exposed to the public usually.

To make this less obnoxious to deal with sbotcli connect $msAddr tries gossip if ctrl fails.

If you want to be sure AND you know what you need use sbotcli call ctrl.connect $msAddr for a Go server and sbotcli call gossipconnect $msAddr against a JS ssb-server.

Where to set the functional options API for sbot?

Thanks to your great docs over at dev.scuttlebutt.nz, I know that it's possible to alter the functionality of the sbot server using the options API (for example, in order to define a custom database mountpoint). However, I'm not sure how to do this in practice.

Based on the test code in ssb/cmd/sbotcli/simple_test.go, my current assumption is that the options need to be defined in ssb/cmd/sbotcli/main.go prior to compilation. Is that correct? I'd be grateful for a short explanation of how to achieve this (for example, where would I inject those definitions? In func initClient() or elsewhere?).

Thanks in advance ☺️

Double follow on redeeming an invite by an already followed feed

To reproduce:

create new identity;
start server fresh;
follow feed A;
generate an invite with sbotcli call invite.create;
send that invite to the owner of feed A, let it be redeemed.
see two follow messages on your feed, feed A is being followed twice.

state of peer invites

plugin currently disabled
only implemented server/"greeting a guest" role
needs create invite and client/redeem

Epic: Only use *testing.T in the testing goroutine

yea sounds good but let's make this a meta-todo for all the repos then.

Ahh, I see. Yeah I think we should use an chan error for that, but I see why you did it how you did. Not sure what the best way is - maybe let everything run in goroutines in the test goroutine just read the error chans that all the goroutines write to?

#1 (comment)

Missing: Outgoing connection sheduler

I totally punted on this for now, mostly because go-sbot was intended as a pub for the most time where gossiping happens naturally with the incoming connections.

My other fear is that I saw how messy this can get on the JS side. One rewrite of ssb-gossip went no-where and the 2nd rewrite (ssb-conn) seems to offer more readable code but still lacks behind in features and reliability.

For local client experiments I have been using this as a stop gap:

cat > /tmp/connect.hosts << EOF
net:cryptbox.mindeco.de:8008~shs:xxZeuBUKIXYZZ7y+CIHUuy0NOFXz2MhUBkHemr86S3M=
net:cryptoscope.co:8008~shs:G20wv2IZkB7BA/LKe7WMuByop3++J0u9+Y32OoEVOj8=
net:one.butt.nz:8008~shs:VJM7w1W19ZsKmG2KnfaoKIM66BRoreEkzaVm/J//wl8=
net:pub.t4l3.net:8008~shs:WndnBREUvtFVF14XYEq01icpt91753bA+nVycEJIAX4=
EOF

cat /tmp/connect.hosts | while read maddr
do
  sbotcli connect $maddr
  sleep 600
done

My third hesitation came from the fact that it's very hard to gauge network errors on mobile clients. Is a connection failing because you just went into a tunnel or because the pub server is unreliable?
For this reason planetary also schedules it's connections just by calling sbot.Network.Connect(...) without any internal helper code from go-ssb.

I still think it would be good to have a variation of that bash script as an included module that can be switched out, similar to the ssb.ConnTracker interface. A very simple first draft would offer adding favorite pubs like my bash script and then dial them in random once it doesn't have enough connections.

Sbot error: broken bitmap file

I'm breaking things over here 😆

go-sbot: failed to instantiate ssb server: sbot: failed compress multilog userFeeds: failed to load all sublogs: roaringfiles: broken bitmap file (key:00002d02007fdfbe8b8d86a7d3f72182e364d4a8bbcb44554222ea406b7dfdaec29a): roaringfiles: unpack of 00002d02007fdfbe8b8d86a7d3f72182e364d4a8bbcb44554222ea406b7dfdaec29a failed: error in roaringArray.readFrom: could not read initial cookie: EOF

I believe this error occurred after I published a post with this command:

sbotcli publish post "[@glyph](@HEqy940T6uB+T+d9Jaa58aNfRzLx9eRWqkZljBmnkmk=.ed25519) 聞こえますか?"

sbotcli --version
alpha4 (rev: snapshot, built: )

go-sbot -version
t=30.312µs version=snapshot build=

Unable to redeem an invite via sbotcli

$ sbotcli call invite.use '{"feed":"scuttle.space:8008:@skBzPazHliOXCWLwloGvHYki0wPLOUeJpvW10U7MOJ4=.ed25519~7kdhRVy9+tfcZC9G6TX58U5BTMIcR0/rN3j3rArKKLI="}'
level=error run-failure="invite.use: call failed.: error reading response from request source: muxrpc: error reading from packet source: muxrpc CallError: Error - unknown method"

Argument format deduced from:

https://github.com/cryptoscope/ssb/blob/a70154f9edc0532f1662f581971896f1dfcc5219/plugins/legacyinvites/guest.go#L29-L31

Well, but it doesn't matter, because it doesn't see the invite.use method.

build failed due to bugs in dependencies

┬─[rabble@r-mbp:~/g/s/g/s/c/go-sbot]─[18:15:15]─[G:master=]
╰─>$ go build -v -i
go.cryptoscope.co/ssb/internal/ctxutils
go.cryptoscope.co/ssb/message
go.cryptoscope.co/ssb/internal/mutil
go.cryptoscope.co/ssb/internal/multiserver
go.cryptoscope.co/ssb/plugins/whoami
go.cryptoscope.co/ssb/plugins/publish
go.cryptoscope.co/ssb/plugins/control
go.cryptoscope.co/ssb/blobstore
go.cryptoscope.co/margaret/multilog/badger
go.cryptoscope.co/ssb/network
go.cryptoscope.co/ssb/graph
go.cryptoscope.co/ssb/private
go.cryptoscope.co/ssb/internal/transform

go.cryptoscope.co/margaret/multilog/badger

../../../margaret/multilog/badger/qry.go:111:13: assignment mismatch: 2 variables but item.Value returns 1 values
../../../margaret/multilog/badger/qry.go:111:26: not enough arguments in call to item.Value
have ()
want (func([]byte) error)
../../../margaret/multilog/badger/sublog.go:36:13: assignment mismatch: 2 variables but item.Value returns 1 values
../../../margaret/multilog/badger/sublog.go:36:26: not enough arguments in call to item.Value
have ()
want (func([]byte) error)

go.cryptoscope.co/ssb/network

../../network/new.go:212:36: undefined: secrethandshake.ErrProtocol
../../network/new.go:234:9: undefined: secrethandshake.ErrProtocol
go.cryptoscope.co/ssb/plugins/rawread

go.cryptoscope.co/ssb/graph

../../graph/builder.go:59:7: undefined: margaret.IsErrNulled
../../graph/builder.go:150:11: assignment mismatch: 2 variables but it.Value returns 1 values
../../graph/builder.go:150:22: not enough arguments in call to it.Value
have ()
want (func([]byte) error)
../../graph/builder.go:238:11: assignment mismatch: 2 variables but it.Value returns 1 values
../../graph/builder.go:238:22: not enough arguments in call to it.Value
have ()
want (func([]byte) error)
go.cryptoscope.co/ssb/plugins/private
go.cryptoscope.co/ssb/plugins/blobs

unix-path based client init failed: ssbClient: failed to open unix path \"/home/peach-go-sbot/.ssb-go/socket\"

just started seeing this error when I run sbotcli call whoami

level=error run-failure="unix-path based client init failed: ssbClient: failed to open unix path \"/home/peach-go-sbot/.ssb-go/socket\""

any ideas?