Hello! There is no reference in the docs for using ./lspd genkey other than the dockerfile you provide, no info is here and newcomers like me might struggle to set LSPD in the future.

BR,

Alice -> LSP -> Bob

• LSP -> Bob don't have a channel yet
  • Bob registered the payment hash
    • Alice pays with 1 HTLC
      • Success: Channel is opened, capacity amount + 100k sats. HTLC succeeds.
      • Failure: Not enough onchain balance: HTLC failed with temporary node failure
      • Failure: Bob is offline. HTLC failed with unknown next peer
        • (Should we open a channel anyway??)
    • Alice pays with multiple HTLCs
      • Amount < 100k sats
        • Success: 1 Channel is opened, capacity single HTLC + 100k sats. All HTLCs succeed.
      • Amount > HTLC size + 100k sats
        • Success: ??? (Multiple channels opened?)
  • Bob did not register the payment hash
    • HTLC is rejected with unknown next peer
• LSP -> Bob do have a channel
  • Bob registered the payment hash
    • Amount <= local balance
      • Success: HTLC succeeds. No additional channel is opened.
    • Amount > local balance
      • Success: Additional channel is opened. HTLC succeeds. (What's the channel size?)
    • Amount of the second HTLC > local balance
      • Success: 1 Additional channel is opened. All HTLCs succeed.
  • Bob did not register the payment hash
    • Amount <= local balance
      • Success: HTLC succeeds.
    • Amount > local balance
      • HTLC fails with temporary channel failure.

https://github.com/BitcoinAndLightningLayerSpecs/lsp/tree/main/LSPS2#1-api-information

Note that there is a difference in the way the opening fee is computed: https://github.com/BitcoinAndLightningLayerSpecs/lsp/tree/main/LSPS2#computing-the-opening_fee

do we require a new database for cln nodes?

Make lspd run standalone. The CLN lspd currently runs as a CLN plugin.

• The channelinformation call should return the right node id / LSP id based on the request.
• Extract the CLN plugin into a small wrapper.
• Hook into the CLN wrapper from the lspd process.
• Make lspd run as a systemd service.
• Handle clean shutdown.

The proposed JIT channel spec exposes an endpoint GET /lsp/jitchannel, which is the entrypoint for clients to request a JIT channel.

• versions will be [1].
• opening_fee_params and payment size/channel size ranges will be configurable.
• promise will be a hex-encoded sha256 hash of the serialized opening fee params + a persisted secret.

The proposed JIT channel spec exposes an endpoint GET /lsp/jitchannel, which reserves a SCID for the client, which it can use in a routing hint to have a JIT channel opened by the LSP.

• if version is not 1: error
• if the promise is not valid: error
• if valid_until expired: error
• if payment_size falls out of range: error
• if channel_size falls out of range: error
• if channel size * 99% > payment size: error
• reserve a unique SCID for this client until the expiration period

Note that the SCID has to be persisted along with the client request parameters.

Depends on promise generation defined in #70
Depends on SCID generation defined in #72

We've been using https://github.com/niftynei/glightning for our communication with the cln node. There's a better supported version at https://github.com/elementsproject/glightning we could use. Another option is to use the grpc api (but that has many breaking changes, so that's probably not a good idea yet).

Add a standalone framework for regression testing lspd.

Tasks:

• Spin up bitcoind node
• Spin up 2 core lightning nodes and open a channel between them.
• Add lspd node to the network
• Add first regression test (open zero conf channel when receiving payment without inbound liquidity)

Add a document that clearly describes all steps to successfully run lspd.

When the cln lspd has a registered payment with a peer that it does not have a channel with yet, and that payment is paid with a multipart payment, cln lspd will attempt to open a channel per htlc.

solution: use a mutex to lock payment channel creation to a payment hash, so no duplicate channels are opened.

1. Pass the apikey to the tag field (from the server)
2. Store it in the channel table
3. Add this field to the spec

https://github.com/BitcoinAndLightningLayerSpecs/lsp/tree/main/LSPS2#2--request-jit-channel

When the cln lspd stops while a htlc is being handled, there is a gap.

• Either lspd will keep receiving new htlcs and is also able to send.
• Or lspd will stop receiving htlcs but is also unable to send.

So in either case there could be htlcs unhandled.
Solutions could be:

• Replay unhandled htlcs that were already sent from the cln plugin when the lspd drops out.
• Make a different stream for receiving htlc_accepted and the htlc resolutions. This way the receiving context could be cancelled first, then the remaining htlcs could be sent out and then the sending context can be cancelled.

• Merging lnd-15 to master
• Merging master to greenlight branch
• Share code between cln and lnd interception.

test LND testOfflineNotificationZeroConfChannel

goroutine 287 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/debug/stack.go:16 +0x19
github.com/breez/lntest.CheckError(0xc0001ca340, {0x1133ee0, 0xc000014a80})
	/home/runner/go/pkg/mod/github.com/breez/[email protected]/test_common.go:44 +0x39
github.com/breez/lntest.(*LndNode).ConnectPeer(0xc0000eca00, {0x1146318, 0xc0001e7e00})
	/home/runner/go/pkg/mod/github.com/breez/[email protected]/lnd_node.go:421 +0x285
github.com/breez/lspd/itest.testOfflineNotificationZeroConfChannel.func2()
	/home/runner/work/lspd/lspd/itest/notification_test.go:285 +0xaa
created by github.com/breez/lspd/itest.testOfflineNotificationZeroConfChannel
	/home/runner/work/lspd/lspd/itest/notification_test.go:281 +0xd4a
    test_common.go:45: rpc error: code = Unknown desc = already connected to peer: 038e10c282da1febc7aac2ef6e0836ebde537594c40fc0125097217dd773d013d5@127.0.0.1:45975

Make sure the cln lsp works with v22.11 (it's currently built on v0.12.1)

Currently the LND version of lspd constructs a new onion in order to be able to forward htlcs with the lspd fees deducted after a channel open. We really only need to modify the next hop id record and lower the amount to forward record in the tlv payload and keep the original onion intact.

This needs some work in order to be able to start:

• LND needs to support modifying (only) the onion payload
• The breez client needs to accept the modified onion payload.
• Need a mechanism to distinguish clients that support the modified onion payload and those that don't, in order to ensure backwards compatibility.

• Clean up created docker volume after test
• Clean up also if the test if interrupted (ctrl + c)

The interruption seems to be hard to handle. If you use signal.Notify to intercept the stop signal, the tests do not stop and t.Fatal no longer works.

https://github.com/BitcoinAndLightningLayerSpecs/lsp/tree/main/LSPS2#5--channel-opening-and-forwarding

The lspd process is not cleaned up by the integration tests, because it doesn't exit. It appears to be stuck in the grpc 'Recv'. This should be cancelled when interceptor.Stop is called. It appears interceptor.Stop is not called, because the stop signal is not intercepted properly.

Tracking issue for the complete status of LSPS2 implementation.

The initial implementation will be built on top of CLN.
Initial implementation issues:

• #104
• #106
• #107
• #108
• #121
• #109
• #126
• #159
• #127
• #128

https://github.com/BitcoinAndLightningLayerSpecs/lsp/blob/main/LSPS0/README.md

https://github.com/BitcoinAndLightningLayerSpecs/lsp/tree/main/LSPS2#5--channel-opening-and-forwarding

checkPayment() uses incomingAmountMsat to calculate fee.
breez calculate fee using user requested amount.
So I thought that the lspd fee check would use outgoingAmountMsat instead of incomingAmountMsat.

                       +------+                       +-------+
--incomingAmountMsat-->| lspd |--outgoingAmountMsat-->| breez |
                       +------+                       +-------+


lspd_fee = incomingAmountMsat - outgoingAmountMsat?

This feature set the basis of the ability to receive funds when the user node is offline.
Event though lots of the end users nodes are operated from mobile, we can't use mobile notifications directly as there are multiple apps involved and the LSP doesn't have the certificates

An optional flow when there are already channels:

1. LSP receive htlc to forward.
2. LSP detects the channel to forward is not active.
3. LSP notifies the webhooks associated with this channel.
4. LSP waits a certain time for the channel to be active, then forward the htlcs.

An optional flow when there is no channel:

1. LSP receive htlc to forward.
2. LSP needs to open a channel but the peer is offline.
3. LSP fetch the webhook associated with the registered invoice and notifies it.
4. LSP waits a certain time for the channel to be active, then forward the htlcs.

https://github.com/BitcoinAndLightningLayerSpecs/lsp/tree/main/LSPS2#0-api-version

In the proposed JIT channel spec a client gets assigned a unique SCID that is used to identify a payment as a JIT channel request.

There's a few options:

1. Keep track of all SCIDs of open and pending channels and of pending JIT channel requests and make sure the new SCID does not yet exist.
2. Reserve a range of numbers for JIT channel requests, so only JIT channel requests can have a SCID in that range. That way there's only JIT channel SCIDs to track.

Either way, the SCIDs have to be globally unique throughout the node. They have to be persisted on creation. So upon generating one, it has to be reserved until the expiration time.

The cln lsp should open an anchor channel

If CLN is restarted, but lspd is not, the glightning client gets shut down. After that, calls to the RPC will fail with Client is shutdown, even aftre CLN is back up.
Create a mechanism to restart the CLN client, so we can reconnect to the CLN rpc when it becomes available again and lspd does not need to be restarted when CLN is restarted.

Error during test https://github.com/breez/lspd/actions/runs/5782332133/job/15669005279 (first run)

goroutine 21 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	/opt/hostedtoolcache/go/1.20.6/x64/src/runtime/debug/stack.go:16 +0x19
github.com/breez/lntest.CheckError(0xc0004e2340, {0x1133ee0, 0xc00011ad08})
	/home/runner/go/pkg/mod/github.com/breez/[email protected]/test_common.go:44 +0x39
github.com/breez/lntest.(*ClnNode).GetChannels(0xc0000aa300)
	/home/runner/go/pkg/mod/github.com/breez/[email protected]/cln_node.go:584 +0x9f
github.com/breez/lspd/itest.testDynamicFeeFlow(0xc0000af680)
	/home/runner/work/lspd/lspd/itest/dynamic_fee_test.go:137 +0x1259
github.com/breez/lspd/itest.runTest(0xc0004e2340, 0x1725740, {0xf345ae?, 0x0?}, 0x1056520, 0x1056518)
	/home/runner/work/lspd/lspd/itest/lspd_test.go:89 +0x422
github.com/breez/lspd/itest.runTests.func1(0x0?)
	/home/runner/work/lspd/lspd/itest/lspd_test.go:47 +0x2d
testing.tRunner(0xc0004e2340, 0xc0000adb00)
	/opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1576 +0x10b
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.20.6/x64/src/testing/testing.go:1629 +0x3ea
    test_common.go:45: rpc error: code = Unknown desc = Error calling method ListPeerChannels: RpcError { code: None, message: "Malformed response from lightningd: unknown variant `RCVD_REMOVE_REVOCATION`, expected one of `SENT_ADD_HTLC`, `SENT_ADD_COMMIT`, `RCVD_ADD_REVOCATION`, `RCVD_ADD_ACK_COMMIT`, `SENT_ADD_ACK_REVOCATION`, `RCVD_ADD_ACK_REVOCATION`, `RCVD_REMOVE_HTLC`, `RCVD_REMOVE_COMMIT`, `SENT_REMOVE_REVOCATION`, `SENT_REMOVE_ACK_COMMIT`, `RCVD_REMOVE_ACK_REVOCATION`" }

The ListPeers function will no longer return the peer's channels after cln v23.11. That will break this function call:

Options:

• Add ListPeerChannels to our fork glightning
• Move to using the supported version of glightning https://github.com/elementsProject/glightning/
• Move to using grpc instead (that's probably prone to more breaking changes than glightning.

The CLN lsp should open a channel with zero reserve.

Taken from #63 (comment)

• maintain a log of published notifications (for diagnostics/support)
• mechanism to remove webhook subscriptions by expiry and/or when the webhook call fails
• retry mechanism is useful when interacting with third party services (so webhook data needs an identifier for deduplication?) could be useful to send an initial 'subscribed' notification when ready.
• the time when waiting for the channel to be active is interesting, especially regarding restarts of lspd/the underlying lightning node.

Identify forwards from the proposed JIT channel spec based on their SCID. If the SCID is persisted as a JIT channel request, fall into a channel flow that should be newly created as implementation of this issue.

• aggregate htlcs until all parts sum up to at least payment_size plus opening_fee
  • point of caution, think about restarts of lspd and/or the lightning node. Should be able to handle both replays of the same htlc when the node is restarted and missing replays of htlcs when lspd is restarted.
• Make sure htlc timeouts are handled appropriately, either by the node itself of by lspd explicitly. In either case, lspd will have to know about htlc expiry.
• keep track of when htlcs 'expire' through expiry of valid_until of the opening parameters.
• Make a decision on what to do with too many payment parts. Probably want to just take the excess amount?
• When there are enough payment parts, make sure the client is connected and valid_until has not expired.
• open the channel, when channel_ready is received, forward the htlcs with lowered amounts defined in the provided algorithm.
• Probably we won't do anything custom with the channel alias. We'll let the node specify a new channel alias for the channel open.
• Caution with htlcs that arrive after the decision to open the channel, those need to be handled as well.

Currently there are a few lookups

• Check whether peer is online
• Lookup peer id by scid/short channel id
• Check whether channel is active

These checks are now inefficient, because they iterate basically every channel. And they are used on almost every htlc.
Build a quick cached lookup for more efficiency.

https://github.com/BitcoinAndLightningLayerSpecs/lsp/blob/main/LSPS0/README.md

Intro

LSPD allows clients to register a payment with the LSP, so that a just-in-time channel will be opened to the client node on the fly at the moment the payment is forwarded. The payment is identified by the payment hash.

When the client creates an invoice that needs an inbound channel, the flow is the following. The client creates an invoice in its node of amount (to receive - lsp fee). Then it creates a new invoice outside the node of the actual amount to receive, and with a hop hint to the LSP with a fake channel id (1x0x0). The sender pays this invoice, and routes over the LSP.

When a htlc for the given payment hash is intercepted by the LSP, a channel is opened to the client. In order to then forward the HTLC, a few things are modified.

1. The LSP takes a fee for opening the channel, so the forwarded amount is deducted.
2. The channel id has to be adjusted to the newly created channel alias.

Current situation

Currently, in the LND implementation for lspd, we create a whole new onion. Any information from the sender is basically dropped. The client shared the payment secret with lspd when it registered the payment, that's why this works. The client doesn't even know it received a new onion from the LSP. There are some caveats to this, however.

1. The client has to share the payment secret with the LSP (that's not the preimage), so it cannot verify the amount received was the intented amount by the sender.
2. These new onion packets are encrypted with a new session key. That means if the client returns an error, the sender is unable to read this error, because it's encrypted with an unknown key.
3. Any additional metadata from the sender is lost.

Currently, that new onion is created here

New situation

We want to change this to include the original onion from the sender and only modify the amount and channel id in the htlc for the next hop. We already do this in the CLN implementation of lspd and this mechanism is also part of the lsp spec, so this would get us closer to implementing the spec.

What to change

In order to do this we will need a few things:

• Modify our fork of LND for the LSP, so the htlc interceptor accepts a different amount + channel id, in such a way that the original onion and tlvs are kept intact, but only the amount and channel id will be different in the update_add_htlc message.
• Modify our fork of LND for the client so that it accepts the lower amount. It's mainly the difference between the forwarded amount and the total_msat provided by the sender that causes a mismatch.
• Modify lspd, so it uses the newly created flow.

Keep in mind that some clients will not support the new flow yet, so we also have to support the old flow for old clients.

CLN reference

The implementation we currently already have for CLN is here. When the htlc is intercepted in CLN, we modify the amount and channel id in the payload for the current hop, so the payload for the LSP itself. When that's sent back to CLN, CLN takes that new payload to continue its flow. So it will eventually construct an update_add_htlc message for the client with the lower amount and different channel id.

On the client side the CLN client needs a htlc interceptor, the total_msat is modified in the payload to the amount in the invoice. And the amount is set to whatever was received in the HTLC. There's a naive implementation in the tests here and a more stable rust implementation here.

I'm not sure whether this kind of mechanism is reusable in LND or not, or whether we have to come up with something new.