Hi guys, thank you for this awesome project.

Do you plan to release NATS Streaming Server to Docker Hub anytime soon?

Remove "STAN:" from the individual log lines to instead prefix in the logger.

clientID should be command-line, URL, filesystem, and JSON-friendly (i.e. not containing special characters that might interfere with using or parsing a clientID in any of these)

Limits are checked per file (out of the 5) but not based on the total for this store.

Getting the following from my travis build for stan-java-client. Is the master branch of stan-server out of sync with the master branch of gnatsd?

$ go get github.com/nats-io/stan-server
# github.com/nats-io/stan-server
../../../gopath/src/github.com/nats-io/stan-server/stan-server.go:93: natsOpts.BufSize undefined (type "github.com/nats-io/gnatsd/server".Options has no field or method BufSize)


The command "go get github.com/nats-io/stan-server" failed and exited with 2 during .

Your build has been stopped.

As of now, we persist the state only through subscriptions.

The problem is if an application "connects" and before creating its first subscription, the server is restarted, there would be no client state restored, which would cause other actions to fail (create subscriber, etc).

Same goes for STAN applications that would be only publishing: the server would not be able to check the client health through heartbeats since it would not know about those clients.

If a durable starts, then stops, then the server is restarted, the durable will no longer receive messages.

This is introduced by #58.

Some additional tuning parameters need to be added:

• Maximum number of channels.
• Maximum number of messages.
• Maximum memory/messages per channel.

This is directly analogous to TimeDeltaStart vs. StartTime. The user doesn't know (or care) what the current sequence number is, but wants the last, say, 1000 messages in order to prime e.g. a moving-window time series display. Right now there is no way to do that.

Ideally the user could just enter a negative offset, but that won't work because StartSequence is defined as uint64. Therefore I propose we just add DeltaOffset or OffsetDelta as another option and do the math in the server.

From @ColinSullivan1 on March 15, 2016 21:18

When an application crashes, it may not call the STAN Close() function, so STAN will hold the application's clientID as if it were connected. This leaves STAN in a state where it will prevent a connection with the same ClientID, erroneously indicating it is a duplicate.

I propose we:
a) Add a disconnected alert/advisory from NATS that STAN can listen to. Upon a client disconnect, the NATS server publishes an advisory message containing the client name, perhaps in JSON format.
b) STAN uses the client ID as the NATS client name to connect.
c) STAN listens on the advisory subject and removes the client ID when notified a client has been disconnected.

Copied from original issue: nats-io/stan.go#30

From @mcqueary on March 18, 2016 19:14

We need a file store module.

Copied from original issue: nats-io/stan.go#40

Support user authentication - some basic authentication can be added today (user/pass).

We changed the HB and Timeout to something like 200ms and 2 seconds, respectively. By default, a NATS client reconnects after 2 seconds. So say that you stop/restart the NATS server (STAN server when NATS is embedded), then when STAN applications reconnect to NATS Server, it may be too late: the STAN server will have purged those clients because of lack of heartbeats.

This should also incorporate updating dependencies/imports, travis and coverall configurations

With some batterie powered disks, sync may not be required. Make it possible to disable through command line/options.

Introspection into the NATS streaming server via a monitoring interface is required. An interface similar to that of the NATS server should suffice.

Some discussion points:

• Which data points should be exposed, and how should they be presented e.g. /server, /stores, /clients?
• Can a pass through using he embedded NATS server port work, or should the NATS streaming server have it's own dedicated port?
• How will authorization/authentication look?

FileStore assigns a subscription ID when storing a subscription. However, on recovery, the subscription ID for new subscriptions don't take into account the ids of the subscriptions recovered.

Say a subscriber is running. It receives messages 1 to 100. At this point, the server (with file store) is restarted. On new message, old messages are redelivered. This does not seem to be right.

Add vendoring for the gogo protobuf packages.

From @mcqueary on March 15, 2016 19:14

See here and here in server.go.

As a user, if my SubscriptionRequest specifies a time 100 seconds in the past, but the earliest stored message was 80 seconds in the past, I get stan: invalid start time. IMO that's broken, conceptually.

Think of how you'd use a "since" date when qualifying Google search results (i.e. show me hits since this date). If I entered 1/1/2016, but the first result was from 1/3/2016, I'd expect to get everything starting from 1/3/2016, and not an error indicating that my "since" date was wrong. The same applies here: if my start time is before the earliest message, just start with the earliest message!

It's tempting to say "well, just use StartPosition.First instead", but I as a user actually require data since exactly (at most) 100 seconds ago, and I have no idea when "First" will start (3 days, 2 hours, or 5 seconds).

I propose that if start time (or sequence) is earlier than the first message, that we start with the first message, and likewise that if the start time (or sequence) is later than the last message, we set the start time/sequence to 1ns or 1 seqno after the last message (i.e. default to StartPosition.NewOnly).

Copied from original issue: nats-io/stan.go#24

When a server with FileStore with stored messages is restarted, all messages are marked as redelivered, regardless if they have ever been sent to any subscriber. This is not correct.

Only un-acknowledged messages previously sent to a subscriber should be marked as redelivered.

Test case:

• start server with FileStore
• send 1 msg to "foo"
• start sub on "foo" with DeliverAllAvailable() and SetManualAckMode(), don't ack the message.
• restart server
• start a new sub on "foo" with DeliverAllAvailable().
• message should be received by both subs as not-redelivered once, and the first sub should have receive the message again as redelivered.

Persistent, ordered and highly-available messages enable a great many client use cases (support for recovery from client down time, cluster healing etc).

There are additional use cases where being able to (minimally) filter/query historic messages and export the results to pre-defined format in non-realtime (or near real-time if possible). For example:

• audit trail / compliance reporting
• production debugging
• alternate outcome testing
• shovel to a data pipeline / analytics / optimizing projections

First thoughts are you should be able to filter by topic, date range and sequence range.

If the store contains no messages, cs.msgs.FirstMsg() and cs.msgs.LastMsg() return nil, ergo any subscription containing a time or time delta will crash the server:

https://github.com/nats-io/stan-server/blob/master/server/server.go#L1296-L1304 (edited for line number change)

A configuration files passed via the -config command line is not parsed, and should be in order to support all of the NATS server options. This will likely be required for multi-user authorization/authentication.

Need support for Fault Tolerance and High Availability

• Shared Storage Fault Tolerance (#263)

If a durable is stopped after the server sent up to MaxInFlight messages, when the durable is restarted, it does not receive the pending messages.

https://github.com/nats-io/stan-server/blob/master/server/server.go#L49-L51

If a subscriber hits the MaxInFlight limit with all un-acknowledged messages, the server attempts but does not send un-acknowledged messages. This may prevent an application that missed messages (as opposed to just delaying the acknowledgements) to receive them again.
Because of the same issue, a durable would never receive old/new messages after a server restart if the number of pending acks is >= MaxInFlight.

Suppose that you have 2 queue subscribers on "foo".
One message is sent, the first queue subscriber receiving it does not acknowledge it.
Message is redelivered to second queue subscriber which acknowledges it.
Server is restarted.
The message is going to be redelivered.

When a server with FileStore is restarted, and there are existing subscribers, messages in the log should be delivered automatically. Right now, new message need to be sent to trigger delivery.

The subscription store in the file store implementation overtime will contain information no longer needed. The file should be compacted automatically.

Users will want the NATS Streaming clients to connect using TLS. The embedded NATS server can receive TLS parameters through the NATS streaming server command line, but we need the NATS streaming server to use TLS with it's underlying connection to the NATS server.

To support this, TLS client side certificates need to be provided to the streaming server, which can be achieved through using command line parameters.

Create a durable "mydur"
Stop the application
Restart the application: OK
Stop the application
Restart the application: fails with duplicate durable.

Say that you set some store limits, for instance 1 channel max. Publishing on "foo" works fine. Publishing on "bar should report an error (too many channels). This is not happening.

The reason is that the server acks the publisher before trying to persist the message (where limits are checked).

This is because of tracing, which should be moved in performRedelivery so that sub fields are accessed under lock. At the same time, I'd like to revisit performDurableRedelivery so that it has its own delivery and not use common code in performRedelivery which deals with potentially expiration and queue subscribers. Although it would cause some code duplication, it will be much clearer.

==================
WARNING: DATA RACE
Read by goroutine 48:
  runtime.convT2E()
      /usr/local/go/src/runtime/iface.go:128 +0x0
  github.com/nats-io/stan-server/server.(*StanServer).performAckExpirationRedelivery()
      /Users/ivan/dev/go/src/github.com/nats-io/stan-server/server/server.go:825 +0x95
  github.com/nats-io/stan-server/server.(*StanServer).sendMsgToSub.func1()
      /Users/ivan/dev/go/src/github.com/nats-io/stan-server/server/server.go:975 +0x36

Previous write by goroutine 25:
  github.com/nats-io/stan-server/server.(*subStore).Remove()
      /Users/ivan/dev/go/src/github.com/nats-io/stan-server/server/server.go:233 +0x80
  github.com/nats-io/stan-server/server.(*StanServer).processUnSubscribeRequest()
      /Users/ivan/dev/go/src/github.com/nats-io/stan-server/server/server.go:1106 +0xb43
  github.com/nats-io/stan-server/server.(*StanServer).(github.com/nats-io/stan-server/server.processUnSubscribeRequest)-fm()
      /Users/ivan/dev/go/src/github.com/nats-io/stan-server/server/server.go:560 +0x37
  github.com/nats-io/nats.(*Conn).waitForMsgs()
      /Users/ivan/dev/go/src/github.com/nats-io/nats/nats.go:1355 +0x3aa

If you start a subscriber with start position "Last Received", then send a message on the subject, the subscriber does not receives it.

The issue is that if there is was no message ever sent on that subject, LastSequence() would be 0, and the server tries to get LastSequence() - 1, which because of the use of uint64 makes it the max uint64 value!

UPDATE: This is true for other start positions.

When a server is restarted, it gets new subjects for pub/sub/etc... which makes an already "connected" STAN application "fail" publishing new messages since it uses the old subjects, so those messages go nowhere.

Running more than one streaming server with the same ID in a NATS server cluster will result in undefined behavior, and should not be allowed. Add defensive code to prevent startup if another streaming server is detected with the same cluster ID.

If an application with a subscriber stops without calling Close(), if new messages on the subject are received by the server, their sequence number will be persisted in the channel's subscriptions file. The server will try send those messages, and even attempt to redeliver them (until the first missed heartbeat).

For durables, this is more of an issue because on durable restart, all those messages will be considered redelivered - while one would argue that they should be considered new messages since the application never received them (I understand the limitations on how much the server can know if those messages were actually received or not, regardless of the described behavior).

Until we have a solid story about application stopping and STAN server knowing about it, should we reduce the client timeout from 5 minutes to something smaller (but not go too small like we did previously)? Any other approach that would eliminate this behavior, or is this considered ok?

Have a durable running with auto-ack and receive new-only.
Send messages 1 and 2.
Restart durable.
Send messages 3 and 4.
Restart durable => it receives messages 3 and 4 as redelivered (EXPECTED: nothing)
Stop durable.
Restart server.
Start durable => it receives messages 3, 4, 1, 2, 3, 4 (EXPECTED: nothing)

From @ColinSullivan1 on March 16, 2016 22:25

Server-To-Client heart beats need to be configurable:

Something along the lines of:

• HeartBeatInterval
• HeartBeatTimeout
• HeartBeatMaxFailCount

We should discuss if the server sends the interval in the connect response and the clients behave accordingly, or we require the client to set this explicitly and independently. IMO sending these to the client will reduce support costs and make STAN simpler for users.

Copied from original issue: nats-io/stan.go#39