Code Monkey home page Code Monkey logo

ably-go's Introduction

.github/workflows/check.yml .github/workflows/integration-test.yml Features

Go Reference

Ably is the platform that powers synchronized digital experiences in realtime. Whether attending an event in a virtual venue, receiving realtime financial information, or monitoring live car performance data – consumers simply expect realtime digital experiences as standard. Ably provides a suite of APIs to build, extend, and deliver powerful digital experiences in realtime for more than 250 million devices across 80 countries each month. Organizations like Bloomberg, HubSpot, Verizon, and Hopin depend on Ably’s platform to offload the growing complexity of business-critical realtime data synchronization at global scale. For more information, see the Ably documentation.

Overview

This is a Go client library for Ably. Supported features and known limitations are documented here.

Installation

~ $ go get -u github.com/ably/ably-go/ably

See Requirements

Usage

Using the Realtime API

Creating a client

client, err := ably.NewRealtime(ably.WithKey("xxx:xxx"))
if err != nil {
        panic(err)
}
channel := client.Channels.Get("test")

Subscribing to events

You may monitor events on connections and channels.

client, err = ably.NewRealtime(
ably.WithKey("xxx:xxx"),
ably.WithAutoConnect(false), // Set this option to avoid missing state changes.
)
if err != nil {
        panic(err)
}
// Set up connection events handler.
client.Connection.OnAll(func(change ably.ConnectionStateChange) {
        fmt.Printf("Connection event: %s state=%s reason=%s", change.Event, change.Current, change.Reason)
})
// Then connect.
client.Connect()
channel = client.Channels.Get("test")
channel.OnAll(func(change ably.ChannelStateChange) {
        fmt.Printf("Channel event event: %s channel=%s state=%s reason=%s", channel.Name, change.Event, change.Current, change.Reason)
})

Subscribing to a channel for all messages

unsubscribe, err := channel.SubscribeAll(ctx, func(msg *ably.Message) {
        fmt.Printf("Received message: name=%s data=%v\n", msg.Name, msg.Data)
})
if err != nil {
        panic(err)
}

Subscribing to a channel for EventName1 and EventName2 message names

unsubscribe1, err := channel.Subscribe(ctx, "EventName1", func(msg *ably.Message) {
        fmt.Printf("Received message: name=%s data=%v\n", msg.Name, msg.Data)
})
if err != nil {
        panic(err)
}
unsubscribe2, err := channel.Subscribe(ctx, "EventName2", func(msg *ably.Message) {
        fmt.Printf("Received message: name=%s data=%v\n", msg.Name, msg.Data)
})
if err != nil {
        panic(err)
}

Publishing to a channel

err = channel.Publish(ctx, "EventName1", "EventData1")
if err != nil {
        panic(err)
}

Publish will block until either the publish is acknowledged or failed to deliver.

Alternatively you can use PublishAsync which does not block:

channel.PublishAsync("EventName1", "EventData11", func(err error) {
	if err != nil {
		fmt.Println("failed to publish", err)
	} else {
		fmt.Println("publish ok")
	}
})

Note the onAck callback must not block as it would block the internal client.

Handling errors

Errors returned by this library may have an underlying *ErrorInfo type.

See Ably documentation for ErrorInfo.

badClient, err := ably.NewRealtime(ably.WithKey("invalid:key"))
if err != nil {
        panic(err)
}
err = badClient.Channels.Get("test").Publish(ctx, "event", "data")
if errInfo := (*ably.ErrorInfo)(nil); errors.As(err, &errInfo) {
        fmt.Printf("Error publishing message: code=%v status=%v cause=%v", errInfo.Code, errInfo.StatusCode, errInfo.Cause)
} else if err != nil {
        panic(err)
}

Announcing presence on a channel

err = channel.Presence.Enter(ctx, "presence data")
if err != nil {
        panic(err)
}

Announcing presence on a channel on behalf of other client

err = channel.Presence.EnterClient(ctx, "clientID", "presence data")
if err != nil {
        panic(err)
}

Updating and leaving presence

// Update also has an UpdateClient variant.
err = channel.Presence.Update(ctx, "new presence data")
if err != nil {
        panic(err)
}
// Leave also has an LeaveClient variant.
err = channel.Presence.Leave(ctx, "last presence data")
if err != nil {
        panic(err)
}

Getting all clients present on a channel

clients, err := channel.Presence.Get(ctx)
if err != nil {
        panic(err)
}
for _, client := range clients {
        fmt.Println("Present client:", client)
}

Subscribing to all presence messages

unsubscribe, err = channel.Presence.SubscribeAll(ctx, func(msg *ably.PresenceMessage) {
        fmt.Printf("Presence event: action=%v data=%v", msg.Action, msg.Data)
})
if err != nil {
        panic(err)
}

Subscribing to 'Enter' presence messages only

unsubscribe, err = channel.Presence.Subscribe(ctx, ably.PresenceActionEnter, func(msg *ably.PresenceMessage) {
        fmt.Printf("Presence event: action=%v data=%v", msg.Action, msg.Data)
})
if err != nil {
        panic(err)
}

Update MaxMessageSize/read limit for realtime message subscription

  • The default MaxMessageSize is automatically configured by Ably when connection is established with Ably.
  • This value defaults to 16kb for free and 64kb for PAYG account, please get in touch if you would like to request a higher limit for your account.
  • Upgrading your account to higher limit will automatically update MaxMessageSize property and should accordingly set the client side connection read limit.
  • If you are still facing issues when receiving large messages or intentionally want to reduce the limit, you can explicitly update the connection read limit:
client, err = ably.NewRealtime(ably.WithKey("xxx:xxx"))
if err != nil {
        panic(err)
}
client.Connection.SetReadLimit(131072) // Set read limit to 128kb, overriding default ConnectionDetails.MaxMessageSize

Note - If connection read limit is less than size of received message, the client will throw an error "failed to read: read limited at {READ_LIMIT + 1} bytes" and will close the connection.

Using the REST API

Introduction

All examples assume a client and/or channel has been created as follows:

client, err := ably.NewREST(ably.WithKey("xxx:xxx"))
if err != nil {
        panic(err)
}
channel := client.Channels.Get("test")

Publishing a message to a channel

err = channel.Publish(ctx, "HelloEvent", "Hello!")
if err != nil {
        panic(err)
}
// You can also publish multiple messages in a single request.
err = channel.PublishMultiple(ctx, []*ably.Message{
        {Name: "HelloEvent", Data: "Hello!"},
        {Name: "ByeEvent", Data: "Bye!"},
})
if err != nil {
        panic(err)
}
// A REST client can publish messages on behalf of another client
// by providing the connection key of that client.
err := channel.Publish(ctx, "temperature", "12.7", ably.PublishWithConnectionKey("connectionKeyOfAnotherClient"))
if err != nil {
        panic(err)
}

Querying the History

pages, err := channel.History().Pages(ctx)
if err != nil {
        panic(err)
}
for pages.Next(ctx) {
        for _, message := range pages.Items() {
                fmt.Println(message)
        }
}
if err := pages.Err(); err != nil {
        panic(err)
}

Presence on a channel

pages, err := channel.Presence.Get().Pages(ctx)
if err != nil {
        panic(err)
}
for pages.Next(ctx) {
        for _, presence := range pages.Items() {
                fmt.Println(presence)
        }
}
if err := pages.Err(); err != nil {
        panic(err)
}

Querying the Presence History

pages, err := channel.Presence.History().Pages(ctx)
if err != nil {
        panic(err)
}
for pages.Next(ctx) {
        for _, presence := range pages.Items() {
                fmt.Println(presence)
        }
}
if err := pages.Err(); err != nil {
        panic(err)
}

Fetching your application's stats

pages, err := client.Stats().Pages(ctx)
if err != nil {
        panic(err)
}
for pages.Next(ctx) {
        for _, stat := range pages.Items() {
                fmt.Println(stat)
        }
}
if err := pages.Err(); err != nil {
        panic(err)
}

Getting the channel status

status, err := channel.Status(ctx)
if err != nil {
        panic(err)
}
fmt.Print(status, status.ChannelId)

Configure logging

  • By default, internal logger prints output to stdout with default logging level of warning.
  • You need to create a custom Logger that implements ably.Logger interface.
  • There is also an option provided to configure loglevel.
type customLogger struct {
	*log.Logger
}

func (s *customLogger) Printf(level ably.LogLevel, format string, v ...interface{}) {
	s.Logger.Printf(fmt.Sprintf("[%s] %s", level, format), v...)
}

func NewCustomLogger() *customLogger {
	logger := &customLogger{}
	logger.Logger = log.New(os.Stdout, "", log.LstdFlags)
	return logger
}

client, err = ably.NewRealtime(
        ably.WithKey("xxx:xxx"),
        ably.WithLogHandler(NewCustomLogger()),
        ably.WithLogLevel(ably.LogWarning),
)

Proxy support

The ably-go SDK doesn't provide a direct option to set a proxy in its configuration. However, you can use standard environment variables to set up a proxy for all your HTTP and HTTPS connections. The Go programming language will automatically handle these settings.

Setting Up Proxy via Environment Variables

To configure the proxy, set the HTTP_PROXY and HTTPS_PROXY environment variables with the URL of your proxy server. Here's an example of how to set these variables:

export HTTP_PROXY=http://proxy.example.com:8080
export HTTPS_PROXY=http://proxy.example.com:8080
  • proxy.example.com is the domain or IP address of your proxy server.
  • 8080 is the port number of your proxy server.

Considerations

  • Protocol: Make sure to include the protocol (http or https) in the proxy URL.
  • Authentication: If your proxy requires authentication, you can include the username and password in the URL. For example: http://username:[email protected]:8080.

After setting the environment variables, the ably-go SDK will route its traffic through the specified proxy for both Rest and Realtime clients.

For more details on environment variable configurations in Go, you can refer to the official Go documentation on http.ProxyFromEnvironment.

Setting Up Proxy via custom http client

For Rest client, you can also set proxy by providing custom http client option ably.WithHTTPClient:

ably.WithHTTPClient(&http.Client{
        Transport: &http.Transport{
                Proxy:        proxy // custom proxy implementation
        },
})

Important Note - Connection reliability is totally dependent on health of proxy server and ably will not be responsible for errors introduced by proxy server.

Note on usage of ablytest package

Although the ablytest package is available as a part of ably-go, we do not recommend using it as a sandbox for your own testing, since it's specifically intended for client library SDKs and we don’t provide any guarantees for support or that it will remain publicly accessible. It can lead to unexpected behaviour, since some beta features may be deployed on the sandbox environment so that they can be tested before going into production.

You should rather use, ably.NewRealtime by passing the ABLY_KEY, which would be using the Ably production environment.

client, err := ably.NewRealtime(ably.WithKey("xxx:xxx"))

You can also use the control api to setup a test environment using https://github.com/ably/ably-control-go/.

Resources

Demo repositories hosted at ably-labs which use ably-go.

Broaden your knowledge of realtime in Go with these useful materials:

Requirements

Supported Versions of Go

Whenever a new version of Go is released, Ably adds support for that version. The Go Release Policy supports the last two major versions. This SDK follows the same policy of supporting the last two major versions of Go.

Breaking API Changes in Version 1.2.x

Please see our Upgrade / Migration Guide for notes on changes you need to make to your code to update it to use the new API introduced by version 1.2.x.

Users updating from version 1.1.5 of this library will note that there are significant breaking changes to the API. Our current approach to versioning is not compliant with semantic versioning, which is why these changes are breaking despite presenting only a change in the minor component of the version number.

Feature support

This library targets the Ably 1.2 client library specification. List of available features for our client library SDKs can be found on our feature support matrix page.

Known limitations

As of release 1.2.0, the following are not implemented and will be covered in future 1.2.x releases. If there are features that are currently missing that are a high priority for your use-case then please contact Ably customer support. Pull Requests are also welcomed.

REST API

Realtime API

  • Inband reauthentication is not supported; expiring tokens will trigger a disconnection and resume of a realtime connection. See server initiated auth for more details.

  • Realtime connection failure handling is partially implemented. See host fallback for more details.

  • Channel suspended state is partially implemented. See suspended channel state.

  • Realtime Ping function is not implemented.

  • Message Delta Compression is not implemented.

  • Push Notification Target functional is not applicable for the SDK and thus not implemented.

Support, feedback and troubleshooting

Please visit https://faqs.ably.com/ for access to our knowledgebase. If you require support, please visit https://ably.com/support to submit a support ticket.

You can also view the community reported Github issues.

Contributing

For guidance on how to contribute to this project, see CONTRIBUTING.md.

ably-go's People

Contributors

amnonbc avatar andydunstall avatar andynicks avatar andytwf avatar audiolion avatar gernest avatar jmgr avatar ken8203 avatar kouno avatar lacriment avatar lmars avatar mattheworiordan avatar mohyour avatar morganamilo avatar mschristensen avatar niksilver avatar orbat avatar owenpearson avatar paddybyers avatar quintinwillison avatar ramiro-nd avatar rjeczalik avatar rosalita avatar sacoo7 avatar tcard avatar tomkirbygreen avatar zknill avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ably-go's Issues

Revert broken encoding on 0.8.1

[This commit] (97429bb) has broken the library. It is not valid to specify an encoding of utf-8 when the caller just passed a string as data and the message is forwarded to Ably with a string in data. utf-8 indicates that the data is binary, but it contains an encoded string.

I know that this is fixed in the 1.1 work but we need a 0.8.2 release that fixes this problem. The fix is to omit the encoding completely.

Spec validation

For version 0.8 of our client libraries, we have a documented specification at http://docs.ably.io/client-lib-development-guide/features/

It is required that all of our client libraries that use a 0.8.x version number, where possible, adhere to this specification ensuring feature compatibility across all languages and platforms. To ensure that we do not have regressions in future from this specification, all of the features within the 0.8 spec should have a matching test as well.

Please can you review this library against the feature set and do the following:

  • Please copy & paste all the feature details in http://docs.ably.io/client-lib-development-guide/features/ into a Google Spreadsheet or something similar, and where possible, provide a link to a test this library that assures us that the intended functionality is implemented. Sometimes a test is not required, but please make us aware when this is not the case in the comments. See the PHP Rest library example. This is quite onerous, however it is far more difficult for us to validate this than the client library developer who knows the code.
  • Check that the README.md is up to date with the Ruby version with details such as how to install it from a package manager if applicable.
  • If any features from above are missing, please can you add a "Known Limitations" section to your README documenting the missing features from the client library.

Fix REST/Presence tests to not fail randomly.

Running all tests one after another causes Presence tests to fail sporadically. They started to be flaky after merging all ably-go packages into one.

Example output:

~ $ for j in "-race" ""; do for i in {1..10}; do go test $j ./...; done; done
(...)

[Fail] Presence tested against presence fixture data set up in test app Get [It] returns current members on the channel 
/Users/rjeczalik/ably/src/github.com/ably/ably-go/ably/rest_presence_test.go:25

[Fail] Presence tested against presence fixture data set up in test app Get with a limit option [It] returns a paginated response 
/Users/rjeczalik/ably/src/github.com/ably/ably-go/ably/rest_presence_test.go:32

[Fail] Presence tested against presence fixture data set up in test app History [It] returns a list of presence messages 
/Users/rjeczalik/ably/src/github.com/ably/ably-go/ably/rest_presence_test.go:50

[Fail] Presence tested against presence fixture data set up in test app History with start and end time [It] can return older items from a certain date given a start / end timestamp 
/Users/rjeczalik/ably/src/github.com/ably/ably-go/ably/rest_presence_test.go:63

Flaky TestAuth_ClientID test

This test occasionally fails with

--- FAIL: TestAuth_ClientID (9.17s)
	auth_test.go:591: want err to be *ably.Error; was <nil>: <nil>

ID and Client ID is blank on realtime incoming messages.

Found a possible bug that is causing trouble for me. Incoming realtime messages have a blank message ID and some other blank fields like Client ID. Historical messages DO have an ID. Other SDKs such as the Javascript library do report IDs in realtime.

Here is some test code that you can run, you should see that the message IDs are blank.

{code}
func main() {
options := ably.NewClientOptions("APIKEY")
var err error
ablyClient, err := ably.NewRealtimeClient(options)
ablyChannel := ablyClient.Channels.Get("test")

go func() {
	number := 1
	for {
		time.Sleep(time.Second * 2)
		ablyChannel.Publish("test", fmt.Sprintf("Message number %v", number))
		number++
	}
}()

sub, err := ablyChannel.Subscribe("test")
if err != nil {
	panic(err)
}
for msg := range sub.MessageChannel() {
	fmt.Printf("recv:\n\tName: %s\n\tData: %s\n\tID: %s \n\tClient ID: %s\n", msg.Name, msg.Data, msg.ID, msg.ClientID)
}

}
{code}

Support client instantiation via Token or TokenRequest

On the REST and Realtime library.

/!\ Do not implement this yet as some discussion are going on about the naming convention for Token and TokenRequest.

Our client libraries are still not in line with each other on that point.

Google App Engine

Is there a way to use it inside Google Apple Engine standard environment? If not, I think it would be an interesting addition, any thoughts?

┆Issue is synchronized with this Jira Uncategorised by Unito

API completeness

Currently the API implemented is incomplete in the following areas, we should try and fix that first before add new features required in the 0.8 spec.

REST API:

  • protocol.Message should have automatic message decoding (#9 (comment)) (TM2g, RSL4a, RSL4c3, RSL5, RSL6, RSP5, RTL7d, RTL7e)
  • add RestChannels container (RSN1)
  • add missing methods to PaginatedResults (TG5-TG-7)
  • ensure Stats struct conforms to the spec (TS1-TS9)
  • ensure ClientOptions struct conforms to the spec - mostly timeouts (TO3l)

Realtime client API completeness:

  • add ChannelOptions struct (TB)
  • add CipherParams struct (TZ)
  • change Channels (RealtimeChannels) method signature to accept ChannelOptions, CipherParams
  • add RealtimePresence.History method and PresenceMessage filtering functions (RTP12, RTPC11c1-RTPC11c2)

Validation, test, and readme example for publishing object data

Customer reports that publishing a message with the data as a struct doesn't work:

type message struct {
  Title string `json:"title"`
  Body string `json:"body"`
  DisplayTime int `json:"displayTime"`
}
msg := message{"Hey", "Test", 12345}
err = channel.Publish("name", msg)

produces msgpack encode error: unsupported payload type (status=0, internal=80013)


I've looked through the tests, and as far as I can tell, every test of publishing uses string data. Do we really not have any tests of publishing and receiving an object?

┆Issue is synchronized with this Jira Task by Unito

API changes

Don't action yet, but take a look at ably/ably-js#33 where we are proposing a final set of API changes before we roll out stable versions.

API changes Apr 2015

Before our stable release, we agreed to make a few final amends to our client library APIs to improve consistency and reduce ambiguity.

The token related changes are breaking changes, so we need these changes to be rolled out for this client library ASAP as we intend on migrating to the new API early next week. This means that this library will no longer be able to use Token auth when this change is rolled out.

Unhelpful error message when failing to parse an HTTP response

See https://github.com/ably/ably-go/blob/develop/ably/error.go#L99

This is giving a customer an error message that's pretty uninformative: Forbidden (status=403, internal=50000).

There's a few things wrong with this code:

  • it doesn't check the Content-Type. If we got an HTML response from a proxy or something (as is typical with a 403 error from Cloudflare or whatever) then we would just try to parse it as JSON;

  • even though we fail to parse the body, there is other useful information that could be included in the error; perhaps there are Ably headers with an error message or code, or even seeing the raw HTML text would help;

  • the Error type doesn't comply with the spec (ie it doesn't have Message, Href).

We need to provide something more helpful in the output and do a release with this.

Undescriptive errors; possible broken fallback functionality

Customer reporting "can't scan type: *interface {}" and " mime: no media type" errors from client.request().

First problem: this was during a time when us-east-1 router had [elevated levels of 5xxs|https://status.ably.io/incidents/652], but the client lib should have retried any requests which resulted in 5xxs against the fallback hosts. Is that functionality definitely working? Including with client.request()?

Second problem: those errors are useless; even in the event that the client library tries a request against all fallback hosts and fails each time, the err should be an {{ErrorInfo}} with an appropriate code, statusCode, and descriptive message of the reason for the last failure.

Clarification on client fallbacks/failures?

This blog-post claims that ably client libraries:

  • fallback to ably-realtime.com if the main ably.io domain is unavailable
  • retries requests to the API up to five times

But I can't see implementations of these claims in the client libraries. Is this information still accurate?

I also noticed that it's possible to configure various timeouts on the client, but I can't see them being used anywhere. Is there something I'm missing?

┆Issue is synchronized with this Jira Task by Unito

Go JSON / Binary / String support

As discussed with @kouno, we need to be aware that ALL of our client libraries natively support three payload data types namely:

  • String UTF-8
  • Byte arrays (binary)
  • JSON

Typically in dynamically typed languages, when you subscribe to messages, you will be delivered the data type automatically because when the message is received, it is decoded automatically, see https://github.com/ably/ably-ruby/blob/master/SPEC.md#ablymodelsmessageencoders.

In more strongly typed languages such as Java, we still encode & decode automatically and return the correct data type, see https://github.com/ably/ably-java/blob/db50628fceb0af37c63c33693ab77dbb35d3c245/src/io/ably/types/BaseMessage.java#L97-L135.

However, in Go, JSON is not a common data type and instead the norm is to Marshal JSON strings into typed Structs. The problem for us is that whilst we return a JSON data type in Ruby / Javascript / Java, for us to do this in Go we'd need to return a generic {{Interface{}}} object, see http://golang.org/src/encoding/json/decode.go?s=2340:2388#L57. We could do this, but my understanding is that this is largely frowned upon the Go community and instead JSON should be marshalled to a Struct. If we did return a generic interface, it's quite probable the developer would only then need to marshal it into another object so we're just adding unnecessary processing.

I don't have a solution, but am raising this issue so that we can discuss.

@paddybyers any thoughts?

Do not persist authorise attributes force & timestamp

A recent discovery in the iOS library has unearthed a small shortcoming in the spec, corrected at [https://github.com/ably/docs/commit/f768f0db76020d945cf34974ef7d14d36113f0cd|https://github.com/ably/docs/commit/f768f0db76020d945cf34974ef7d14d36113f0cd]

Push Notification support

We will need to support Push notifications in the Go library, and once that's been implemented update our documentation.

┆Issue is synchronized with this Jira Task by Unito

Client appears to be leaking TCP connections/file descriptors

We have some code in our service that is doing this:

import (
	"github.com/ably/ably-go/ably"
)

msg := "..."
client := ably.NewRestClient(ably.NewClientOptions(apiKey))

if err := client.Channel("private:some-channel").Publish("data_ready", string(msg)); err != nil {
	return err
}

We are using revision 0cadbda of the ably-go client.

When this runs, the service starts to leak a large number of TCP connections to Ably's rest.ably.io load balancer. Here's the output of lsof -p {service pid} after a few seconds of running:

COMMAND   PID   USER   FD      TYPE   DEVICE SIZE/OFF     NODE NAME
{service} 27085 {user}   14u     IPv4 42056111      0t0      TCP ip-10-0-2-94.ec2.internal:43861->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   15u     IPv4 42057076      0t0      TCP ip-10-0-2-94.ec2.internal:46022->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   16u     IPv4 42057083      0t0      TCP ip-10-0-2-94.ec2.internal:43913->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   17u     IPv4 42056169      0t0      TCP ip-10-0-2-94.ec2.internal:46034->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   18u     IPv4 42057114      0t0      TCP ip-10-0-2-94.ec2.internal:43933->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   19u     IPv4 42057151      0t0      TCP ip-10-0-2-94.ec2.internal:43959->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   20u     IPv4 42057127      0t0      TCP ip-10-0-2-94.ec2.internal:20860->ec2-52-20-107-126.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   21u     IPv4 42056184      0t0      TCP ip-10-0-2-94.ec2.internal:46050->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   22u     IPv4 42056206      0t0      TCP ip-10-0-2-94.ec2.internal:46078->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   23u     IPv4 42056216      0t0      TCP ip-10-0-2-94.ec2.internal:43975->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   24u     IPv4 42056233      0t0      TCP ip-10-0-2-94.ec2.internal:46102->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   25u     IPv4 42056248      0t0      TCP ip-10-0-2-94.ec2.internal:44005->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   26u     IPv4 42057212      0t0      TCP ip-10-0-2-94.ec2.internal:46128->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   28u     IPv4 42056281      0t0      TCP ip-10-0-2-94.ec2.internal:44039->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   29u     IPv4 42056288      0t0      TCP ip-10-0-2-94.ec2.internal:46156->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)

Eventually the connections enter a CLOSE_WAIT state, but still persist:

COMMAND   PID   USER   FD      TYPE   DEVICE SIZE/OFF     NODE NAME
{service} 27085 {user}   12u     IPv4 42056997      0t0      TCP ip-10-0-2-94.ec2.internal:45966->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   13u     IPv4 42057706      0t0      TCP ip-10-0-2-94.ec2.internal:44331->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   14u     IPv4 42056111      0t0      TCP ip-10-0-2-94.ec2.internal:43861->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   15u     IPv4 42057076      0t0      TCP ip-10-0-2-94.ec2.internal:46022->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   16u     IPv4 42057083      0t0      TCP ip-10-0-2-94.ec2.internal:43913->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   17u     IPv4 42056169      0t0      TCP ip-10-0-2-94.ec2.internal:46034->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   18u     IPv4 42057114      0t0      TCP ip-10-0-2-94.ec2.internal:43933->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   19u     IPv4 42057151      0t0      TCP ip-10-0-2-94.ec2.internal:43959->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   20u     IPv4 42057127      0t0      TCP ip-10-0-2-94.ec2.internal:20860->ec2-52-20-107-126.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   21u     IPv4 42056184      0t0      TCP ip-10-0-2-94.ec2.internal:46050->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   22u     IPv4 42056206      0t0      TCP ip-10-0-2-94.ec2.internal:46078->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   23u     IPv4 42056216      0t0      TCP ip-10-0-2-94.ec2.internal:43975->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   24u     IPv4 42056233      0t0      TCP ip-10-0-2-94.ec2.internal:46102->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   25u     IPv4 42056248      0t0      TCP ip-10-0-2-94.ec2.internal:44005->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   26u     IPv4 42057212      0t0      TCP ip-10-0-2-94.ec2.internal:46128->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   27u     IPv4 42056339      0t0      TCP ip-10-0-2-94.ec2.internal:44057->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   28u     IPv4 42056281      0t0      TCP ip-10-0-2-94.ec2.internal:44039->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   29u     IPv4 42056288      0t0      TCP ip-10-0-2-94.ec2.internal:46156->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   30u     IPv4 42057383      0t0      TCP ip-10-0-2-94.ec2.internal:46186->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   31u     IPv4 42056381      0t0      TCP ip-10-0-2-94.ec2.internal:44093->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   32u     IPv4 42057433      0t0      TCP ip-10-0-2-94.ec2.internal:46226->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   33u     IPv4 42057489      0t0      TCP ip-10-0-2-94.ec2.internal:46276->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   34u     IPv4 42056411      0t0      TCP ip-10-0-2-94.ec2.internal:44117->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   35u     IPv4 42057454      0t0      TCP ip-10-0-2-94.ec2.internal:46246->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   36u     IPv4 42056469      0t0      TCP ip-10-0-2-94.ec2.internal:44169->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   37u     IPv4 42056431      0t0      TCP ip-10-0-2-94.ec2.internal:44137->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
{service} 27085 {user}   38u     IPv4 42057523      0t0      TCP ip-10-0-2-94.ec2.internal:46306->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   39u     IPv4 42056511      0t0      TCP ip-10-0-2-94.ec2.internal:44201->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   40u     IPv4 42056539      0t0      TCP ip-10-0-2-94.ec2.internal:46338->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)

Eventually (within 5-10 minutes) lsof stops reporting the state of the connection and merely states it's a TCP connection:

COMMAND   PID   USER   FD      TYPE   DEVICE SIZE/OFF     NODE NAME
{service} 27085 {user}   12u     sock      0,8      0t0 42056997 protocol: TCP
{service} 27085 {user}   13u     sock      0,8      0t0 42057706 protocol: TCP
{service} 27085 {user}   14u     sock      0,8      0t0 42056111 protocol: TCP
{service} 27085 {user}   15u     sock      0,8      0t0 42057076 protocol: TCP
{service} 27085 {user}   16u     sock      0,8      0t0 42057083 protocol: TCP
{service} 27085 {user}   17u     sock      0,8      0t0 42056169 protocol: TCP
{service} 27085 {user}   18u     sock      0,8      0t0 42057114 protocol: TCP
{service} 27085 {user}   19u     sock      0,8      0t0 42057151 protocol: TCP
{service} 27085 {user}   20u     IPv4 42057127      0t0      TCP ip-10-0-2-94.ec2.internal:20860->ec2-52-20-107-126.compute-1.amazonaws.com:https (ESTABLISHED)
{service} 27085 {user}   21u     sock      0,8      0t0 42056184 protocol: TCP
{service} 27085 {user}   22u     sock      0,8      0t0 42056206 protocol: TCP
{service} 27085 {user}   23u     sock      0,8      0t0 42056216 protocol: TCP
{service} 27085 {user}   24u     sock      0,8      0t0 42056233 protocol: TCP
{service} 27085 {user}   25u     sock      0,8      0t0 42056248 protocol: TCP
{service} 27085 {user}   26u     sock      0,8      0t0 42057212 protocol: TCP
{service} 27085 {user}   27u     sock      0,8      0t0 42056339 protocol: TCP
{service} 27085 {user}   28u     sock      0,8      0t0 42056281 protocol: TCP
{service} 27085 {user}   29u     sock      0,8      0t0 42056288 protocol: TCP
{service} 27085 {user}   30u     sock      0,8      0t0 42057383 protocol: TCP
{service} 27085 {user}   31u     sock      0,8      0t0 42056381 protocol: TCP
{service} 27085 {user}   32u     sock      0,8      0t0 42057433 protocol: TCP
{service} 27085 {user}   33u     sock      0,8      0t0 42057489 protocol: TCP
{service} 27085 {user}   34u     sock      0,8      0t0 42056411 protocol: TCP
{service} 27085 {user}   35u     sock      0,8      0t0 42057454 protocol: TCP
{service} 27085 {user}   36u     sock      0,8      0t0 42056469 protocol: TCP
{service} 27085 {user}   37u     sock      0,8      0t0 42056431 protocol: TCP
{service} 27085 {user}   38u     sock      0,8      0t0 42057523 protocol: TCP
{service} 27085 {user}   39u     sock      0,8      0t0 42056511 protocol: TCP
{service} 27085 {user}   40u     sock      0,8      0t0 42056539 protocol: TCP
{service} 27085 {user}   41u     sock      0,8      0t0 42056547 protocol: TCP

It appears rest.ably.io is resolving to the load balancer mentioned in the lsof output:

dig rest.ably.io

; <<>> DiG 9.10.3-P4-Ubuntu <<>> rest.ably.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52768
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;rest.ably.io.			IN	A

;; ANSWER SECTION:
rest.ably.io.		17	IN	A	52.207.88.5
rest.ably.io.		17	IN	A	52.0.70.16

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Sep 24 12:07:02 UTC 2018
;; MSG SIZE  rcvd: 73

These leaking connections eventually overwhelm our service, causing our systems to fail. Looking at the client interface it appears there's no obligation on the consumer of the client lib to call a Close() method, so it seems like the client may be failing to do some cleanup internally?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.