cyberluisda / devo-go Goto Github PK

Description

devo-sender: ReliableClient does not reconnect after relay endpoint outage and recovery

Steps to reproduce

1. Create a local relay mock, in example using tcpserver tool:

tcpserver 0.0.0.0 13000 sh -c "cat 1>&2"

1. In a separate shell session, create a reliable client pointed to mock server, in example using ./examples/reliablecentralrelay.go provided in the project:

go1.14 run ./examples/reliablecentralrelay.go -seconds 240 -relay tcp://localhost:13000 -mps 10

1. Check that events are displayed in the shell session which mock server is running

2. Stop (Ctrl+C) the mock server while reliablecentralrelay is sending events

3. Starts the mock server again (tcpserver)

4. You will see that any new event is not arriving nor displaying in the mock server shell session

Expected behavior

Client will reconnect automatically, assuming that clientReconnectionDaemon is up and running, after a outage and recovery of Devo endpoint service

From request from a client of the library I decide to change the license to MIT

There are some issues caused by the internal data structures used by ReliableClient to persist data in status db.

On the other hand, record update implementation based on save new record and delete old one in order to get Client ID reference is not optimal and require major amount of resources as strictly required by nutsdb while do this operations.

Current version of ReliableClient does not do any house-keeping over internal status database (nutsdb). This implies that files used to persisted status grown without limit. As side effect memory is grown too in order to manage those files

We need a automatic, or semi-automatic (on-demand call) mechanism to consolidate files and prevent high memory/disk space consumption. At first glance this mechanims should be based on Merge call to nutsdb object.

Maybe re-develop how to update internal status when a event is resend (copy to change the key only) will be necessary too

Create ID and save async operation as pending is done by go func (fork) in current implementation. See next code snipped:

func (dsc *Client) SendWTagAsync(t, m string) string {
	dsc.waitGroup.Add(1)
	id := uuid.NewV4().String()
...

	// Run Send with go routine (concurrent call)
	go func(id string) {
		// Save asyncItems ref ids
		dsc.asyncItemsMutext.Lock()
		dsc.asyncItems[id] = nil
		dsc.asyncItemsMutext.Unlock() 

...

This implies that ask for pending tasks using funcs like and can return empty list becasue de go func is not running jet or it is blocked at dsc.asyncItemsMutext.Lock() in the example.

This is an example of calling that can ends with wrong behaviour:

dsc := // Whatever you do ti instantiate *Client
dsc.SendWTagAsync("the tag", "the message")
fmt.Println(dsc.AreAsyncOps) // Print false if fork is blocked in dsc.asyncItemsMutext.Lock() line

Parent must be the responsible of open the async pending operations

Current unit-tests implementation is based on connection to external services like www.example.com:80. The mainly affected modules are devo-sender and other ones that depend on it.

The idea is add saverserver-go dependency for tests, start the server during test, and then connect to that internal service and check the result.

saverserver-go supports TLS connections too.

Vulnerabilities scan method (like snyk) detected next critial vulnerability in devosender module: CVE-2021-3538

We need to replace github.com/satori/go.uuid dependency because there is not any update with this issue fixed in the repository

We must to check and replace this dependency in the rest of devo-go modules

Steps to reproduce.

Create a client with insanely short timeout and send async message in stand by mode

c, _ := := devosender.NewLazyClientBuilder().
  ClientBuilder(devosender.NewClientBuilder().EntryPoint("udp://localhost:13000")).
  FlushTimeout(time.Microsecond * 2). // to force error when wake up
  // EnableStandByModeTimeout(time.Millisecond * 100).
  Build()
c.StandBy()
c.SendWTagAsync("test.keep.free", "msg")

Then call WakeUp and Close:

c.WakeUp()
c.Close()

Process is blocked and never finish

Current implementation of LazyClient never tried to reconnect if connection to Devo relay is lost. One valid implementation should be copy and adapt the implementation of ReliableClient.clientReconnectionDaemon

LazyClient is not designed to be reliable as ReliableClient but temporal cut or lost connection to Devo relay is a very common scenarion that must be managed by LazyClient too.

We need a couple of methods to expose information about the pending events in devosender.SwitchDevoSender interface:

• IsLimitReachedLastFlush Return true if the last call of Flush could not send all events because the max limit of events to send by Flush was reached.
• PendingEventsNoConn Return the number of events that saved in buffer that was created (or updated) when connection to Devo Central or on-prem relay was not available

Current version of Client in devosender only allows, when tls is enabled, two modes to load CA to trust in the Devo certificates, provided with a file or []bytes, or not provided none.

We need that ClientBuilder adds SystemCertPool certs by default, to enable compatibility with Devo end-points signed by a trusted public CA.

This behavior can be deactivated with a flag passed to ClientBuilder: TLSNoLoadPublicCAs(noLoadPublicCAs bool)

We expect that devosender.*Client follow ``devosender.DevoSender` interface.

In same way we expect that NewDevoSenderXXX factory funcs return DevoSender interface elements instead of *Client type objects

Steps to reproduce

• Create a lockup server that can be easily shutdown, with tcpserver in example:

  tcpserver 0.0.0.0 13000 sh -c "cat 1>&2"
  

• Open new ReliableClient connected to server previously up

  c, err := devosender.NewReliableClientBuilder().
     DbPath("/tmp/test-rc").
     ClientBuilder(devosenderNewClientBuilder().EntryPoint("tcp:/localhost:13000")).
     Build()
  if err != nil {
    panic(err)
  }
  fmt.Println("standby", c.IsStandBy()) // returns false
  time.Sleep(time.Second * 10)
  // Stop tcpserver command (`Ctrl` + `c`)
  fmt.Println("standby", c.IsStandBy()) // returns false, But SHOULD return true because connection can not be established

Steps to reproduce.

• Create new LazyClient instance with small value in NewLazyClientBuilder().MaxRecordsResendByFlush() parameter
• Pass the LazyClient to StandBy() mode an add more elements that value set bellow
• WakeUp() and call several time to Flush() to force limit reached.
• Same events are sent again and again because buffer is not properly cleaned.

Current behavior
App crash displaying next error message:

anic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x8b2fe7]

goroutine 1 [running]:
github.com/cyberluisda/devo-go/devosender.(*LazyClient).SendAsync(0xc0001882d0, 0xc0000b27e0, 0x111, 0xc0002f6a20, 0x2d)
        /XXXXXX/go/pkg/mod/github.com/cyberluisda/[email protected]/devosender/devosender_lazy.go:434 +0x27
gitlab.devotools.com/engineering/endpointagent/invent/pkg/devosink.(*DevoSink).Save(0xc0001c8200, 0xc0003d6000, 0x2ed, 0x400, 0xa95fe0, 0xc0001c29f0)
        /XXXXXX/workspace/engineering/endpointagent/invent/pkg/devosink/devosink.go:119 +0x11a
gitlab.devotools.com/engineering/endpointagent/invent/pkg/sfwinvt.(*SfwInvtProcessor).send(0xc0001209c0, 0xc0003d6000, 0x2ed, 0x400, 0xc0004177a0, 0x119)
        /XXXXXX/workspace/engineering/endpointagent/invent/pkg/sfwinvt/sfwinvt.go:148 +0x13c
gitlab.devotools.com/engineering/endpointagent/invent/pkg/sfwinvt.(*SfwInvtProcessor).Process(0xc0001209c0, 0x0, 0x0)
        /XXXXXX/workspace/engineering/endpointagent/invent/pkg/sfwinvt/sfwinvt.go:131 +0x362
gitlab.devotools.com/engineering/endpointagent/invent/cmd.glob..func1(0xe18240, 0xc000120820, 0x0, 0x2, 0x0, 0x0)
        /XXXXXX/workspace/engineering/endpointagent/invent/cmd/start.go:32 +0x103
github.com/spf13/cobra.(*Command).execute(0xe18240, 0xc000120800, 0x2, 0x2, 0xe18240, 0xc000120800)
        /XXXXXX/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0xe17fc0, 0x0, 0x0, 0x0)
        /XXXXXX/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x35a
github.com/spf13/cobra.(*Command).Execute(...)
        /XXXXXX/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
gitlab.devotools.com/engineering/endpointagent/invent/cmd.Execute()
        /XXXXXX/workspace/engineering/endpointagent/invent/cmd/root.go:29 +0x2d
main.main()
        /XXXXXX/workspace/engineering/endpointagent/invent/main.go:10 +0x20
exit status 2

Steps to reproduce

• Instantiate LazyClient ensuring that default Devo tag is set in ClientBuilder:

  cb, _ := devosender.NewLazyClientBuilder().
    ClientBuilder(
      devosender.NewClientBuilder().
        EntryPoint("localhost").
        DefaultDevoTag("my.app.default").
        IsConnWorkingCheckPayload("\x00")
    )
  cl, _ := cb.Build()

• Send a message to ensure it is working

  cl.SendAsync("message 1)

• Set to standby mode

  cl.StandBy()

• Send new message to ensure it is working

  cl.SendAsync("message 2")

This value exposure help library clients to identify if should take some actions if the client has not the sufficient level of reliability based on if implementation is working in-memory only.

func name suggested to define/implement: SwitchDevoSender.OnlyInMemory() bool