dudewhocode / kulay Goto Github PK

Add a channel on all producers and consumers so that other module can pull/push from the channel.

Similar to passing messages from redis queue to other services, Need to implement redis pubsub module where we can publish messages to a redis channel or subscribe messages from different channels and pass them around to different services.

Add basic logging to all modules.

Right now, kulay command takes AWS credentials from ~/.aws/config, instead take it from kulay config file

Rotate the jsonl files by batch count taken from config.toml, once batch count is reached close current file and open new one.

When I run the test for fileWriter.go I get following error but it doesn't happen when the code is run properly from main.

/usr/local/Cellar/go/1.8/libexec/bin/go test -c -i -o /private/var/folders/74/ld7ykl517ks_pdvqj28q3yhw0000gn/T/fileWriter_test_gogo naren/kulay/backend/fileio
/private/var/folders/74/ld7ykl517ks_pdvqj28q3yhw0000gn/T/fileWriter_test_gogo -test.v -test.run ^TestPut$
fatal error: all goroutines are asleep - deadlock!

As the project is recently moved from another service to github, refactor all imports to support go get command

The shared configuration in ~/.aws/config overrides the custom config.toml, When region is present in config.toml it should override ~/.aws/config by only taking secret key and value from it.

Change command design from
$ kulay from us_logs to sg_logs to
$ kulay -f sqs.us_logs to sqs.sg_logs
Commands doesn't suite the use case, so use flags

Create a module that writes all messages from a channel to file, This can be used to backup messages from other services.
$ kulay -f sqs.prod_queue -to file.jsonl
$ kulay -f sqs.prod_queue -to file.csv

Write more documentation on the config file structure, what all the options are available for the available service in kulay.

kulay -f random_sqs.test_queue -t redis.test_buffer
In above command the main section random_sqs is invalid and test_queue sub-section is valid. The command doesn't throw any error and silently waits.

As the tests include using AWS sqs, Its difficult to run it via CI. Use localstack instead of AWS cloud.

kulay should support passing messages between zeromq and other services

This module/command should pull from redis queue and push to sqs queue and vice versa

Write an aws SQS producer to complete the sqs module of kulay.

Add test cases for all initial features and check coverage.

Right now only single goroutine kicks off per command, Implement a worker pool where workers will be assigned for routing msg between services.

SQS to jsonl module is done, should implement the vice versa, picking up message from file and sending it to queues