1xyz / coolbeans Goto Github PK
View Code? Open in Web Editor NEWCoolbeans is a distributed work queue that implements the beanstalkd protocol.
License: Mozilla Public License 2.0
Coolbeans is a distributed work queue that implements the beanstalkd protocol.
License: Mozilla Public License 2.0
> git clone https://github.com/1xyz/coolbeans.git
> make build
go: module github.com/golang/protobuf is deprecated: Use the "google.golang.org/protobuf" module instead.
protoc -I api/v1 api/v1/*.proto --go_out=plugins=grpc:api/v1 --go_opt=paths=source_relative
--go_out: protoc-gen-go: plugins are not supported; use 'protoc --go-grpc_out=...' to generate gRPC
See https://grpc.io/docs/languages/go/quickstart/#regenerate-grpc-code for more information.
What is the expected behavior?
Other:
go version version 1.18.1
Describe the bug:
beanstalkd-proxy hangs and does not handle a SIGTERM properly
Provide the steps to reproduce the behavior:
The process does not exit.
What is the expected behavior?
Expect the process to gracefully exit
Other:
Describe the bug:
The getting started guide for a three node HA setup does not work as expected. Reported by external user
Provide the steps to reproduce the behavior:
Other:
Related bug 1xyz/coolbeans-backup#13
The beanstalkd state should have various elements of its Grpc configuration exposed as a configuration file (JSON), state server configuration
GrpC Server
Additional context
I have seen the last release is from 2020, so I wonder is it is halt or still under development :)
Problem: Refer beanstalkd/proxy/client.go
Here, we setup a streaming connection between the client and the leader server, however, if the connection is lost for some reason, we retry to connect with no delay or exponential back off. The bug suggests, that it might be problematic to consider a case with no exponential backoff an consider adding a backoff strategy for this
Describe the bug
Binary Releases are not getting posted in releases page
Expect: Binary Releases to get posted in releases page
Expected behavior
Binary Releases are not getting posted in releases page
beanstalk-client only has put & reserve commands, as we add more commands to the service, it would be nice to have them implemented
Have an option to start a prometheus server within beanstalkd that can expose the metrics
If a GRPC call fails due to a connection error, then we may want to retry the call using an exponential backoff strategy. This approach is different from GRPCs standard retry strategy, which I think does not handle a connection being reset by a peer
Problem: when a job > the prescribed job size limit is posted a ErrBadFormat is returned, it is expected to return ErrJobSizeTooBig
Fix: Added special case handling when the put job's size is higher than the posted value
Problem: Add a coolbeans specific CLI with commands as follows:
coolbeans client leave [node-id] [--node-addr=<addr>]
coolbeans client isReady [--node-addr=<addr>
Why is this important?
Describe the bug
Update Getting Started Documentation for non k8s
Problem: when an assigned bootstrap node attempts to bootstrap an existing cluster, it fails with bootstrap only works on new clusters
Repro: steps
make run-cluster
generated data in /tmp/bean*
make run-cluster
. The bootstrap node fails to bootstrap the cluster with bootstrap only works on new clustersExpected: the bootstrap to start the cluster.
Root cause: Since raft performs bootstrap for new clusters only.
Fix detail: Only the bootstrap node can fails with raft.ErrCantBootstrap
. In that case of the bootstrap document, we are going to ignore this error and continue to to start the Grpc server allowing other peers to connect
// called on an un-bootstrapped Raft instance after it has been created. This
// should only be called at the beginning of time for the cluster with an
// identical configuration listing all Voter servers. There is no need to
// bootstrap Nonvoter and Staging servers.
//
// A cluster can only be bootstrapped once from a single participating Voter
// server. Any further attempts to bootstrap will return an error that can be
// safely ignored.
//
// One sane approach is to bootstrap a single server with a configuration
// listing just itself as a Voter, then invoke AddVoter() on it to add other
// servers to the cluster.
func (r *Raft) BootstrapCluster(configuration Configuration) Future {```
Here are some documentation fixes for Readme
cc: @jrgallelli
See https://github.com/1xyz/coolbeans/runs/6379258692?check_suite_focus=true
The CI build failed with the following error.
Run make build
rm -rf bin/
go clean -cache
gofmt -s -l -w ./cluster/server/serverfakes/fake_jsm_tick.go ./cluster/server/jsm_server.go ./cluster/server/reservations_controller.go ./cluster/server/reservations_controller_test.go ./cluster/server/cluster_server.go ./cluster/server/health_server.go ./cluster/cmd/cmd_cluster.go ./cluster/cmd/cmd_client.go ./cluster/client/cluster_client.go ./main.go ./state/state.go ./state/client_resv.go ./state/jsm.go ./state/jsm_test.go ./state/errors.go ./state/state_string.go ./state/job_heap.go ./state/index.go ./state/job_heap_test.go ./state/client_resv_test.go ./tools/tools.go ./tools/opts.go ./tests/e2e/protocol_test.go ./beanstalkd/proto/conn_state_string.go ./beanstalkd/proto/conn.go ./beanstalkd/proto/conn_test.go ./beanstalkd/proto/conn_state.go ./beanstalkd/proto/tcp_server.go ./beanstalkd/core/cmd_type_string.go ./beanstalkd/core/client_test.go ./beanstalkd/core/parse_test.go ./beanstalkd/core/cmd_data.go ./beanstalkd/core/core.go ./beanstalkd/core/client.go ./beanstalkd/core/cmd_type.go ./beanstalkd/core/parse.go ./beanstalkd/core/cmd_data_test.go ./beanstalkd/core/cmd_proc.go ./beanstalkd/proxy/bool.go ./beanstalkd/proxy/bool_test.go ./beanstalkd/proxy/client.go ./beanstalkd/cmd/cmd_beanstalkd.go ./beanstalkd/cmd/beanstalkd.go ./store/store.go ./store/snapshot.go ./store/snapshot_test.go ./store/client_uri_test.go ./store/client_uri.go ./api/v1/empty.pb.go ./api/v1/cluster.pb.go ./api/v1/job.pb.go ./api/v1/jsm.pb.go ./api/v1/client.pb.go
./tools/tools.go
./tests/e2e/protocol_test.go
go get -u github.com/golang/protobuf/protoc-gen-go
go: downloading github.com/golang/protobuf v1.5.2
go: downloading google.golang.org/protobuf v1.2[6](https://github.com/1xyz/coolbeans/runs/6379258692?check_suite_focus=true#step:5:6).0
go: downloading google.golang.org/protobuf v1.2[8](https://github.com/1xyz/coolbeans/runs/6379258692?check_suite_focus=true#step:5:8).0
go: module github.com/golang/protobuf is deprecated: Use the "google.golang.org/protobuf" module instead.
go: upgraded github.com/golang/protobuf v1.4.3 => v1.5.2
go: upgraded google.golang.org/protobuf v1.25.0 => v1.28.0
protoc -I api/v1 api/v1/*.proto --go_out=plugins=grpc:api/v1 --go_opt=paths=source_relative
protoc-gen-go: program not found or is not executable
Please specify a program using absolute path or make sure the program is available in your PATH system variable
--go_out: protoc-gen-go: Plugin failed with status code 1.
make: *** [Makefile:81: protoc] Error 1
What?
How to measure?
The list-tubes command returns a list of all existing tubes. Its form is: list-tubes\r\n
The list-tube-used command returns the tube currently being used by the
client.
Its form is: list-tube-used\r\n
The list-tubes-watched command returns a list tubes currently being watched by the client. Its form is: list-tubes-watched\r\n
The pause-tube command can delay any new job being reserved for a given time. Its form is:
pause-tube \r\n
Add context in all method of state.go interface, this is to measure the time and to do detailed debugging. Also, see how you can span tracing.
tracing library: https://github.com/opentracing/opentracing-go
Problem: when you release a job from reserved state, the delay parameter is not working.
Repro:
reserve --tube=bar --release --delay 10
Actual: the delay is 1024 seconds
Expect: the delay to be 10 seconds
Hello! Thanks for starting and putting forward such a nice project.
Currently investing lots of time and effort into beanstalkd and your "drop-in" replacement is very interesting.
I'm currently wondering about the impact on speed that adding the Raft replication algorithm to beanstalkd would have. I know it's been benchmarked in the past to be very quick but I also know that replicating and having network round-trips are very costly.
Have you tried comparing the two side by side?
Thanks in advance!
Problem: Refer 1xyz/coolbeans-backup#24
The approach is unclean, consider alternate approach such as maintain multiple TCP connections between proxy client and cluster-node server.
Or maintain your own timers for timing out reservations.
In pr 1xyz/coolbeans-backup#21 . We exposed the creation of raft's user defined snapshot via the cluster RPC
We need to do a few things to make that snapshot really usable
Provide the facility to stream out bytes of the snapshot via RPC
Ability to verify the snapshot (a tool) attached sample source code
Provide an ability to start another cluster w/ this snapshot (or restore this snapshot)
package main
import (
"github.com/1xyz/coolbeans/state"
"github.com/1xyz/coolbeans/store"
log "github.com/sirupsen/logrus"
"os"
"time"
)
func init() {
log.SetFormatter(&log.TextFormatter{})
log.SetOutput(os.Stdout)
log.SetLevel(log.DebugLevel)
}
func main() {
filename := "/tmp/bean0/snapshots/2-418905-1587171565583/state.bin"
timeout := 120 * time.Second
rdr, err := os.Open(filename)
if err != nil {
log.Panicf("err = %v", err)
}
jsm, err := state.NewJSM()
if err != nil {
log.Panicf("err = %v", err)
}
err = store.RestoreSnapshotTo(rdr, jsm, timeout)
if err != nil {
log.Panicf("err = %v", err)
}
ss, err := jsm.Snapshot()
if err != nil {
log.Panicf("err= %v", err)
}
clients, err := ss.SnapshotClients()
if err != nil {
log.Panicf("err = %v", err)
}
nc := 0
for c := range clients {
nc ++
log.Infof("c = %v. heapIndex=%v", c.CliID, c.HeapIndex)
}
log.Infof("nc = %v", nc)
bucket := make(map[state.JobState]int)
bucket[state.Reserved] = 0
bucket[state.Initial] = 0
bucket[state.Ready] = 0
bucket[state.Buried] = 0
bucket[state.Delayed] = 0
bucket[state.Deleted] = 0
jobs, err := ss.SnapshotJobs()
if err != nil {
log.Panicf("err = %v", err)
}
tubeBuckets := make(map[state.TubeName]int)
nj := 0
for j := range jobs {
nj++
bucket[j.State()]++
_, ok := tubeBuckets[j.TubeName()]
if !ok {
tubeBuckets[j.TubeName()] = 0
}
tubeBuckets[j.TubeName()]++
}
log.Infof("total of %v jobs encountered", nj)
log.Infof("buckets %v", bucket)
log.Infof("tubeBuckets %v", tubeBuckets)
}
/(ใoใ)/~~
Run the beanstalk-benchmark on a kubernetes setup
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.