This is along the lines of how Kafka works. Given that bigqueue is persistent, it make

I was thinking something along the lines - <div class="snippet-clipboard-content n

This is the final API - We can have only one producer, which c

Update APIs to use offsets about bigqueue HOT 3 CLOSED

grandecola commented on June 15, 2024

Update APIs to use offsets

from bigqueue.

Comments (3)

mangalaman93 commented on June 15, 2024

I was thinking something along the lines -

q := NewQueue()
(q *Queue) NewConsumer() (*Consumer, error)
(q *Queue) NewProducer() (*Producer, error)
(q *Queue) Close() error

(c *Consumer) Peek() ([]byte, error)
(c *Consumer) PeekString() (string, error)
(c *Consumer) Dequeue() error
(c *Consumer) PeekAndDequeue() error

(p *Producer) Enqueue(b []byte) error
(p *Producer) EnqeueuString(s string) error

type Policy struct {
  block    bool
  timeout time.Duration
}

from bigqueue.

mangalaman93 commented on June 15, 2024

This is the final API -

We can have only one producer, which can be shared across multiple routines.
Multiple consumers are allowed and can read from different offsets simultaneously.

NewQueue(dir string, opts ...Option)
IsEmpty() bool
Flush() error
Close() error

(q *Queue) Enqueue(b []byte) error
(q *Queue) EnqueueString(s string) error

(q *Queue) NewConsumer(name string, opts ...ConsumerOption) (*Consumer, error)
(c *Consumer) Dequeue() error
(c *Consumer) DequeueString() error

from bigqueue.

mangalaman93 commented on June 15, 2024

Data Formats

Metadata File

Head points to the first non-deleted entry in the queue.
Tail points to the last written entry +1 in the queue

Bytes	Data
0-7	version Number
8-15	head arena id
16-23	head byte offset in arena
24-31	tail arena id
32-39	tail byte offset in arena
40-47	arena size
48-55	number of consumers
repeated
56-63	length of consumer name (L)
64-71	arena id (head) of consumer
72-79	byte offset in the arena (head) of consumer
80-(L+79)	consumer name