iggy-rs / iggy Goto Github PK

Implement connection string for IggyClient to automatically connect and authenticate the client connection.

To verify if there aren't any deadlocks. And run perf tests.

EDIT:
just to add - this is pure exercise that can result in not-producing any code that'll be commited to master. in other words, it's just a task to check if everything's fine ;)

The current implementation of ConfigProvider uses the following code, to allow the overriding of server configuration via environment variables:

let config: Result<ServerConfig, figment::Error> = config_builder
    .merge(Env::prefixed("IGGY_").split("_"))
    .extract();

For example, the following setting:

[system.database]
path = "database"

Can be easily overwritten with:

export IGGY_SYSTEM_DATABASE_PATH=mydb

However, this will not work with the underscored field names, such as:

[quic.certificate]
self_signed = true

Will not be updated with:

export IGGY_QUIC_SELF_SIGNED=false

More information about this issue can be found here and in the mentioned nesting docs.

So we can avoid this kind of use:

Also, add DEFAULT_ROOT_* to this.

Maybe we could make use of some spinners or other kinds of indicators displaying the overall status, ongoing command execution etc.? Available crates spinoff, indicatif - I've tried to use indicatif once on the server-side, but had some issues with the tracing logger integration, even when using tracing-indicatif.

https://crates.io/crates/keepcalm
instead of the one from tokio, for System

I suggest doing it (if possible) in the logger format string configuration.

Hey mate, this project looks great.

I am building a video streaming platform: videocall-rs https://github.com/security-union/videocall-rs

We currently use NATs, it works great but I am very interested in contributing to a pure rust message streaming platform that is more aligned with our stack and values.

What really called my attention is that you are planning to add a REST api, with NATs we all end up just writing this by hand, so this really makes me happy

How can we help you? what are you working on?

CC @griffobeid

Remove the conditional flags from the server configuration, given that all SDKs at least support the user_login().

Something like

struct Messages {
    messages: Vec<Arc<Message>>
    total_size_bytes: usize
}

so we can have faster putting into cache, without iterator in buffer.rs, method extend.

It should work similarly to current release of iggy-server docker image to dockerhub.

Unfortunately we don't have this tested (yet).

Or create new action specifically for this.

For now, when calling the poll_messages() the underlying consumer or consumer group, make use of u32 ID. Investigate the usage of Identifier instead (same like when working with the streams, topics, users or other resources) to improve the developer experience by passing either numeric or text ID.

https://crates.io/crates/atone
instead of VecDeque

The idea is to start ssh server inside iggy-server docker container, add new user with minimal priviliges, add iggy-cmd to PATH.

In the web gui, it should be possible to ssh user@server:port and see output of terminal.

Main purpose of this is to access this terminal, and use iggy-cmd thru it. (so it would be good to have nice ssh hello msg)

So the out-binary (or library) can be smaller.

Besides that, implement previous, unsafe ring-buffer caching style, add it to config.

They are already stored, just need to release and add them.

When iggy-server is starting and cache is enabled, it tries to load everything to Vec<Arc<Messages>>. If on SSD there is more messages than memory, we get killed by system. Aim of this issue is to fix this problem by proposing new API, something like:
async fn get_newest_messages_by_size(u64: size_bytes)
and iterate starting from newest segments.

This way we would fetch only the amount we need, not everything.

So we can have a nice, separate channel with constant stream of updates.

https://github.com/iggy-rs/iggy/actions/runs/6275017384/job/17042092773

https://nnethercote.github.io/perf-book/build-configuration.html

Maybe we could make use of some spinners or other kinds of indicators displaying the overall status, ongoing command execution etc.? Available crates spinoff, indicatif - I've tried to use indicatif once on the server-side, but had some issues with the tracing logger integration, even when using tracing-indicatif.

So we could know anything regarding cache. Also, maybe think of some other metrics (not cache related) that are worth tracking.

I suggest using Display trait for all config structs.

Hi @spetz , the discord link is not working. Can you update it with a working link.

In order to make a new CLI compatible with existing API, we could allow the user to provide either numeric ID (e.g. Stream = 1), or string ID (e.g. Topic = "orders") for the resources such as stream, topic and user.

For example, the SDK command looks like this:

pub struct DeleteTopic {
    pub stream_id: Identifier,
    pub topic_id: Identifier,
} 

And the CLI command could either use the same Identifier wrapper or just make use of String? to work similarly:

pub(crate) struct TopicDelete {
    stream_id: u32,
    topic_id: u32,
}

The only exceptions are CreateStream and CreateTopic commands (in the latter, you can use the Identifier for stream_id only).

For example, for TCP we could expose buffer sizes, enabling/disabling Nagle algorithm.

M]()]()

Instead of ringbuffer or slow atomic-based global counter of used bytes.

Perhaps it should be possible to use crate https://docs.rs/rolling-file/latest/rolling_file/
instead of tracing-appender.

The parameter itself is already in the config.

When trying to publish a crate with cargo publish it fails with:

Caused by:
  Source directory was modified by build.rs during cargo publish. Build scripts should not modify anything outside of OUT_DIR.

Similar to the existing DeletePartitions command, we could introduce another kind of "administrative" request for deleting the segments. Given the specific Stream ID + Topic ID + Partition ID, the segments_count should remove all the segments starting from the oldest one (the lowest start offset in the log filename). The manual segments removal should act in the same way as the automated message expiration feature.

Refactor the existing Error into more concise (and easier to use) component with a helper methods for converting the values to/from string, int etc.