Code Monkey home page Code Monkey logo

zkstruct's Introduction

zkstruct

Simple way to share a struct with zookeeper. You can subscribe to changes and handle data changes.

This is a prototype, and should not be considered stable or be used for production purposes. You're going to do that anyways, but FYI.

overview

ZkStruct is designed to provide a struct that can act as a shared state.

We're building a little agent that will monitor a list of processes on the host. The agent uses ZkStruct for its state. An external actor can come along and modify that state object (either using ZkStruct or something else) to update the list of monitored processes. Let's assume this is a Vec<String>, where each String is a glob of the process we want to match.

This external actor modifies our Vec from vec!["systemd*", "rsync*"] to vec!["systemd*", "rsync*", "firefox*"]. The library will register the change, either from a Watcher event or refresh event, and update the internal state struct. The library will do a diff on the local state, emit all changes to a channel you can subscribe to, and then update the local state.

reads

Reads using .read() are not guaranteed to be fresh, and might be out of date. There is an eventual guarantee that the data will be consistent, as reads will fail if the object has not been verified consistent within a certain time (default is 30 seconds).

Reads using .c_read() are guaranteed to be fresh, as the local object version is compared to the zookeeper version. This will, of course, be slower as the client needs to request data from zookeeper and compare it before it will be returned, but you can be sure that the data being returned is correct.

writes

There are two write guarantees:

  1. You will always update the latest version of the object. If the local version of the object does not match the version in zookeeper, the update request will fail with an error.
  2. Only one update operation can take place at a time, either locally or among clients. An exclusive write lock is used both internally, and in zookeeper to prevent any unexpected updates or data loss.

Update Stream

Any changes, either local or remote, can be consumed and acted on. All clients can subscribe to a stream of changes to the local object. This can be done using the .update_handler method to attach a closure that will be caused everytime there is change registered.

There are 4 change types:

  1. Added
  2. Modified
  3. Removed
  4. Unchanged

Each Enum will come with the names of keys updated, and serde_json::Value body that contains the values added or removed. Modified changes will return the old and new values. Unchanged results are not enqueued and therefore not returned.

shared_state.update_handler(|msg| {
    println!("got stage change message: {:?}", msg);
});

Changes are emitted before the local object is updated, but the local object can be updated before all messages in the channel can be consumed.

zookeeper structure

/<dir>                  Where the Data will be stored
/<dir>/payload          JSON Payload
/<dir>/listeners        Ephemeral records with the IDs of every listener active

zkstruct's People

Contributors

volfco avatar

Watchers

 avatar  avatar

zkstruct's Issues

Local changes are not reflected in Change Messages

    let c = zkstate::ZkState::new(zka.clone(),dir, Testing { my_field: "yoyo".to_string() })?;

    let th = c.clone();
    std::thread::spawn(move || {
        let th = th;
        loop {
            println!("my data is {:?} and I have {} changes pending", th.read().unwrap(), th.metadata().0);
            std::thread::sleep(Duration::from_secs(1));
        }
    });

    std::thread::sleep(Duration::from_secs(5));

    c.update(|mut p| {
       p.my_field = "hello world again".to_string()
    });

    std::thread::sleep(Duration::from_secs(15));

Running the above code will result in 2 Noop changes, and not 1 Noop and 1 Change Change operations

Add better handling of Seralization Errors internally

Right now, ZkStruct can blow up if the struct changes; like so:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error("missing field `resources`", line: 0, column: 0)', /home/colum/.cargo/registry/src/github.com-1ecc6299db9ec823/zkstate-0.1.6/src/lib.rs:249:43
stack backtrace:
   0: rust_begin_unwind
             at /rustc/b3e19a221e63dcffdef87e12eadf1f36a8b90295/library/std/src/panicking.rs:493:5
   1: core::panicking::panic_fmt
             at /rustc/b3e19a221e63dcffdef87e12eadf1f36a8b90295/library/core/src/panicking.rs:92:14
   2: core::option::expect_none_failed
             at /rustc/b3e19a221e63dcffdef87e12eadf1f36a8b90295/library/core/src/option.rs:1329:5
   3: core::result::Result<T,E>::unwrap
             at /home/colum/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1037:23
   4: zkstate::state_change
             at /home/colum/.cargo/registry/src/github.com-1ecc6299db9ec823/zkstate-0.1.6/src/lib.rs:249:17
   5: zkstate::ZkState<T>::initialize
             at /home/colum/.cargo/registry/src/github.com-1ecc6299db9ec823/zkstate-0.1.6/src/lib.rs:90:9
   6: zkstate::ZkState<T>::new
             at /home/colum/.cargo/registry/src/github.com-1ecc6299db9ec823/zkstate-0.1.6/src/lib.rs:71:9
   7: stardance::executor::Executor::new
             at ./stardance/src/executor.rs:43:19
   8: executor::main
             at ./stardance/examples/executor.rs:42:20
   9: core::ops::function::FnOnce::call_once
             at /home/colum/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

There should be:

  1. better handling of this error so it doesn't panic
  2. documentation around best practices on changing the fields of this object

Add compression

Let's add the ability to compress the json blob, because why not

Add expect initalization method

There should be an expect method added to ZkStruct that blocks until data appears in the assumed location. Like this:

zkstate::ZkState::expect(
    zk.clone(), 
    "/testing"
).unwrap()

Expect should block until /testing/payload is populated by someone else. This is useful when you're creating a ZkState object from a ZooKeeper Watcher firing, and you get a notification that a node is appearing and you want to create a local copy of it. Using new requires a bogus struct that might be visible to listeners locally before this library populates it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.