rdaum / moor Goto Github PK

Similar to #21 , I would like to see how this would perform and work out in code. I played with this in an earlier sketch of things, but didn't go far with it. In theory everything that is done via rocks can be done with FDB, but with distribution baked in (at the cost of performance and complexity)

They share a similar limitation though in that they're both optimized for a single-writer (per relation) but multiple reader workflow. Which may not describe the characteristics of what a transaction load is actually like in a working system.

Need a way to create a dump of the world state that can be used independent of the binary database, most especially useful for migration between versions of the DB format while things are still fairly unstable.

Does not necessarily have to be backwards compatible with LambdaMOO textdump

If command matching fails, the server should call huh on the current player object, if that verb exists.

LambdaCore and others use this to implement "feature" objects.

instead of concatenation with MkSingletonList.

    }

    fn generate_arg_list(&mut self, args: &Vec<Arg>) -> Result<(), CompileError> {
        // TODO(rdaum): Check recursion down to see if all literal values, and if so reduce to a Imm value with the full list,
        //  instead of concatenation with MkSingletonList.
        if args.is_empty() {
            self.emit(Op::ImmEmptyList);

we're not dependent on this.

        args: Vec<String>,
        attach: bool,
    ) -> Result<RpcResponse, RpcRequestError> {
        // TODO(rdaum): change result of login to return this information, rather than just Objid, so
        //   we're not dependent on this.
        let connect_type = if args.first() == Some(&"create".to_string()) {
            ConnectType::Created

7

Equality/inequality

        .op(Op::infix(Rule::lor, Assoc::Left))
        // 11. Logical and.
        .op(Op::infix(Rule::land, Assoc::Left))
        // TODO(rdaum): bitwise operators here (| 10, ^ XOR 9, & 8) if we ever get them.
        // 7
        // Equality/inequality
        .op(Op::infix(Rule::eq, Assoc::Left) | Op::infix(Rule::neq, Assoc::Left))

So host connections can remain live while the daemon is restarted for upgrades / bug fixes.

Core needs to be given a chance to parse the command before the builtin-parser gets to it.

These are library code now, and shouldn't be using anyhow, really. Most places are using specific Error trait implementations via thiserror but not all.

the list of functions for the `function_info` built-in right now. It could be used for

validating arguments, and could be part of the registration process for the actual builtin

implementations.

}

// Originally generated using ./generate_bf_list.py
// TODO(rdaum): this list is inconsistently used, and falls out of date. It's only used for generating
//  the list of functions for the `function_info` built-in right now. It could be used for
//  validating arguments, and could be part of the registration process for the actual builtin
//  implementations.

I would like a suite of tests to check for correctness of concurrent behaviour; linearizability, consistency, serialization conflicts, etc.

Still very vague in my mind, but probably a suite of Jepsen tests and/or Stateright (https://github.com/stateright/stateright).

Essentially, instantiating kernel + db with a predefined simple set of objects and then perform concurrent evaluations of mutations to provoke serialization conflicts, retries, and verify correct behaviour under concurrent load.

This will be essential in the short run run for a) finishing the transactional layer (retries on conflict) and b) making sure the Rocks etc layer works as hopes and in the medium/long run for swapping in alternative world state storage models.

Many games, and maybe all of them, require the ability to have verbs that open TCP sockets as listeners or to establish outgoing network connections at the verb level.

I understand this is contrary to your vision but it will present a significant if not insurmountable barrier of entry in limiting adoption of your server.

Games like Torchship use sockets to talk to external simulation layer services like our login server which uses TCP and not REST to communicate. Similarly, our space simulation server which is written in Rust also uses TCP to communicate with the game over a custom messaging protocol I wrote for that specific implementation. Which would be unideal to move to the server layer because it's iterated on so frequently it would result in significant service downtimes or stability issues during testing, which verbs offer in safety on the VM.

Other games use those connections to talk to things like mail forwarding agents.

LambdaMOO cores' connect function uses a variant of the Unix crypt function, ala old Unix passwd files, and stores the passwords in-db.

For compatibility with existing cores (and to make servicing existing telnet clients feasible), it's necessary to keep this process for now. But this isn't going to cut it as a security story.

Some kind of SSO story & auth token is going to be needed. For the latter, one option is JWT, another is PASETO, which sounds more promising.

Then the websocket connection (and any HTTP sessions generally) will need to be authenticated with this token before client connections can proceed.

The long-term future for the telnet modality will need to be thought about in this context.

Not a requirement for 1.0 (LambdaMOO compatible) release.

Performs a synchronous blocking read on the network connection. Will require some thought on how to do this right.

Now that I've pretty much de-cloaked on this project and there's some external people watching, it's probably best to document the decisions, vision, and architecture a bit.

Basically a couple diagrams breaking down the layers and components, thread and process and network boundaries, and a description of the why and how.

Right now task.rs just aborts on commit-conflict instead of re-running the transaction. Need to get around to the retry logic.

Hi, I'm Volund of the MUSH Community. Been looking into MOO on and off and I found this thanks to reddit... well, really I found it thanks to ToastStunt community talking about it and have a few things to say and suggest....

1. SPECTACULAR JOB at getting this to where it is. I was thinking about doing something similar if I had infinite time - which I don't - and now I don't have to, I can learn from and possibly help out with this one instead. I can't wait to tear into this and learn from it. :D And see where it's going.

2. Suggest a more careful rewording of the readme section that mentions why you'd use Moor instead of ToastStunt. It can be read to imply that ToastStunt doesn't have an active developer community, and it does.

3. Please create a Discord or similar chat-server for the Moor community to meet up on and grow around. Evennia has one, RhostMUSH has one, ToastStunt has one, I think AresMUSH has one... projects like this need lots of cultivation from a community to grow and thrive, so let's get cracking. 😄

Hello! I have some questions about the specific way docker is being used in this project. I'd like some help understanding the purpose/usecase of having the whole project recompile itself with cargo watch while watching for changes.

In my experience with docker, this tends to make the image more bloated and run slower.

cache this or index it better

    }

    fn get_players(&self) -> Result<ObjSet, WorldStateError> {
        // TODO(rdaum): this is going to be not-at-all performant in the long run, and we'll need a way to
        //   cache this or index it better
        get_all_object_keys_matching(
            &self.tx,

I may not be able to do with the current structure. See if I need to

            }
            Op::EndCatch(_) | Op::Continue | Op::EndExcept(_) | Op::EndFinally => {
                // Early exit; main logic is in TRY_FINALLY or CATCH etc case, above
                // TODO(rdaum): MOO has "return ptr - 2;"  -- doing something with the iteration, that
                //   I may not be able to do with the current structure. See if I need to
                unreachable!("should have been handled other decompilation branches")
            }

>@list $mcp:session_for
Verb 9 on #213 -- E_RANGE

>;verbs($mcp)
=> {"create_session", "destroy_session", "initialize_connection", "finalize_connection", "parse_version", "compare_version_range", "compare_version", "unparse_version", "session_for", "user_created", "user_disconnected", "do_out_of_band_command", "package_name", "wait_for_package", "nominate_for_core", "handles_package", "init_for_core"}

>;verb_code($mcp, "session_for")
=> E_INVARG  (E_INVARG)

Either in decompilation, or in verb resolution.

For support of e.g. MCP clients.

For moor, these would be, most meaningfully:

memory_usage : Just grab process RSS for the daemon PID. Simpler and more accurate than manually summing up the DB buffer pools etc.

db_disk_size: This would be the the total byte count of all tuples in the DB. For now since I'm not yet LRU-paging out tuples, that's the exact same as the usage count of the primary buffer pool. This gets trickier to calc after paging gets supported.

should be mmap'd to disk or something.

    client_id: Uuid,
    rpc_server: Arc<RpcServer>,
    player: Objid,
    // TODO(rdaum): manage this buffer better -- e.g. if it grows too big, for long-running tasks, etc. it
    //  should be mmap'd to disk or something.
    // TODO(rdaum): We could also use Boxcar or other append-only lockless container for this, since we only
    //  ever append.
    session_buffer: Mutex<Vec<(Objid, NarrativeEvent)>>,
}

ever append.

    client_id: Uuid,
    rpc_server: Arc<RpcServer>,
    player: Objid,
    // TODO(rdaum): manage this buffer better -- e.g. if it grows too big, for long-running tasks, etc. it
    //  should be mmap'd to disk or something.
    // TODO(rdaum): We could also use Boxcar or other append-only lockless container for this, since we only
    //  ever append.
    session_buffer: Mutex<Vec<(Objid, NarrativeEvent)>>,
}

Problem in stack unwinding related to error Catch handling.

Found while running:
help $scheduler

Doesn't happen for other :help_msg invocations it seems. Haven't tracked down what verb flow is causing this yet.

daemon log:

note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2023-09-24T21:59:30.157150364Z 2023-09-24T21:59:30.156994Z  WARN ThreadId(15) do_process: crates/kernel/src/tasks/scheduler.rs:787: Task sys-errored task=69 error=Disconnected
2023-09-24T22:00:42.543329425Z thread 'tokio-runtime-worker' panicked at 'TODO: CatchLabel where we didn't expect it...', crates/kernel/src/vm/vm_unwind.rs:367:25
2023-09-24T22:00:42.543368941Z stack backtrace:
2023-09-24T22:00:42.543373580Z    0: rust_begin_unwind
2023-09-24T22:00:42.543376937Z              at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/std/src/panicking.rs:593:5
2023-09-24T22:00:42.543380263Z    1: core::panicking::panic_fmt
2023-09-24T22:00:42.543383389Z              at /rustc/eb26296b556cef10fb713a38f3d16b9886080f26/library/core/src/panicking.rs:67:14
2023-09-24T22:00:42.543388429Z    2: moor_kernel::vm::vm_unwind::<impl moor_kernel::vm::VM>::unwind_stack
2023-09-24T22:00:42.543392156Z              at ./crates/kernel/src/vm/vm_unwind.rs:367:25
2023-09-24T22:00:42.543410261Z    3: moor_kernel::vm::vm_execute::<impl moor_kernel::vm::VM>::exec::{{closure}}
2023-09-24T22:00:42.543413777Z              at ./crates/kernel/src/vm/vm_execute.rs:502:28
2023-09-24T22:00:42.543446871Z    4: <moor_kernel::tasks::moo_vm_host::MooVmHost as moor_kernel::tasks::vm_host::VMHost<moor_kernel::vm::opcode::Program>>::exec_interpreter::{{closure}}
2023-09-24T22:00:42.543462882Z              at ./crates/kernel/src/tasks/moo_vm_host.rs:182:68

so we can check for multiple flags at once.

    ) -> Result<Objid, WorldStateError> {
        if parent != NOTHING {
            let (flags, parent_owner) = (self.flags_of(parent)?, self.owner_of(parent)?);
            // TODO check_object_allows should take a BitEnum arg for `allows`
            //   so we can check for multiple flags at once.
            self.perms(perms)?
                .check_object_allows(parent_owner, flags, ObjFlag::Read)?;
            self.perms(perms)?

Right now it's task.rs that does the logic with this.

With the command interpreter loop there's 2 separate transactions: one to match for parsing the command, and another to execute the command. And with support for $do_command there's a third, and once I fix missing huh handling, that's a fourth! Not a good situation. 3/4 of these transactions all occur within the scheduler, and so could be all subsumed under that one place.

I played with optionally passing an already-existing transaction into the task, but the problem with this is that to do the :huh support the transaction will already have been committed when the task completes with error.

So my proposal is to go through and clean things up potentially so that tasks always begin and end with an open transaction, and it's Scheduler that is responsible for the lifecycle of transactions -- passing ownership of them to the task, and getting them back again at completion.

moor itself. And really only something with a WorldState can do that. So it's not

enough to have validated the auth token here, we will need to pepper the scheduler/task

code with checks to make sure that the player objid is valid before letting it go

            }
        }

        // TODO(rdaum): we will need to verify that the player object id inside the token is valid inside
        //   moor itself. And really only something with a WorldState can do that. So it's not
        //   enough to have validated the auth token here, we will need to pepper the scheduler/task
        //   code with checks to make sure that the player objid is valid before letting it go

but we have no real way to signal to this loop that it should newline for

cleanliness. Need to figure out something for this.

    let mut rl = DefaultEditor::new().unwrap();
    loop {
        // TODO(rdaum): unprovoked output from the narrative stream screws up the prompt midstream,
        //   but we have no real way to signal to this loop that it should newline for
        //   cleanliness. Need to figure out something for this.
        let input_request_id = input_request_id.lock().unwrap().take();

This is both untested and mostly unimplemented. Forked tasks are unlikely to be currently doing any of the following properly:

• @list / view of the verb code
• disassemble builtin output
• Reporting line numbers correctly in tracebacks

Not sure what I'm missing in the command matching cycle, but commands on feature objects don't seem to be getting matched.

/ the line number spans in the bytecode.

    /// Note that this is not necessarily the same as the line number that will be reported into
    /// codegen, and may not correspond to what shows as a result of `unparse`; that line number
    /// is derived from the AST, not the parser.
    /// TODO(rdaum): I may or may not keep this field around.
    pub parser_line_no: usize,
    /// This line number is generated during a second pass over the tree, and is used to generate
    /// the line number spans in the bytecode.

The tasks in the scheduler are currently not persisted. Problem because:

• Restarts of the daemon for upgrades, etc. will kill background tasks without recovery on restart, interrupting the user experience.
• LambdaMOO does do this, so we should be doing it for compatibility.

scheduler.rs and task.rs will need some reworking to synchronize their state out to WorldState, which will need new interfaces for this.

And for compatibility, we might need to support the textdump'd background task set from LambdaMOO textdump imports.

LambdaMOO:
=> "Thu Sep 28 13:53:54 2023 PDT"
vs moor:
=> "Thu, 28 Sep 2023 20:53:39 +0000"

To be compliant with the MOO spec for ctime, this should a) return the timezone descriptor instead of offset and b) use the local timezone, not UTC.

Symptoms are that some internal core functions that parse the ctime don't work:

@shutdown
Do you really want to shut down the server in 2 minutes? [Enter `yes' or `no']
yes
#39:english_time (line 22): Range error
... called from #51:@shutdown (this == #2) (line 31)
(End of traceback)

It would be nice to try providing an alternative to the rocksdb db implementation that is based around a relational database.

When I first started, I used SQLite for this -- and in fact used SQL's transitive closure / recursive joins feature for inherited verb resolution even -- but it got awkward as I was hacking heavily, so it got pulled out.

But I'd like to maybe revisit this, with a Postgres (and maybe other) backend, In all likelihood it would need a caching layer in front, but in fact the Rocks stuff really needs this, too. (And the challenge at the caching layer remains transactional consistency, but that's a whole other kettle of fish).

In the end I'm interested in:

• How awkward the generated relations and queries end up being
• What the performance is like

Requires at least estimating object sizes and tracking them per user / owner.
Need some hooks on the WorldState interface.

Needs to dispatch properly to user_created rather than user_connected, and send "** Created " to the user instead of " Connected **".

LambdaMOO does this by comparing max_object() before and after login, but we will have to find a better way because that is bad in all sorts of ways.

If the intended owner of the new object has a property named `ownership_quota' and the value of that property is an integer, then `create()' treats that value

as a "quota". If the quota is less than or equal to zero, then the quota is considered to be exhausted and `create()' raises `E_QUOTA' instead of creating an

object. Otherwise, the quota is decremented and stored back into the `ownership_quota' property as a part of the creation of the new object.

        let owner = (owner != NOTHING).then_some(owner);

        // TODO(rdaum): ownership_quota support
        //    If the intended owner of the new object has a property named `ownership_quota' and the value of that property is an integer, then `create()' treats that value
        //    as a "quota".  If the quota is less than or equal to zero, then the quota is considered to be exhausted and `create()' raises `E_QUOTA' instead of creating an
        //    object.  Otherwise, the quota is decremented and stored back into the `ownership_quota' property as a part of the creation of the new object.
        let attrs = ObjAttrs {
            owner,
            name: None,

Not sure why it was being called this way, but 'tis caused a panic:

2023-09-25T11:50:18.029710411Z thread 'tokio-runtime-worker' panicked at 'overflow when adding duration to instant', library/std/src/time.rs:584:31

argument was: -1090650497