archseer / enigma Goto Github PK

Timers includes:

• bif erlang:start_timer/3
• bif erlang:start_timer/4
• bif erlang:send_after/3
• bif erlang:send_after/4
• bif erlang:cancel_timer/1
• bif erlang:cancel_timer/2
• bif erlang:read_timer/1
• bif erlang:read_timer/2

As well as WaitTimeout and Send with timeout instructions. The implementation should use a hierarchical timing wheel, can probably base the implementation on https://github.com/tokio-rs/tokio/tree/master/tokio-timer/src/wheel

If we decide to integrate with Tokio for the async I/O (yields the BIF when blocking), we could avoid copying the impl. If we copy it, we probably want a new thread that will keep track of the wheel and schedule processes when their timers are up (similar to what process::spawn internals do)

I think it is right time to get the travis involved.
At first, I think we can build and test with stable version of rust.
The pull request is required to pass all the unit tests then.

Update the module registry to store RcModule (Arc) instead of Box, and update all *const Module to be RcModule. Loading the same module into the VM should swap it in the registry, meaning all new module registry lookups will return the new module. Meanwhile, references to the old Module will still work until they all die, at which point the module will be dropped.

After Issue #34 was resolved, I was able to update otp and start the build. However, the build failed due to the following errors:

=== Entering application tools
make[3]: Entering directory '/home/yima/play/enigma/otp/lib/tools/c_src'
MAKE opt
make[4]: Entering directory '/home/yima/play/enigma/otp/lib/tools/c_src'
CC ../obj/x86_64-unknown-linux-gnu/opt/emem/erl_memory.o
CC ../obj/x86_64-unknown-linux-gnu/opt/emem/erl_memory_trace_block_table.o
LD ../bin/x86_64-unknown-linux-gnu/emem
/usr/bin/ld: cannot find -lerts_r
/usr/bin/ld: cannot find -lethread
/usr/bin/ld: cannot find -lerts_internal_r
collect2: error: ld returned 1 exit status
x86_64-unknown-linux-gnu/Makefile:182: recipe for target '../bin/x86_64-unknown-linux-gnu/emem' failed
make[4]: *** [../bin/x86_64-unknown-linux-gnu/emem] Error 1

We need to place stubs in the exports registry for modules that aren't loaded yet, but referenced elsewhere. These should call the error handler, (which in the erlang stdlib will try to load the module or error).

http://erlang.org/doc/man/error_handler.html

Search for references to error_handler inside beam_emu.c and enigma for a quickstart.

It's time to get all these out of the way.

x_to_list

• erlang:atom_to_list/1
• erlang:binary_to_list/1
• erlang:binary_to_list/3
• erlang:float_to_list/1
• erlang:float_to_list/2
• erlang:integer_to_list/1
• erlang:integer_to_list/2
• erlang:pid_to_list/1
• erlang:tuple_to_list/1
• erlang:ref_to_list/1
• erlang:port_to_list/1
• erlang:fun_to_list/1
• erlang:bitstring_to_list/1

list_to_x

• erlang:list_to_atom/1
• erlang:list_to_binary/1
• erlang:list_to_float/1
• erlang:list_to_integer/1
• erlang:list_to_pid/1
• erlang:list_to_port/1
• erlang:list_to_ref/1
• erlang:list_to_tuple/1
• string:list_to_integer/1
• erlang:list_to_integer/2
• string:list_to_float/1
• erlang:list_to_existing_atom/1
• erlang:list_to_bitstring/1

binary_to_x

• erlang:binary_to_float/1
• erlang:binary_to_term/1
• erlang:binary_to_atom/2
• erlang:binary_to_existing_atom/2
• erlang:binary_to_integer/1
• erlang:binary_to_integer/2

x_to_binary

• erlang:atom_to_binary/2
• erlang:float_to_binary/1
• erlang:float_to_binary/2
• erlang:integer_to_binary/1
• erlang:integer_to_binary/2
• erlang:term_to_binary/1
• erlang:term_to_binary/2

Special

• unicode:characters_to_binary/2
• unicode:characters_to_list/2
• erlang:iolist_to_binary/1
• erlang:iolist_to_iovec/1

• ubif erlang:’and’/2
• ubif erlang:’or’/2
• ubif erlang:’xor’/2
• ubif erlang:’not’/1
• ubif erlang:’>’/2 sgt_2
• ubif erlang:’>=’/2 sge_2
• ubif erlang:’<’/2 slt_2
• ubif erlang:’=<’/2 sle_2
• ubif erlang:’:‘/2 seq_2
• ubif erlang:’==’/2 seqeq_2
• ubif erlang:’/‘/2 sneq_2
• ubif erlang:’/=’/2 sneqeq_2

Look at the related opcode implementations (Opcode::IsLe, etc.)

Some of the unit tests are not implemented yet.
Required someone to fill them up.

• Math functions
• List functions
• Chrono functions

Here are the incomplete unit tests (without testing the exception case).

• erlang functions

We currently only do (partially) ETF decoding. This isn't important just yet, but we'll need to implement encoding in the future. We should aim to implement an encoder inside etf.rs, then expose that as a serde protocol. Take inspiration from https://github.com/obmarg/serde_eetf

Currently, some of the bif implementation is written in bif.rs and some of them are in bif folder.
We can gather them into bif folder instead.
Then, bif.rs can be served as an index of the current bif and file in bif are the implementation with unit tests.

• erlang function
• math function
• list function

• bif erlang:make_tuple/2
• bif erlang:append_element/2
• bif erlang:make_tuple/3
• bif erlang:setelement/3
• bif erlang:tuple_to_list/1

Most of these can be done in a very similar way to the tup macros:

For setelement/any functions that copy the old tuple, we can 1) allocate a new tuple, 2) use std::ptr::copy_nonoverlapping to do a fast memcpy of the old tuple to the new tuple 3) set the new elements

I think we can trim the testing code with a macro for calling the bif function.
During my development, the following codes were repeatedly entered.

let vm = vm::Machine::new();
let module: *const module::Module = std::ptr::null();
let process = process::allocate(&vm.state, module).wrap();
let args = vec![xxxxxxx];
let res = bif_the_function_I_want_to_test(&vm, &process, &args);

Most of the codes are not used for testing. The code duplication is so high that I think it can be replaced by a macro as a high order function like the following one.

macro_rules! test_it(func: func ; $($arg: expr), *) => {
  // The code as the above
}

I can do this in Elixir but not sure if it is valid in Rust.
Does it sound a valid solution to you? @archseer
If it is okay, I will do it and refactor the unit tests.

Hi @archseer,

I'm very new to Rust (familiar with Erlang), so please forgive me.
I am attempting to compile Enigma and receiving the following messageto use a constant of type value::Term in a pattern, value::Term must be annotated with #[derive(PartialEq, Eq)].
From what I can tell this seems to be related to the APPLY_2 const but again, I'm not familiar at all with Rust.
Any idea what is causing this?

rustc 1.39.0-nightly (72b2abfd6 2019-08-29)
rustup 1.18.3 (435397f48 2019-05-22)
cargo 1.39.0-nightly (22f7dd049 2019-08-27)

Beam implements a transform engine that transforms beam opcodes during the load phase, that way they can implement optimizations that are independent of the OTP version.

https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/beam_makeops.md

The problem is, it's a large perl script that generates a bunch of C arrays. The loader then uses that array as a series of simple instructions in state machine. The state machine part should be easy, for the generator, we might just fork the perl script to generate rust arrays. Or implement some form of a rust codegen.

https://github.com/erlang/otp/blob/49eb25c80514b1e221cc516c423bbc45ba7cc107/erts/emulator/beam/beam_load.c#L5207

This is a time consuming issue, but will yield a lot of performance improvements (optimize bif calls, instruction combining, etc).

ETS: https://docs.rs/chashmap for db_hash based tables.

Probably either immix or orca. Each process gets it's own heap consisting of blocks that we independently GC.

At the moment we just use blocks as a sort of an arena GC: they belong to a process and get deallocated when a process drops (though no destructors are called, and we need to ensure the process unregisters in the process registry!!!)

Immix

• http://www.cs.utexas.edu/users/speedway/DaCapo/papers/immix-pldi-2008.pdf
• https://www.cs.tufts.edu/~nr/cs257/archive/steve-blackburn/rust-ismm-2016.pdf
• https://gitlab.com/inko-lang/inko/tree/master/vm/src/immix

Orca

• https://www.ponylang.io/media/papers/orca_gc_and_type_system_co-design_for_actor_languages.pdf

Golang GC

• https://blog.golang.org/ismmkeynote
• https://github.com/golang/go/blob/master/src/runtime/mgc.go

Hi @archseer,
it seems the code in maps is being rebased.
Some of the implementation is cleared. Can you please help fix the problem?
Thank you!

Running cargo test from root directory results in the following error:

warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /home/cgm/devel/rust/enigma/enigma/Cargo.toml
workspace: /home/cgm/devel/rust/enigma/Cargo.toml
   Compiling quickcheck v0.9.0
   Compiling instruction-codegen v0.1.0 (/home/cgm/devel/rust/enigma/instruction-codegen)
error[E0433]: failed to resolve: use of undeclared type or module `trybuild`
 --> instruction-codegen/tests/progress.rs:3:13
  |
3 |     let t = trybuild::TestCases::new();
  |             ^^^^^^^^ use of undeclared type or module `trybuild`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0433`.
error: could not compile `instruction-codegen`.
warning: build failed, waiting for other jobs to finish...
error[E0432]: unresolved import `instruction`
  --> instruction-codegen/tests/01-parse-header.rs:23:5
   |
23 | use instruction::ins;
   |     ^^^^^^^^^^^ use of undeclared type or module `instruction`

error: cannot determine resolution for the macro `ins`
  --> instruction-codegen/tests/01-parse-header.rs:25:1
   |
25 | ins!(
   | ^^^
   |
   = note: import resolution is stuck, try simplifying macro imports

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0432`.
error: could not compile `instruction-codegen`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Seems a similar problem to #34

When I run git submodule update --init -depth 1 I get the following error:

Submodule 'otp' (https://github.com/erlang/otp) registered for path 'otp'
Cloning into '/home/cgm/devel/rust/enigma/otp'...
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
error: Server does not allow request for unadvertised object 9c35d42191003d1325ad99189d7327f87c9f6fd4
Fetched in submodule path 'otp', but it did not contain 9c35d42191003d1325ad99189d7327f87c9f6fd4. Direct fetching of that commit failed.

Monitors & links

• Track parent process
• Add a link/monitor tree to each process
• Modify process::spawn to accept extra flags which register the required tree structures
• On process exit, dispatch exit signals

Signal queue

• Modify mailbox to hold a queue of enum Signal { Message(Term), Exit(Reason), ... }
• Check the mailbox on each process swap-in / (swap-out?)

We currently rely on some parts of std, but having a no_std build would be interesting for embedded devices and WebAssembly. Note that a lot of the std stuff exists in core, so we could get far with just conditional imports.

https://github.com/serde-rs/serde/blob/bb99b31eb0a55393101f9c80cd959b3a739ad70f/serde/src/lib.rs#L141-L143

• Fix all the imports to import core:: + alloc versions on the no_std feature flag.
• Make sure all the dependencies are no_std capable.
• Move any OS specific code to a sys subfolder.
• Abstract the tokio runtime so we can switch to juliex or any other no_std capable executor. Look into fuchsia's executor as an example.

When I ran "cargo run", once it got into compiling enigma, it encountered multiple errors, such as:

error[E0432]: unresolved import byteorder
--> src/etf.rs:313:5
|
313 | use byteorder::{BigEndian, WriteBytesExt};
| ^^^^^^^^^ help: a similar path exists: libflate::byteorder

error[E0433]: failed to resolve: use of undeclared type or module md5
--> src/bif/erlang.rs:14:18
|
14 | let digest = md5::compute(bytes);
| ^^^ use of undeclared type or module md5
error[E0599]: no method named write_u8 found for type &mut std::vec::Vec<u8> in the current scope
--> src/etf.rs:330:21
|
330 | res.write_u8(i as u8)?;
| ^^^^^^^^
|
= help: items from traits can only be used if the trait is in scope
= note: the following trait is implemented but not in scope, perhaps add a use for it:
use byteorder::io::WriteBytesExt;

The last error occurs on multiple lines in etf.rs.

In addition, many warning about unused import and macro definition, such as:

warning: unused macro definition
--> src/exception.rs:208:1
|
208 | / macro_rules! native_exception {
209 | | ($x:expr) => {
210 | | $x & Reason::EXF_NATIVE
211 | | };
212 | | }
| |_^

warning: unused import: BigInt
--> src/bif.rs:11:26
|
11 | use crate::value::{self, BigInt, Cons, Term, CastFrom, CastInto, Tuple, Variant};
| ^^^^^^
|
= note: #[warn(unused_imports)] on by default

And warnings about deprecated features, such as:

warning: ... range patterns are deprecated
--> src/bif/erlang.rs:230:51
|
230 | Variant::Integer(i @ 0...255) => {
| ^^^ help: use ..= for an inclusive range
|
= note: #[warn(ellipsis_inclusive_range_patterns)] on by default
warning: trait objects without an explicit dyn are deprecated
--> src/ets/error.rs:88:32
|
88 | fn cause(&self) -> Option<&StdError> {
| ^^^^^^^^ help: use dyn: dyn StdError

I am getting this when I follow the instruction to update otp. Do we have a workaround?

~/play/enigma$ git submodule update
fatal: reference is not a tree: 7bcc044cb00b8dfb8eb2eb220dbda8a395af72a7
Unable to checkout '7bcc044cb00b8dfb8eb2eb220dbda8a395af72a7' in submodule path 'otp'

• Binary matching opcodes
• Binary building opcodes
• NIFs

This issue concerns the re module.

We want to use the regex crate, because it's mostly PCRE compatible, but drops lookaheads and some other features to make it execute in linear time. This way we'd be able to estimate the reductions based on the pattern and binary length.

This issue first requires binaries to be implemented (refs #20).

Just saw this project on hackernews and noticed that to_opcode is a problem. If the input falls outside of the opcode range it produces undefined behavior. This probably needs to either assert with a panic or return Option for unknown opcodes.