sesa / ebbrt Goto Github PK

When running multi-core w/ qemu, we must extend the default system memory (using qemu's -m flag) or face unexpected crush. Alternatively, the system should assert when the loaded memory is insufficient.

Let's reserve a room for a couple of hours to review and discuss portions of the core EbbRT code up on the projector. This can be useful in getting (re)acquainted with the underlying components of EbbRT and the design decisions / implications which accompany them.

Instead of having a single-presenter or a targeted region of code, I suggest this be held as a free-form discussion. For example, we begin with a high-level interface (e.g., gp_allocator->Alloc(size)) and follow whichever code+question path we decide is most interesting.

In several places we synchronously invoke callbacks which is not right. These should all be turned to either be asynchronous (through the new EventManager interface), or should use futures (which use the new EventManager interface under the cover)

We use std::map liberally in both the DistributedRoot and EbbManager. These data structures are used on multiple cores and so must be thread safe. The std::map structure is not inherently thread safe and so we currently lock around accesses. Right now we use a spinlock. A relatively easy improvement would be a reader/writer lock instead of spinlock.

Longer term we should think about building good data structures for this:

http://www.youtube.com/watch?v=HJ-719EGIts
http://code.google.com/p/nbds/

We need a simple way to quiet all debug print output when building back-end Release. I'm thinking a pre-processor define target within the app. We'll also need a way for the programmer to specify debug-only and required prints. For example: #define __EBBRT_DEBUG_PRINT_ON__ 1

Would be nice to have suite of test cases for ebbrt to pass before merging pull requests.

Two calls required to define a new ebb: alloc ID, bind to factory. Add a placeholder C++ object to encapsulate this process

Currently async() does not flatten a Future<Future > to a Future. I believe it should flatten and we should provide a function for users to do so.

Bugs possible when dealing with messages longer-than one buffer.
TODO: confirm this.

The location of the backend .elf images relative to the frontend binary have been hard coded in the app's source and we've relied on the app's build system to put the binaries in the correct locations. Preferably, we provide a method for selecting the binary at runtime. For example, a call to node-allocator->LoadBinary(std::string path); accepts a pathname, but can also prompt the user to enter the path at the time of invocation if the argument is empty.

Hey! I was trying to build an EbbRT dependent application, so I followed the instructions given in https://github.com/SESA/EbbRT/wiki/Build-Tutorial-Notes. While building the helloworld application, I got this error:

-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
System is unknown to cmake, create:
Platform/EbbRT to use this system, please send your config file to [email protected] so it can be added to cmake
Your CMakeCache.txt file was copied to CopyOfCMakeCache.txt. Please send that file to [email protected].
CMake Error at CMakeLists.txt:2 (project):
The CMAKE_C_COMPILER:

/home/user/ebbrt-sysroot/ebbrt/usr/bin/x86_64-pc-ebbrt-gcc

is not a full path to an existing compiler tool.

Tell CMake where to find the compiler by setting either the environment
variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
the compiler, or to the compiler name if it is in the PATH.

CMake Error at CMakeLists.txt:2 (project):
The CMAKE_CXX_COMPILER:

/home/user/ebbrt-sysroot/ebbrt/usr/bin/x86_64-pc-ebbrt-g++

is not a full path to an existing compiler tool.

Tell CMake where to find the compiler by setting either the environment
variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
to the compiler, or to the compiler name if it is in the PATH.

-- Configuring incomplete, errors occurred!

If I understood it correctly this means that cmake is trying to find a cross compiler binary "x86_64-pc-ebbrt-gxx" as specified by ebbrt.cmake. However, during the build, no such file was created.
Any pointers on how to solve this would be highly appreciated.

EbbRT will hit an "unimplemented" assert when it tried to free very large regions of memory. Examples to follow.

Header-only changes do not trigger source to be re-built

The current roots are not thread safe, the reps map is concurrently accessed and modified. We either need to use a thread-safe implementation of a map or use locking before accessing the map to avoid this.

Hi All,

I read the paper on EbbRT and wanted to give it a try. Specifically, I would like to run a node application on it. How do I go about doing it? I looked in code/docs but did not any clue.
Any pointers would be appreciated.

Currently we assume that event::Locations which run from zero to n-1 are the same as Apic IDs which is an incorrect assumption on real hardware (this works on VMs). The ACPI tables should be parsed to create this mapping so that SMP startup works.

Currently we statically map in 2 MB for the local translation table. This is insufficient due to the partitioning of the ID space, they should be faulted in through page faults.

I haven't finish reading all the code yet, and C++ is a great language, but still wonder if it can be done only with C or other programming language?

Sorry if it's a dumb question.

Future references aren't supported yet.

Current libstdc++ does not build with threading support. If we add a few more functions to the gthread library then we can enable it by defining __GTHREADS_CXX0X to 1

Name the docker networks/containers created by the NodeAllocator (e.g., ebbrt_net_XXXX).
Doing so will more easily allows scripting of the EbbRT-created docker state. For example, a script which can force remove all EbbRT networks & nodes.

Currently the primitive memory allocator ebb does not actually support freeing memory.

Bug exists where VM container will be allocated despite there being no network.

This occurs when there are 12+ running docker networks

Currently we call LRT_ASSERT on baremetal, but we can change the assert provided by newlib to call our own noreturn function to handle a failed assert. This way our asserts are the same on linux and baremetal

I am working on a fix... using this to learn about how to push a fix

BUG: Release hosted build can expose a initialization bug in the NodeAllocator that will result in defaults not being properly initialized when the first node allocation is done such that a node allocate with no parameters will lead to qemu being invoked with bad arguments:

+ exec qemu-system-x86_64 -m 0G -smp cpus=0 -cpu host -serial stdio -display none -enable-kvm -netdev tap,script=no,downscript=no,ifname=tap0,id=net0,vhost=on,queues=0 -device virtio-net-pci,netdev=net0,mac=02:42:ac:14:00:02,mq=on,vectors=2 --gdb tcp:0.0.0.0:1234 -kernel /root/img.elf -append 'host_addr=2886991873;host_port=37807;allocid=0'

Fix is to remove the class wide initialization from the rep creation path and move it to a ClassInit

The C standard assert continues to printf, and so it won't work for us.

Ebb is not actually an Ebb.. its an Ebb reference.

Rename to EbbRef

EbbRT's "front-end" runtime, or perhaps EbbRT-on-Linux, should be exist as a set of shared object libraries and headers installed onto the host operating system. This way an application can link in the EbbRT libraries and interact through interfaces defined in the EbbRT headers. In general, we should strive to follow the "Linux way" of shared library organization, naming, placement, versioning and compatibility.

I image EbbRT built as a collection of object libraries, wherein each library encapsulates one-or-more Ebbs which in turn depend a lower-level runtime library. Application-specific objects can be built as static libraries and decoupled from the host installation.

When we implement this approach it will bring to the forefront interesting issues about how EbbRT libraries should be written and how code can be shared between hosted and native implementations.

Thoughts?

This enhancement would address issue #94