hhu-bsinfo / dxram Goto Github PK

Separate repository with source + instructions

The job service needs some refactoring and testing. We haven't used it for more than a year and I assume some things are currently broken.
There is also an important TODO that needs to be taken care of to avoid blocking message handlers:
"use event system here to avoid blocking the network thread for too long?" (in JobService, incomingJobEventTriggeredMessage)

Implement a lock free version of a work stealing queue

ChunkService:
If a remove message arrives from a remote node, the message handler has to send a request to the superpeer overlay to remove the chunk from the b-tree. However, the response to that request cannot be processed because it requires another message handler. If the message handler count is set to 1, no handler is available and the request will always fail.

Solution:
Put remove operations as "jobs" (not related to the job system) into a queue and handle them by a separate remove thread (-> separate service: ChunkRemoveService). On incoming remove messages, just put a job into the queue and return. The job gets processed by the dedicated remove thread to avoid sending the request in a message handler and blocking it

• Check available amount of RAM before allocating the SmallObjectHeap -> error and abort if system does not have enough RAM installed
• Warning: If less than 1GB available to the system after SmallObjectHeap init
• Use MonitoringComponent

Reserve operation returns a LID (i.e. higher 16 bits are 0) without NID (memory is allocated later via createReserved). However create or even createLocal does return CIDs with the valid NID.

Expected behavior:
reserve does return CIDs not just NID, otherwise it cannot be used by remote peers.

Similar to a DXRAM service, a test module can access components to execute tests to verify the component is working. A main class (DXRAMMainTest) can call these test modules by providing parameters on execution (e.g. ChunkTest).

Barriers seem to be implemented using dynamic message receivers, which block until a specific message arrives. Whenever someone creates two of them, the second overwrites the firsts subscription. In addition a semaphore is used, which results in blocking one of the available message handler threads.

Looks like you got a bug or some weird code in DynamicMultiByteUnsignedInteger in the below method. The pos is never changed, the while condition is always true. And I think that the if condition is always true as well because I don't see how the operation of x & 0x80 will result in a value equal to 1

public static long fromMultiByte(final byte[] p_array, final int p_arrayOffset) {
        int pos = 0;
        long ret = 0;
        
        while (pos < 9) {
            ret |= (long) (p_array[p_arrayOffset + pos] & 0x7F) << 7 * pos;
            if ((p_array[p_arrayOffset + pos] & 0x80) != 1) {
                break;
            }
        }

        return ret;
    }

Create MonitorComponent and MonitorService to get CPU load, RAM usage, network load etc. (use and code of the bachelor thesis)

Redirect log messages to network and output on the terminal

In Class Create and CreateLocal are the following methods:
`
public int createSizes(final long[] p_cids, final int p_offset, int... p_sizes){...}

public int createSizes(final long[] p_cids, int... p_sizes){...}
`

It compiles but when you try to you it java cannot reference a method because both will fit.

Example:

...createSizes( ..., (int) 3, (int) 4)

In this case java dont know which method i want to use because two Integer could be interpreted as int and an array with one entry or as an array with two entries.

https://github.com/hhu-bsinfo/dxram/blob/master/src/main/java/de/hhu/bsinfo/dxram/app/ApplicationRunner.java

• SmallObjectHeapRecorder: Record alloc/free calls
• SmallObjectHeapPlayer/Debugger: Play recorded pattern, option to check heap after every operation to detect memory corruption of the allocator
• Apply this idea to the MemoryComponent level as well