tsung-wei-huang / dtcraft Goto Github PK

A High-performance Cluster Computing Engine

Home Page: https://tsung-wei-huang.github.io/DtCraft/

License: MIT License

Shell 5.03% Makefile 2.01% M4 0.83% C++ 91.47% HTML 0.04% CSS 0.02% JavaScript 0.03% CMake 0.01% C 0.57%

distributed-systems parallel-computing cluster-computing event-driven-programming reactor streaming dataflow-programming stream-processing-engine network-programming

dtcraft's People

Contributors

Stargazers

Watchers

dtcraft's Issues

Support for executing a graph at a particular host

We will add support for specifying a particular host to run your application. New API will be added to our container builder.

Fail to run master/agent under gcc 7.1.0 in centos 7

By following the instruction of DtCraft Distributed Mode (Single Host),
I tried to run bin/master and bin/agent, however, it crashed.

$ ./master
terminate called after throwing an instance of 'std::system_error'
what(): unshare failed: Invalid argument
Abort (core dumped)

Catch exceptions in main()?

I expect that exception handling is usually supported by a C++ program. I wonder why your function “main” does not contain corresponding try and catch instructions so far.

How do you think about recommendations by Matthew Wilson in an article?

Would you like to adjust the implementation if you consider effects for uncaught/unhandled exceptions?

Compilation fails on linux-centos7-cascadelake

I am getting the compilation error on linux-centos7-cascadelake platform with [email protected] compiler.

Attaching error log.
mk.log

Cheers,
Samir Shaikh

Tutorial for event-driven programming (reactor)

DtCraft has a built-in reactor for event-driven programming. We will add a chapter discussing the basics in our cookbook.

Tutorial for Stream Graph

We will add a chapter to our cookbook to give a tutorial of the stream graph.

Support for Vertex Program

The vertex will support running external program (exec) in the release 0.2.2. Example usage is like :

G.vertex().on([](dtc::Vertex& v) { std::cout << "ready for ls\n"; }).program("/bin/ls / -al");

The external program will have DTC_BRIDGES in its environment variable to store the file descriptor mapping for its input/output stream connections with other vertices or so.

unittest/archive.sh fail

Centos7, Linux 3.10.0-514.21.1.el7.x86_64
gcc 7.1.0

=====================================

DtCraft 0.2.1: ./test-suite.log

TOTAL: 9

PASS: 8

SKIP: 0

XFAIL: 0

FAIL: 1

XPASS: 0

ERROR: 0

.. contents:: :depth: 2

FAIL: unittest/archive.sh

0.010 s: ArchiverTest.BinaryPOD
0.020 s: ArchiverTest.BinaryStruct
0.023 s: ArchiverTest.BinaryBasicString
1.854 s: ArchiverTest.BinaryVector
1.155 s: ArchiverTest.BinaryDeque
1.573 s: ArchiverTest.BinaryList
1.573 s: ArchiverTest.BinaryForwardList
2.374 s: ArchiverTest.BinaryMap
2.053 s: ArchiverTest.BinaryUnorderedMap
1.608 s: ArchiverTest.BinarySet
1.524 s: ArchiverTest.BinaryArray
0.002 s: ArchiverTest.BinaryVariant
0.003 s: ArchiverTest.SmartPointer
0.002 s: ArchiverTest.ErrorCode

lt-archive is a Catch v2.0.1 host application.
Run with -? for options
-------------------------------------------------------------------------------
ArchiverTest.BinaryAtomicity
-------------------------------------------------------------------------------
unittest/archive.cpp:764
...............................................................................

unittest/archive.cpp:764: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGTERM - Termination request signal

0.000 s: ArchiverTest.BinaryAtomicity
===============================================================================
test cases:    15 |    14 passed | 1 failed
assertions: 98226 | 98225 passed | 1 failed

Control group support

The current master branch is adding the support for Linux control group. Launching agent requires sudo permission.

Wrong status message in cluster mode

Sequence of commands:

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/kernel.sh --action=start
start master @frankliu@arlz010:/home/frankliu/Projects/DtCraft [OK]
start agent @frankliu@arlz010:/home/frankliu/Projects/DtCraft [OK]
start agent @frankliu@arlz009:/home/frankliu/Projects/DtCraft [OK]

But returned results from ping are wrong:

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/kernel.sh --action=ping
master @frankliu@arlz010:/home/frankliu/Projects/DtCraft [STOPPED]
agent @frankliu@arlz009:/home/frankliu/Projects/DtCraft [RUNNING]
agent @frankliu@arlz010:/home/frankliu/Projects/DtCraft [STOPPED]

The cluster appears to be alive and functional:

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/submit.sh --master=arlz010 /home/frankliu/Projects/DtCraft/example/pi
I 47872 2018-05-29 20:46:30 executor.cpp:186] Executor @arlz010 [stdout:36041|stderr:34575]
I 47872 2018-05-29 20:46:30 executor.cpp:191] Submitting graph to master @arlz010:9910
pi estimation: 3.14146
I 59136 2018-05-29 20:46:32 executor.cpp:200] Solution received
[Graph 5]
+--------+------+-------+
|Topology|Status|Message|
+--------+------+-------+
|5-3     |Exit 0|Success|
+--------+------+-------+
|5-2     |Exit 0|Success|
+--------+------+-------+
|5-0     |Exit 0|Success|
+--------+------+-------+
|5-1     |Exit 0|Success|
+--------+------+-------+
Graph finished with 0 error(s)

Tried on the other node, works fine!

dtcraft web engr illinois edu website down

Describe the bug
http://dtcraft.web.engr.illinois.edu account isn't active

To Reproduce
Steps to reproduce the behavior:

Go to git hub page of the project clicking on the DtCraft link in the top left
Click on Quickstart
See "This account isn't active." error

Expected behavior
To get your first DtCraft program up and running, visit QuickStart and Write the First DtCraft Program.

Hopefully it is just a misunderstanding with the managers of illinois.edu

Tutorial for input/output stream interface

We will add a chapter to our cookbook discussing hw to use our input and output stream events.

Ver 0.2.2 error in single-host distributed mode

Updated to ver. 0.2.2. Local mode works fine. But the encountered following error in single-host distributed mode. Error message in the master:

frankliu@arlz010:~/Projects/DtCraft$ sudo ./bin/dtc-master           
I  6016 2018-05-29 20:22:09 master.cpp:200] Master @127.0.0.1 [agent:9909|graph:9910|webui:9912]                                          
I  6016 2018-05-29 20:22:42 master.cpp:271] Agent 0 connected @127.0.0.1 [cpu:0|mem:135081242624|disk:17340207104]                        
I  6016 2018-05-29 20:23:20 master.cpp:300] Graph 0 connected @arlz010 [vertex:4|stream:6|container:4]                                    
W  6016 2018-05-29 20:23:20 master.cpp:303] Graph 0 doesn't fit with available resources                                                  
I  6016 2018-05-29 20:23:20 master.cpp:142] Graph 0 is removed from the master                                                            
I  6016 2018-05-29 20:23:30 master.cpp:300] Graph 1 connected @arlz010 [vertex:2|stream:2|container:2]                                    
W  6016 2018-05-29 20:23:30 master.cpp:303] Graph 1 doesn't fit with available resources                                                  
I  6016 2018-05-29 20:23:30 master.cpp:142] Graph 1 is removed from the master

Error message in the submission window:

frankliu@arlz010:~/Projects/DtCraft$ sbin/submit.sh --master=127.0.0.1 /home/frankliu/Projects/DtCraft/example/hello_world                
I  1920 2018-05-29 20:26:33 executor.cpp:159] Executor @arlz010 [stdout:38211|stderr:38045]                                               
I  1920 2018-05-29 20:26:33 executor.cpp:161] Submit graph to master @127.0.0.1:9910                                                      
I 55040 2018-05-29 20:26:33 executor.cpp:173] Solution received      
[Graph 4]                         
+----+-----+------+-----------+-------------------+                  
|Task|Agent|Status|Elapsed (s)|Memory (peak/limit)|                  
+----+-----+------+-----------+-------------------+                  
Graph finished with 1 error(s): Resource request doesn't fit in cluster

OS: Ubuntu 17.10