tsung-wei-huang / dtcraft Goto Github PK

A High-performance Cluster Computing Engine

Home Page: https://tsung-wei-huang.github.io/DtCraft/

License: MIT License

Shell 5.03% Makefile 2.01% M4 0.83% C++ 91.47% HTML 0.04% CSS 0.02% JavaScript 0.03% CMake 0.01% C 0.57%

distributed-systems parallel-computing cluster-computing event-driven-programming reactor streaming dataflow-programming stream-processing-engine network-programming

dtcraft's People

Contributors

Stargazers

Watchers

dtcraft's Issues

Support for Vertex Program

The vertex will support running external program (exec) in the release 0.2.2. Example usage is like :

G.vertex().on([](dtc::Vertex& v) { std::cout << "ready for ls\n"; }).program("/bin/ls / -al");

The external program will have DTC_BRIDGES in its environment variable to store the file descriptor mapping for its input/output stream connections with other vertices or so.

dtcraft web engr illinois edu website down

Describe the bug
http://dtcraft.web.engr.illinois.edu account isn't active

To Reproduce
Steps to reproduce the behavior:

Go to git hub page of the project clicking on the DtCraft link in the top left
Click on Quickstart
See "This account isn't active." error

Expected behavior
To get your first DtCraft program up and running, visit QuickStart and Write the First DtCraft Program.

Hopefully it is just a misunderstanding with the managers of illinois.edu

Fail to run master/agent under gcc 7.1.0 in centos 7

By following the instruction of DtCraft Distributed Mode (Single Host),
I tried to run bin/master and bin/agent, however, it crashed.

$ ./master
terminate called after throwing an instance of 'std::system_error'
what(): unshare failed: Invalid argument
Abort (core dumped)

Compilation fails on linux-centos7-cascadelake

I am getting the compilation error on linux-centos7-cascadelake platform with [email protected] compiler.

Attaching error log.
mk.log

Cheers,
Samir Shaikh

Ver 0.2.2 error in single-host distributed mode

Updated to ver. 0.2.2. Local mode works fine. But the encountered following error in single-host distributed mode. Error message in the master:

frankliu@arlz010:~/Projects/DtCraft$ sudo ./bin/dtc-master           
I  6016 2018-05-29 20:22:09 master.cpp:200] Master @127.0.0.1 [agent:9909|graph:9910|webui:9912]                                          
I  6016 2018-05-29 20:22:42 master.cpp:271] Agent 0 connected @127.0.0.1 [cpu:0|mem:135081242624|disk:17340207104]                        
I  6016 2018-05-29 20:23:20 master.cpp:300] Graph 0 connected @arlz010 [vertex:4|stream:6|container:4]                                    
W  6016 2018-05-29 20:23:20 master.cpp:303] Graph 0 doesn't fit with available resources                                                  
I  6016 2018-05-29 20:23:20 master.cpp:142] Graph 0 is removed from the master                                                            
I  6016 2018-05-29 20:23:30 master.cpp:300] Graph 1 connected @arlz010 [vertex:2|stream:2|container:2]                                    
W  6016 2018-05-29 20:23:30 master.cpp:303] Graph 1 doesn't fit with available resources                                                  
I  6016 2018-05-29 20:23:30 master.cpp:142] Graph 1 is removed from the master

Error message in the submission window:

frankliu@arlz010:~/Projects/DtCraft$ sbin/submit.sh --master=127.0.0.1 /home/frankliu/Projects/DtCraft/example/hello_world                
I  1920 2018-05-29 20:26:33 executor.cpp:159] Executor @arlz010 [stdout:38211|stderr:38045]                                               
I  1920 2018-05-29 20:26:33 executor.cpp:161] Submit graph to master @127.0.0.1:9910                                                      
I 55040 2018-05-29 20:26:33 executor.cpp:173] Solution received      
[Graph 4]                         
+----+-----+------+-----------+-------------------+                  
|Task|Agent|Status|Elapsed (s)|Memory (peak/limit)|                  
+----+-----+------+-----------+-------------------+                  
Graph finished with 1 error(s): Resource request doesn't fit in cluster

OS: Ubuntu 17.10

Tutorial for Stream Graph

We will add a chapter to our cookbook to give a tutorial of the stream graph.

Catch exceptions in main()?

I expect that exception handling is usually supported by a C++ program. I wonder why your function “main” does not contain corresponding try and catch instructions so far.

How do you think about recommendations by Matthew Wilson in an article?

Would you like to adjust the implementation if you consider effects for uncaught/unhandled exceptions?

Control group support

The current master branch is adding the support for Linux control group. Launching agent requires sudo permission.

unittest/archive.sh fail

Centos7, Linux 3.10.0-514.21.1.el7.x86_64
gcc 7.1.0

=====================================

DtCraft 0.2.1: ./test-suite.log

TOTAL: 9

PASS: 8

SKIP: 0

XFAIL: 0

FAIL: 1

XPASS: 0

ERROR: 0

.. contents:: :depth: 2

FAIL: unittest/archive.sh

0.010 s: ArchiverTest.BinaryPOD
0.020 s: ArchiverTest.BinaryStruct
0.023 s: ArchiverTest.BinaryBasicString
1.854 s: ArchiverTest.BinaryVector
1.155 s: ArchiverTest.BinaryDeque
1.573 s: ArchiverTest.BinaryList
1.573 s: ArchiverTest.BinaryForwardList
2.374 s: ArchiverTest.BinaryMap
2.053 s: ArchiverTest.BinaryUnorderedMap
1.608 s: ArchiverTest.BinarySet
1.524 s: ArchiverTest.BinaryArray
0.002 s: ArchiverTest.BinaryVariant
0.003 s: ArchiverTest.SmartPointer
0.002 s: ArchiverTest.ErrorCode

lt-archive is a Catch v2.0.1 host application.
Run with -? for options
-------------------------------------------------------------------------------
ArchiverTest.BinaryAtomicity
-------------------------------------------------------------------------------
unittest/archive.cpp:764
...............................................................................

unittest/archive.cpp:764: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGTERM - Termination request signal

0.000 s: ArchiverTest.BinaryAtomicity
===============================================================================
test cases:    15 |    14 passed | 1 failed
assertions: 98226 | 98225 passed | 1 failed

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/kernel.sh --action=start
start master @frankliu@arlz010:/home/frankliu/Projects/DtCraft [OK]
start agent @frankliu@arlz010:/home/frankliu/Projects/DtCraft [OK]
start agent @frankliu@arlz009:/home/frankliu/Projects/DtCraft [OK]

But returned results from ping are wrong:

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/kernel.sh --action=ping
master @frankliu@arlz010:/home/frankliu/Projects/DtCraft [STOPPED]
agent @frankliu@arlz009:/home/frankliu/Projects/DtCraft [RUNNING]
agent @frankliu@arlz010:/home/frankliu/Projects/DtCraft [STOPPED]

The cluster appears to be alive and functional:

frankliu@arlz010:~/Projects/DtCraft$ ./sbin/submit.sh --master=arlz010 /home/frankliu/Projects/DtCraft/example/pi
I 47872 2018-05-29 20:46:30 executor.cpp:186] Executor @arlz010 [stdout:36041|stderr:34575]
I 47872 2018-05-29 20:46:30 executor.cpp:191] Submitting graph to master @arlz010:9910
pi estimation: 3.14146
I 59136 2018-05-29 20:46:32 executor.cpp:200] Solution received
[Graph 5]
+--------+------+-------+
|Topology|Status|Message|
+--------+------+-------+
|5-3     |Exit 0|Success|
+--------+------+-------+
|5-2     |Exit 0|Success|
+--------+------+-------+
|5-0     |Exit 0|Success|
+--------+------+-------+
|5-1     |Exit 0|Success|
+--------+------+-------+
Graph finished with 0 error(s)

Tried on the other node, works fine!