zeroc-ice / datastorm Goto Github PK

View Code? Open in Web Editor NEW

13.0 13.0 7.0 1.08 MB

Data centric pub/sub framework based on Ice

License: GNU General Public License v2.0

Makefile 0.54% Python 31.39% C 0.21% C++ 66.46% Slice 0.92% C# 0.48%

datastorm's People

Contributors

Stargazers

Watchers

Forkers

bentoi longervision pepone mreinart xionghongli945 tianfangyetan123 insertcreativityhere

datastorm's Issues

DataStorm/api test deadlock

#0  0x00007f5e0418d8dd in __lll_lock_wait () from /usr/lib64/libpthread.so.0
#1  0x00007f5e04186af9 in pthread_mutex_lock () from /usr/lib64/libpthread.so.0
#2  0x00005644e3032dc7 in __gthread_mutex_lock (__mutex=<optimized out>)
    at /usr/include/c++/8/x86_64-redhat-linux/bits/gthr-default.h:748
#3  std::mutex::lock (this=<optimized out>)
    at /usr/include/c++/8/bits/std_mutex.h:103
#4  0x00005644e304f877 in std::lock_guard<std::mutex>::lock_guard (__m=..., 
    this=<synthetic pointer>) at /usr/include/c++/8/bits/std_mutex.h:161
#5  DataStormI::AbstractFactoryT<int, DataStormI::KeyT<int> >::remove (v=0x7f5dcc000da0, this=0x5644e457e678)
    at /usr/include/DataStorm/InternalT.h:253
#6  DataStormI::AbstractFactoryT<int, DataStormI::KeyT<int> >::Deleter::operator() (this=<optimized out>, obj=0x7f5dcc000da0)
    at /usr/include/DataStorm/InternalT.h:165
#7  0x00005644e30339e7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7f5dcc002160) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#8  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (
    this=0x7f5dcc002160) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#9  0x00005644e304f98f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7ffd061d61c8, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/shared_ptr_base.h:1167
#10 std::__shared_ptr<DataStormI::KeyT<int>, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7ffd061d61c0, __in_chrg=<optimized out>)
    at /usr/include/c++/8/bits/shared_ptr_base.h:1167
#11 std::shared_ptr<DataStormI::KeyT<int> >::~shared_ptr (this=0x7ffd061d61c0, 
    __in_chrg=<optimized out>) at /usr/include/c++/8/bits/shared_ptr.h:103
#12 DataStormI::AbstractFactoryT<int, DataStormI::KeyT<int> >::remove (v=0x5644e45a61c0, this=0x5644e457e678)
    at /usr/include/DataStorm/InternalT.h:257
#13 DataStormI::AbstractFactoryT<int, DataStormI::KeyT<int> >::Deleter::operator() (this=<optimized out>, obj=0x5644e45a61c0)
    at /usr/include/DataStorm/InternalT.h:165
#14 0x00005644e30339e7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5644e45a61f0) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#15 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (
    this=0x5644e45a61f0) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#16 0x00007f5e0526ddce in std::vector<std::shared_ptr<DataStormI::Key>, std::alloc--Type <RET> for more, q to quit, c to continue without paging--
ator<std::shared_ptr<DataStormI::Key> > >::~vector() ()
   from /usr/lib64/libDataStorm.so.0
#17 0x00007f5e0527175c in DataStormI::KeyDataWriterI::~KeyDataWriterI() ()
   from /usr/lib64/libDataStorm.so.0
#18 0x00005644e30339e7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5644e4576810) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#19 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (
    this=0x5644e4576810) at /usr/include/c++/8/bits/shared_ptr_base.h:148
#20 0x00005644e3028fd1 in main (argc=<optimized out>, argv=<optimized out>)
    at /usr/include/DataStorm/DataStorm.h:722

Default build should be optimized like for Ice

The default build flags are currently debug, when they should be optimized just like ice.

Tested on amzn2.

cpp/clock demo name truncated when city has a space

❯ ./writer
Please enter city name: Foo Bar

❯ ./reader
received time for `Foo': 10/03/18 11:53:51

demos/cpp/clock does not work with localhost without any network

when I disconnected any network(wifi, local lan), I just can not make the demos/cpp/clock or minimal work with only localhost(loopback).
I tried to change reader.config, writer.config, but can't figure it. I searched through the datastorm docs, no solution.
I tried to switch off firewall, but it still does not work.
Hope for some help, thanks.

I am using Windows10 with VisualStudio2019.

There is a memory leak problem, when broadcasting ICE::Proxy with DataStorm.

There is a memory leak problem, when broadcasting ICE::Proxy with DataStorm.
I have already created an issue for zeroc-ice: #13
As I am not sure if the problem is directly caused by ice or datastorm, so may you bother to have a look at it when you are enjoying free leisure moments.

Compilation failure on Ubuntu Jammy and RedHat 9

++ nproc
+ make -j2
Compiling [amd64-shared] test/DataStorm/reliability/Reader.cpp
test/DataStorm/reliability/Reader.cpp: In function ‘int main(int, char**)’:
test/DataStorm/reliability/Reader.cpp:50:34: error: ‘sleep_for’ is not a member of ‘std::this_thread’
   50 |                     this_thread::sleep_for(chrono::milliseconds(200));
      |                                  ^~~~~~~~~
make: *** [Makefile:27: test/DataStorm/reliability/build/x86_64-linux-gnu/shared/pie/Reader.o] Error 1
script returned exit code 2

partial writer hang on Windows

We got a hang with CI testing... The writer from the partial update test was hang on line 40 waiting for readers... no reader process was running.

protocol?

Is it the DDS protocol? Is opendds the same?

memory leak upon node-crashing

At first, thanks for building this wonderful project which helps me a lot!
Everything worked well with DataStorm, but I did encounter a serious memory leak problem.
I tried to locate the problem, but my understanding stopped at leak location "makeXXXKeyReader"/"makeXXXKeyWriter", so
I build a daemon for this problem, and hope for your insight & help.
The main idea of my daemon is to automatically release specified-key writer when last specified-key reader disconnected.

How to reproduce the problem:
1, build Reader.exe & Writer.exe from https://github.com/zengdaiwei/TestDataStormMemoryLeak.git
2, run Writer.exe
3, run Reader with bash: "while true; do ./Reader & sleep 1s; kill %1; wait; sleep 3s; done"
4, now you can see there is a continuous memory leak from Writer.exe

Thanks

make install installs headers with executable bit set

On RHEL8:

*** WARNING: ./usr/include/DataStorm/InternalT.h is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/include/DataStorm/Config.h is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/include/DataStorm/DataStorm.h is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/include/DataStorm/Node.h is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/include/DataStorm/CtrlCHandler.h is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/include/DataStorm/InternalI.h is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/include/DataStorm/Types.h is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/include/DataStorm/Sample.h is executable but has empty or no shebang, removing executable bit

IceGrid object adapter registration

Hi,

I'm experimenting with DataStorm and IceGrid. If I use DataStorm with IceGrid I have to enable IceGrid.Registry.DynamicRegistration because DataStrom creates an object adapter with an UUID and the AdapterId proprerty set (src\DataStorm\Instance.cpp line 99)
// // Create a collocated object adapter with a random name to prevent user configuration // of the adapter. // auto collocated = IceUtil::generateUUID(); properties->setProperty(collocated + ".AdapterId", collocated); _collocatedAdapter = _communicator->createObjectAdapter(collocated);

Ik think this is undesirable behavior, the locator for this object adapter must be nullified.
auto collocated = IceUtil::generateUUID(); properties->setProperty(collocated + ".AdapterId", collocated); _collocatedAdapter = _communicator->createObjectAdapter(collocated); _collocatedAdapter ->setLocator(0);

Cheers,
Emko

Update the library SO version to 11 when we release 1.1.0

The SO version is still 0 for the 1.0.0 release, we should make sure to update it to 11 once we release 1.1.0.

is there a way to achieve zero-copy with datastorm

using [cpp::array] we can achieve zero-copy with Ice, but I cant figure out a way to achieve zero-copy with datastorm?

steady memory leaks with ICE when broadcasting proxy with DataStorm

Version(please complete the following information):

Ice Version 3.7.6
OS: Windows
Distribution: nuget

Describe the bug
1, create a Proxy with DataStorm::Node's communicator, then broadcast the proxy.
2, If the proxy is not received by any Client, try to destroy it.
3, repeat step 1、2
4, obvious memory leak shown in VS analyzer;

here is my demo project to show the bug:
https://github.com/zengdaiwei/ICEMemoryLeak.git
run with visual studio 2019, compile and run; just keep "Enter" key down for a minutes or so, to reproduce the problem.

Expected behavior
no steady memory leaks with "creating proxy"--"broadcasting proxy"--"destroy proxy" loops.

Additional context
test projects to reproduce the bug: https://github.com/zengdaiwei/ICEMemoryLeak.git