eclipse-iceoryx / iceoryx Goto Github PK
View Code? Open in Web Editor NEWEclipse iceoryx™ - true zero-copy inter-process-communication
Home Page: https://iceoryx.io
License: Apache License 2.0
Eclipse iceoryx™ - true zero-copy inter-process-communication
Home Page: https://iceoryx.io
License: Apache License 2.0
Remove asynchronous service discovery feature
iceoryx currently implements asynchronous service discovery feature (implemented with PoshRuntime::startFindService() & PoshRuntime::stopFindService() ), in addition to synchronous service discovery.
This feature belongs to higher layers, (which could implement this feature using iceoryx api)
Remove this for cleaning up iceoryx
Motivated from ros2/rmw_iceoryx#5
The current mempool configuration is hardcoded in https://github.com/eclipse/iceoryx/blob/master/iceoryx_posh/source/mepoo/mepoo_config.cpp#L44-L54 and should be changed to be set dynamically on startup.
To communicate between docker containers and their applications using shared memory, RouDi
and /dev/shm/
-locations need to be shared across these containers. This would allow proper separation of applications while using highly performant shared memory between them.
My initial try was to mount:
/dev/shm/iceoryx_mgmt
/dev/shm/[group-name]
var/run/roudi.pid
While starting RouDi on the host system, in the docker container I could see the mounted locations but of course the pid mentioned in var/run/roudi.pid
is not correct from perspective of the non-RouDi container. I wonder whether this is the reason for the iceoryx application to not find RouDi? Maybe you have an Idea where to look at?
The current defaults for compile time configuration and mempools are quite high. Better have reasonable defaults and let people increase the values if needed
With the current default configuration we need ~200 MB for the internal data structures and the default memory pools have ~ 600 MB. This is a challenge on systems which only have some hundred MBs of RAM left. Beginners could be confused and get out of memory errors on RouDi startup. We support 4096 publishers and subscribers with the default config. Because of max size allocation with fixed size containers this blows up the memory. For most users a lower number is sufficient. The mempools can be easily configured with the config file now, so having the large default does not make sense IMHO
Having the possibility to use iceoryx on Windows 10
define public private
#529Operating system:
Ubuntu 18.04 LTS
Compiler version:
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
Observed result or behaviour:
While running example program(ice-publisher-xxx and ice-subscriber-xxx), subscriber doesn't receive the first data.
Expected result or behaviour:
If we run the subscriber first, and then run publisher, every data should be received by subscriber.
Conditions where it occurred / Performed steps:
-> % ./ice-subscriber-simple
2020-03-12 18:14:48.445 [ Info ]: Application registered management segment 0x7feb693f0000 with size 113308800 to id 1
2020-03-12 18:14:48.446 [ Info ]: Application registered payload segment 0x7feb45c41000 with size 595259200 to id 2
Callback: 1
Callback: 2
Callback: 3
Callback: 4
-> % ./ice-publisher-simple
2020-03-12 18:14:50.683 [ Info ]: Application registered management segment 0x7f3cbd3f0000 with size 113308800 to id 1
2020-03-12 18:14:50.684 [ Info ]: Application registered payload segment 0x7f3c99c41000 with size 595259200 to id 2
Sending: 0
Sending: 1
Sending: 2
Sending: 3
Sending: 4
A gateway to bridge between iceoryx and systems using the cyclonedds stack.
This component handles the communication in the iceoryx -> dds direction.
Should allow for data published in iceoryx systems to be received remotely via DDS.
This component is a building towards achieving internode communication between iceoryx systems.
Operating system:
Any case insensitive filesystem
When checking out (on OSX):
warning: the following paths have collided (e.g. case-sensitive paths
on a case-insensitive filesystem) and only one from the same
colliding group is in the working tree:
'.github/ISSUE_TEMPLATE/BUG_REPORT.md'
'.github/ISSUE_TEMPLATE/bug_report.md'
'.github/ISSUE_TEMPLATE/FEATURE_REQUEST.md'
'.github/ISSUE_TEMPLATE/feature_request.md'
Is that simply a typo or why are these files existing with different cases?
Operating system:
Ubuntu 18.04 LTS
Compiler version:
GCC 7.4.0
Observed result or behaviour:
When allocating a chunk with allocateChunk
or allocateChunkWithInfo
using the dynamic payload option (useDynamicPayloadSizes = true
) the ChunkInfo
field m_payloadSize
is initialized correctly by the first call. Subsequent calls of the same function with an increased payloadSize
do not update the member variable.
Expected result or behaviour:
ChunkInfo::m_payloadSize
should reflect the current payload size under all conditions.
Conditions where it occurred / Performed steps:
// publisher side
auto ci = m_publisher->allocateChunkWithInfo(42, true);
m_publisher->sendChunkWithInfo(ci);
auto ci = m_publisher->allocateChunkWithInfo(42+1, true);
m_publisher->sendChunkWithInfo(ci);
...
// subscriber side
const iox::mepoo::ChunkInfo* ci(nullptr);
while (m_subscriber->getChunkWithInfo(&ci))
{
std::cout << ci->m_payloadSize << std::endl;
m_subscriber->releaseChunkWithInfo(ci);
}
Besides publish/subscribe also request/response communication shall be supported
So far iceoryx only supports publish/subscribe. For frameworks like ROS2 also a request/response communication shall be supported (services in ROS2). The building blocks in posh can be re-used here, so start with client and server ports and then do the layers above
We will have the same layered architecture as for publishers and subscribers. With ChunkSender
and ChunkReceiver
common building blocks are reused but a Client
and a Server
will use one of each for the bidirectional data transfer with requests and responses.
This image is a simplified view that shows the class hierarchy and the public method names for the typed C++ API.
Client
side and take requests and loan responses on the Server
side.Publisher
, a Server
can offer()
or stopOffere()
Subscriber
subscribe()
and unsubscribe()
, the Client
can connect()
, disconnect()
and getConnectionState()Server
for requests and the Client
for responses or use a Waitset
to wait for requests and responses1. Design and interface implementation for ports
2. Implementation and unit tests for ports
3. Integration test for ports
4. C API
iceoryx_posh_types.hpp
RPC related config stuff to config.h
.5. C++ API
6. Integration in runtime and RouDi
ServerPort
's findable via ServiceDiscovery
7. Common port queue policies
SubscriberTooSlowPolicy
-> ConsumerTooSlowPolicy
WAIT_FOR_SUBSCRIBER
-> WAIT_FOR_CONSUMER
WAIT_FOR_SUBSCRIBER = WAIT_FOR_CONSUMER
for migration purposesQueueFullPolicy::BLOCK_PUBLISHER
-> QueueFullPolicy::BLOCK_PRODUCER
BLOCK_PUBLISHER = BLOCK_PRODUCER
for migration purposesSubscriberTooSlowPolicy
, WAIT_FOR_SUBSCRIBER
and BLOCK_PUBLISHER
should be deprecated8. Integration test with clients, server and waitsets (user API level)
9. Example
External Jenkins jobs are used to build and test the pull requests. Usage of github actions would be better,
With the actions, github provides the infrastructure do build the pull requests, this is much more transparent than using external Jenkins servers. The pull requests are also better documented then. In the rmw_iceoryx they were already introduced. We should also use them here.
e.g. see https://github.com/ros2/rmw_iceoryx/blob/master/.github/workflows/main.yml
Create a lock-free multi producer multi consumer queue to be used in multicast scenarios (as in e.g. ROS) and for internal control flow.
Properties:
Having the possibility to use iceoryx on macOS
Needed for ROS2 tier 1 support
ROS2 Tiers
This ticket adresses the cleanup of findings with our static code analysis to fulfil the codingrules.
Will be updated soon.
A gateway to bridge between iceoryx and dds systems using the cyclonedds stack.
This component handles the communication in the dds -> iceoryx direction.
Should make data in published in a dds system (namely, available in the dds global data space (GDS)), visible to iceoryx systems.
Data published by iceoryx->dds gateway provided in #64 should appear as if the originated locally.
Support for data published to the GDS by third party applications is out of scope for the initial version.
#pragma once is indeed a nice feature, yet not part of the standard and shall not be used (for portability issues for example). Not to mention that standards like Autosar Guidelines for C++ 14 and Misra c++ 08 do not allow it.
Modules in C++ 20 will surely be nice, but it's adoption will take some time in the industry. Having pragma once here will definitely slow down the adoption of iceoryx.
introspection –-stdout | log.txt
would be nice to dump the introspection output in a file as line breaks make it hard to parse the introspection output--process
outputIt would be really helpful, if there is an example gateway like there is an example avaialable for publisher and subscriber to see how a CaPro
gateway needs to be implemented.
This could be a simple gateway to connect via TCP-IP
or to connect two individual RouDi
s.
Operating system: Arch Linux
Compiler version: GCC 9.1.0
Observed result or behaviour: Software does not compile (./iceoryx_utils/cxx/variant.hpp:231 error: requested alignment is not an integer constant)
Expected result or behaviour: Software shall compile
Conditions where it occurred / Performed steps: run ./tools/iceoryx_build_test.sh
Capability to destroy single ports in the shared memory individually and not only all together when cleaning up the whole user process.
Today, when publishers and subscribers are destroyed, the shared memory resources are not cleaned up. This is done when the whole process is cleaned up, i.e. when the process is terminated.
For better supporting systems like ROS where publishers and subscribers can be flexibly created and destroyed within the process lifetime, iceoryx must be extended to allow destroying shared memory resources like ports individually.
Introduce a light-weight alternative cxx::function_ref
as std::function
can use heap under certain conditions which does not meet the requirements for high-safety environments
cx::function_ref
- a non-owning reference to a callable
Proposed features:
Loosely based on this proposal: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p0792r5.html
Operating system:
Ubuntu 18.04 LTS
Compiler version:
clang 9
Observed result or behaviour:
Expected result or behaviour:
compile iceoryx with clang 9
Conditions where it occurred / Performed steps:
complile iceoryx with clang 9
Operating system:
Raspbian Buster (32-Bit)
Linux armv7l GNU/Linux
Compiler version:
E.g. GCC 8.3.0
Observed result or behaviour:
Multiple Tests on Raspberry Pi 4 are failing.
Expected result or behaviour:
All tests are green
Conditions where it occurred / Performed steps:
checkout iceoryx master
execute "./tools/iceoryx_build_test.sh test"
ececute "build/utils/utils/utils_moduletests" manually
Test Logs:
iceoryx_posh:
[ RUN ] ShmCreatorDeathTest.AllocatingTooMuchMemoryLeadsToExitWithSIGBUS
/home/pi/dev/iceoryx/iceoryx_posh/test/integrationtests/test_shm_sigbus_handler.cpp:43: Failure
Death test: iox::runtime::SharedMemoryCreatoriox::roudi::MiddlewareShm sut(badconfig)
Result: failed to die.
Error msg:
[ DEATH ] Reserving 111604128 bytes in the shared memory [/iceoryx_mgmt]
[ DEATH ] [ Reserving shared memory successful ]
[ DEATH ] 2020-03-17 08:54:59.689 [ Info ]: Roudi registered management segment 0xffffffffb0121000 with size 111604128 to id 1
[ DEATH ] Reserving 107374188800 bytes in the shared memory [/pi]
[ DEATH ] [ Reserving shared memory successful ]
[ DEATH ] 2020-03-17 08:54:59.690 [ Info ]: Roudi registered payload segment 0xffffffffb6fe5000 with size 107374188800 to id 2
[ DEATH ]
Reserving 111597024 bytes in the shared memory [/iceoryx_mgmt]
[ Reserving shared memory successful ]
2020-03-17 08:55:00.066 [ Info ]: Roudi registered management segment 0xffffffffb0123000 with size 111597024 to id 1
Reserving 1088 bytes in the shared memory [/pi]
[ Reserving shared memory successful ]
2020-03-17 08:55:00.066 [ Info ]: Roudi registered payload segment 0xffffffffb6fe6000 with size 1088 to id 2
[ FAILED ] ShmCreatorDeathTest.AllocatingTooMuchMemoryLeadsToExitWithSIGBUS (750 ms)
iceoryx_utils:
[ RUN ] relativeptrtests/0.ConstrTests
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:127: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:143: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:152: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
[ FAILED ] relativeptrtests/0.ConstrTests, where TypeParam = unsigned char (2 ms)
[ RUN ] relativeptrtests/1.ConstrTests
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:127: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:143: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:152: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
[ FAILED ] relativeptrtests/1.ConstrTests, where TypeParam = signed char (1 ms)
[ RUN ] relativeptrtests/2.ConstrTests
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:127: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:143: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relative_pointer.cpp:152: Failure
Expected equality of these values:
rp.getId()
Which is: 2
1
[ FAILED ] relativeptrtests/2.ConstrTests, where TypeParam = double (1 ms)
[ RUN ] RelocatablePointer_test.relocation
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_relocatable_ptr.cpp:78: Failure
Expected equality of these values:
base2 - base1
Which is: -1024
BLOCK_SIZE
Which is: 1024
[ FAILED ] RelocatablePointer_test.relocation (1 ms)
[----------] 1 test from RelocatablePointer_test (1 ms total)
[ RUN ] Duration_test.divideDuration
/home/pi/dev/iceoryx/iceoryx_utils/test/moduletests/test_unit_duration.cpp:202: Failure
The difference between result.seconds() and 8.666666666666666 is 1.7763568394002505e-15, which exceeds doubleEpsilon, where
result.seconds() evaluates to 8.6666666666666679,
8.666666666666666 evaluates to 8.6666666666666661, and
doubleEpsilon evaluates to 2.2204460492503131e-16.
[ FAILED ] Duration_test.divideDuration (0 ms)
The current build instructions only cover the provided script to build and run tests.
For ROS2 devs, the iceoryx project can alternatively build natively with colcon. The install instructions shall reflect that.
Operating system:
Gentoo Linux, Kernel 5.5.6
Compiler version:
GCC 9.2 and clang 9.0.1
Observed result or behaviour:
These tests are failing:
CMqInterface_test.MqBase_MoveCTor
CMqInterface_test.MqBase_MoveOperator
CMqInterface_test.MqBase_TimedReceive
CMqInterface_test.MqInterfaceUser_MoveCTor
CMqInterface_test.MqInterfaceUser_MoveOperator
CMqInterface_test.MqInterfaceUser_TimedReceive
CMqInterface_test.MqInterfaceCreator_MoveCTor
CMqInterface_test.MqInterfaceCreator_MoveOperator
CMqInterface_test.MqInterfaceCreator_TimedReceive
Operating system:
Ubuntu 18.04 LTS
Compiler version:
GCC 7.4.0
Observed result or behaviour:
When using iceoryx together with rmw_iceoryx, we get an invalid response when creating a runnable
Expected result or behaviour:
runnables can be created without error
Conditions where it occurred / Performed steps:
Use latest master from iceoryx and rmw_iceoryx
Is there any updates on ASIL-D compliance and how do you plan on getting there?
We are looking at using Iceoryx in a commercial product and would like to understand the roadmap for safety.
Currently for each CaProID only one sender is allowed. This shall be extended to support m:n communication.
ROS2 is based on the DDS communication philosophy. In contrast to AUTOSAR, it is allowed that several publishers are writing to the same topic. This is currently not fully supported in iceoryx. For this feature the receiver ports must support multi-pusher queues.
To be more flexible for different communication patterns, we will have a bigger refactoring. Also the iceoryx public API will be affected
chunk queue building block
chunk distributor building block
chunk sender building block
chunk receiver building blocks
publisher port
subscriber port for single producer
condition variable
Integration of condition variable in existing building blocks
MPMC queue and integration in variant queue + harmonization of queue interfaces and port interfaces
subscriber port for multi producer
adaptation of RouDi for managing new ports and condition variable
adaptation of public API
new examples and go live
Step 2 of the RouDi modularisation and shared memory abstraction
There is no clear logging strategy. There is an iceoryx logger but it is not used everywhere. The internal logger should be used and if not in same places a clear strategy should be available when to not use it. Refactoring according to the strategy then
Readme introducing the constructs which are used in iceoryx
If someone implements new features they have to know which building blocks are already available and can be used. Additionally, we should explain our best practices.
Operating system:
Ubuntu 16.04.6 LTS
(as per ubiquity robotics image compiled on raspberryPi 3b)
Compiler version:
GCC 7.4.0 (for arm ->raspi)
Observed result or behaviour:
allocator.cpp:44:75: error: no matching function for call to ‘align(uintptr_t&, const uint64_t&)’
uintptr_t l_alignedPosition = cxx::align(l_currentAddress, f_alignment);
--> template class has only a shared parameter type for both parameters of cxx::align. Add a second template type or make second parameter (alignment) a static type e.g. uint64_t.
Expected result or behaviour:
Does compile.
Conditions where it occurred / Performed steps:
Using ubiquity robotics image compiled on raspberryPi 3b
additional to install instruction added GCC v7 as default.
(Compilation did finish after changing cxx::align 2nd parameter type (in helplets.hpp) to static type uint64_t.)
Brief feature description:
Check the necessity of all occurring const_casts and remove them if possible.
Detailed information:
To check/improve const correctness in iceoryx we should take a look at all occurring const_casts and refactor them if possible.
I get the following error message, when I try to start an example subscriber multiple times:
$ ./build/iceoryx_examples/icedelivery/ice-subscriber-bare-metal
Receiving: 1249
Receiving: 1250
Receiving: 1251
$ ./build/iceoryx_examples/icedelivery/ice-subscriber-bare-metal
2020-01-22 00:45:03.179 [Warning]: MQ still there, doing an unlink of /subscriber-bare-metal
2020-01-22 00:45:33.183 [ Error ]: ICEORYX error! MQ_INTERFACE__REG_ACK_NO_RESPONSE
ice-subscriber-bare-metal: /home/wenwen/tools/iceoryx-rs/iceoryx/iceoryx_utils/source/error_handling/error_handling.cpp:53: static void iox::ErrorHandler::ReactOnErrorLevel(iox::ErrorLevel, const char*): Assertion `false' failed.
fish: “./build/iceoryx_examples/icedel…” terminated by signal SIGABRT (Abort)
Is it possible to support multiple receivers on a topic with the same message queue name? Or it is not allowed because of some good reasons? Better error message for such "misuse" would be very nice.
This is more like a question than an issue, but why are the move semantics deleted for a subscriber?
That makes static initializers a bit complicated as the implicit RVO is disabled.
New features:
Fixes:
A Docker environment allows reproducible builds with controlled dependencies without polluting the user's host machine with extra packages needed for building and running iceoryx applications.
Acceptance criteria:
A related follow-up may be publishing the Docker image to Docker Hub to allow users to run the prebuilt image directly, but this should be decided by the core iceoryx team.
Proposed content in tools/docker
is:
For a fine grained control of the size of strings in different parts of the project, a string with a fixed but compile-time configurable capacity is required. Furthermore, throwing exceptions or using heap is forbidden.
This can be achieved by integrating the new cxx::string which uses a template parameter for its capacity. It also provides a std::string compatible comparison which compares past the first '\0'.
New features:
chunkInfo
to chunkHeader
Fixes:
The warning of Could not open file '/etc/iceoryx/roudi_config.toml'.
is somewhat misleading as it doesn't give the user any more info about what's happening.
Not only does it (annoyingly) blink, the RouDi application runs perfectly fine.
If you look at the two warnings around that, I don't see why this warning has to be red and blinking:
2020-04-13 17:36:36.144 [ Info ]: No config file provided. Using /etc/iceoryx/roudi_config.toml
Could not open file '/etc/iceoryx/roudi_config.toml'.
2020-04-13 17:36:36.144 [ Info ]: No config file found at /etc/iceoryx/roudi_config.toml. Falling back to built-in config.
The object pool is in a very early stage and should be refactored/cleaned up or it should be removed. At the moment it is used nowhere.
Required information
Operating system: Ubuntu 18.04 LTS
Compiler version: GCC 7.4.0
Observed result or behaviour:
Expected result or behaviour: No wrong script names, links or typos
Conditions where it occurred / Performed steps: Github web GUI (Commit e5c23ea)
there are a quite a few trailing whitespaces and mixes of tabs vs spaces.
Operating system:
OSX
Compiler version:
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.6.0
Thread model: posix
Observed result or behavior:
Can't compile workspace with colcon
due to missing cpptoml
dependency.
Expected result or behavior:
A clean colcon
compilation where all dependencies are successfully resolved and I can use ROS2.
Conditions where it occurred / Performed steps:
colcon build --packages-up-to iceoryx_posh
.
When sending a payload chunk via publishers sendChunk method a user can pass the payload memory pointer and it's size to transfer it with zero copy behavior to the subscribers side. If there is any additional content that needs to be transferred too (like counter, user timestamp, hash values ..) there is no option to apply this information additionally into the chunk header. So the only way to transport the payload data and some additional header information as single chunk is to do an extra memory copy in a new payload memory block before sending it as one chunk. So for this use case the amazing zero memory concept is broken.
Provide a reserved space in the ChunkInfo header to hold / transfer a user specific payload header that is not in the same linear memory like the payload data.
edited by @elBoberido
ChunkHeader
and write test for the new functionalityChunkInfo
and use the refactored ChunkHeader
in the code baseChunkHeader
members and hand out references instead of pointerchunk_mock_dds.hpp
and user chunk_mock.hpp
insteadpayload
but chunkPayload
and userPayload
. The chunkPayload
consists of the customHeader/userHeader
and userPayload
Operating system:
Ubuntu 18.04 LTS
Compiler version:
GCC 7.4.0
Observed result or behaviour:
Copy and paste issue in a_typed_api.hpp ?
Shouldn't it be TopicType instead CounterTopic ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.