Code Monkey home page Code Monkey logo

shmqueue's Introduction

共享内存消息队列shmqueue

  • 基于c++内存池,共享内存和信号量实现高速的进程间通信队列,支持单进程读单进程写,单进程多进程写,多进程读单进程写,多进程读多进程写

实现原理

  • 把消息队列对象生成在实现分配好的共享内存区中(共享内存去的大小远远大于消息队列管理对象messagequeue),对象中记录者共享内存去 剩余内存的数据情况,消息内存区为一个环形内存区。如下图:
    原理图
    详细介绍
    参考示例
  • 写的时候移动end索引,读的时候移动begin索引,保证了在单进程读和单进程写的时候是线 程安全的,多进程读多进成写时利用信号量集(一个读信号和写信号)实现进程间的共享内存读写锁来保证多进程安全。

测试用例

  • 源码main.cpp
  • 测试结果: ===============================================================
    Try to malloc share memory of 10240 bytes...
    Successfully alloced share memory block, (key=12345), id = 11272200, size = 10240
    Mem trunk address 0x0x7ffff7ff4000,key 11272200 , size 10240, begin 0, end 0, queue module 0
    Write SingleRWTest thread 1 ,write count 1000000
    Read SingleRWTest ,thread 1 ,read count 1000000
    Touch to share memory key = 12345...
    Now remove the exist share memory 11272200
    Remove shared memory(id = 11272200, key = 12345) succeed.
    =======================SingleRWTest=============================
    SingleRWTest cost time 3046 ms
    Read read_count 1000000
    Write write_count 1000000
    SingleRWTest ok 0
    ===============================================================
    Try to malloc share memory of 10240 bytes...
    Successfully alloced share memory block, (key=1234), id = 11304977, size = 10240
    Mem trunk address 0x0x7ffff7ff1000,key 11304977 , size 10240, begin 0, end 0, queue module 3
    ===============================================================
    Write MulRWTest thread 4 ,write count 100000
    Write MulRWTest thread 1 ,write count 100000
    Write MulRWTest thread 0 ,write count 100000
    Write MulRWTest thread 3 ,write count 100000
    Write MulRWTest thread 2 ,write count 100000
    Read MulRWTest ,thread 3 ,read count 99981
    Read MulRWTest ,thread 1 ,read count 104765
    Read MulRWTest ,thread 2 ,read count 104513
    Read MulRWTest ,thread 4 ,read count 95279
    Read MulRWTest ,thread 0 ,read count 95467
    Touch to share memory key = 1234...
    Now remove the exist share memory 11304977
    Remove shared memory(id = 11304977, key = 1234) succeed.
    =======================MulRWTest===============================
    MulRWTest cost time 24639 ms
    Read read_count 500000
    Write write_count 500000
    MulRWTest ok
  • 为了操作方便,这里的读写操作用全部用线程代替进程,从目的上来说效果是一样的

注意事项

  • 优先考虑单进程读单进程写,无锁读写效率更快,多进程读在一定程度上会降低收发效率,注意这里所说的多进程读写,如果有两个进程 一个进程只读不写一个进程只写不读,此时为单进程读单进程写,而不是多进程读写
  • 如果数据来自网络,注意字节序列,大小端的问题
  • 进程意外退出时可以不卸载共享内存区,重新启动时共享内存中的消息不会丢失,会重新因射到内存中, 可以继续读取
  • 创建共享内存队列时,注意读写模式
    • ONE_READ_ONE_WRITE, //一个进程读消息一个进程写消息(推荐)
    • ONE_READ_MUL_WRITE, //一个进程读消息多个进程写消息
    • MUL_READ_ONE_WRITE, //多个进程读消息一个进程写消息
    • MUL_READ_MUL_WRITE, //多个进程读消息多个进程写消息
  • 如果需要创建多个共享内存队列且共享内存需要加锁,那么共创建共享内存的自定义的key不要连续,不同的key之间至少相隔2,读写锁分别用共享  内存的key的key+1和key+2作为读写锁信号量集的key

shmqueue's People

Contributors

dguco avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

shmqueue's Issues

多个生产者一个消费者的时候,如果生产者动态加入会导致消费者读到的数据异常,请看下。

你好,大神
我用您这个方法测试多进程通讯的时候发现了一个问题:
在多生产者,一个消费者的时候:比如原来一个生产者和一个消费者正在通讯(生产者每次生产+1的整数数据,生产者生产的快,消费者消费的慢),如果中途有一个新的生产者进程加入但是不生产,将会覆盖掉共享内存已有的数据,导致消费者读到的数据发生跳变。
如下,写了一个例程,消费者读的数据在新生产者创建时刻发生了跳变。
如图从34跳变到1294
image

通过代码发现可能是在每次创建新的生产者进程的时候都会调用
CMessageQueue::CreateInstance(SHAR_KEY_1, 10240, eQueueModel::ONE_READ_MUL_WRITE);
thread write_thread(remote_write_func, messQueue, 1, "remoteWrite");
进而调用
CMessageQueue::CMessageQueue(BYTE pCurrAddr, eQueueModel module, key_t shmKey, int shmId, size_t size)
{
m_pShm = (void
) pCurrAddr;
m_pQueueAddr = pCurrAddr;
m_stMemTrunk = new (m_pQueueAddr) stMemTrunk();
m_pQueueAddr += sizeof(stMemTrunk);
m_stMemTrunk->m_iBegin = 0;
m_stMemTrunk->m_iEnd = 0;
m_stMemTrunk->m_iShmKey = shmKey;
m_stMemTrunk->m_iShmId = shmId;
m_stMemTrunk->m_iSize = (unsigned int)size;
m_stMemTrunk->m_eQueueModule = module;
InitLock();
}
此处将重置m_stMemTrunk->m_iBegin = 0; 和 m_stMemTrunk->m_iEnd = 0;
从而新创建进程的时候共享内存读写指针发生了重置,导致原本消费者未读的数据就没读到。
是这样的吗,我顺便向您的邮箱发了下我的测试代码,
您有空可以看下能帮忙解决下吗。
多谢

交叉编译问题

我在pc上运行没有问题,但是交叉编译会发生如下错误
arm-openwrt-linux-gnueabi-g++.bin: warning: environment variable 'STAGING_DIR' not defined
/tmp/cck7LQDM.s: Assembler messages:
/tmp/cck7LQDM.s:249: Error: bad instruction sfence' /tmp/cck7LQDM.s:759: Error: bad instruction sfence'
CMakeFiles/shmqueue.dir/build.make:62: recipe for target 'CMakeFiles/shmqueue.dir/shmmqueue.cpp.o' failed
make[2]: *** [CMakeFiles/shmqueue.dir/shmmqueue.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/shmqueue.dir/all' failed
make[1]: *** [CMakeFiles/shmqueue.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.