fi-mihej / cengal Goto Github PK

Bytecode manipulation in runtime, true shared memory, async LMDB, async Tkinter, async wxPython, async PySide, async PyQt, async loop with almost preemptive multitasking within the single thread, text parsing, rich introspection, etc.

Home Page: https://pypi.org/project/cengal/

License: Apache License 2.0

Python 95.39% Shell 2.57% Go 0.03% JavaScript 0.01% Batchfile 1.63% C 0.22% Nim 0.13% Jinja 0.01%

async async-await asynchronous bytecode bytecode-manipulation coroutines emscripten library lmdb memory-barriers mmap python qt rdtscp shared-dict shared-list shared-memory structured-concurrency tkinter wxpython

cengal's People

Contributors

Stargazers

Watchers

Forkers

tapanhaz ezzalddeenali

cengal's Issues

Whitespace in filename causing Bazel Python build rules to break

Hi, it looks like Bazel python build rules break when trying to build Cengal due to a whitespace in one of the data files. Would it be possible to replace it with an underscore or similar? Thanks!

.../build-runfiles (args bazel-out/k8-opt/bin/...runfiles_manifest bazel-out/k8-opt/bin/...runfiles): link or target filename contains space on line 10415: 'python3_deps_pypi__cengal_light/site-packages/cengal/user_interface/console/encoding_changer/versions/v_0/data/data/Leoš Janáček.py .../external/python3_deps_pypi__cengal_light/site-packages/cengal/user_interface/console/encoding_changer/versions/v_0/data/data/Leoš Janáček.py'

file: https://github.com/FI-Mihej/Cengal/blob/master/cengal/user_interface/console/encoding_changer/versions/v_0/data/data/Leo%C5%A1%20Jan%C3%A1%C4%8Dek.py

[BUG][shared_memory] Error when transfer numpy array

Hi, I got this exception when run my sample code in 2 separate terminal using cengal to test ipc transfer a numpy array:

$ python smem_sender.py

$ python smem_receiver.py

Error

Traceback (most recent call last):
  File "smem_sender.py", line 46, in <module>
    main()
  File "smem_sender.py", line 35, in main
    creator.put_message(x)
  File "/home/ubuntu/python-venv/playaround/lib/python3.8/site-packages/cengal/hardware/memory/shared_memory/versions/v_0/shared_memory.py", line 2324, in put_message
    mapped_obj, offset, message_offset = self.write_message(obj)
  File "/home/ubuntu/python-venv/playaround/lib/python3.8/site-packages/cengal/hardware/memory/shared_memory/versions/v_0/shared_memory.py", line 2309, in write_message
    mapped_obj, offset, size = self.put_obj(obj)
  File "/home/ubuntu/python-venv/playaround/lib/python3.8/site-packages/cengal/hardware/memory/shared_memory/versions/v_0/shared_memory.py", line 2273, in put_obj
    mapped_obj, offset, size = codec.map_to_shared_memory(self, obj)
  File "/home/ubuntu/python-venv/playaround/lib/python3.8/site-packages/cengal/hardware/memory/shared_memory/versions/v_0/shared_memory.py", line 279, in map_to_shared_memory
    write_uint64(shared_memory.base_address, offset + bs * len(BaseObjOffsets) + bs * BytesOffsets.data_size, data_size)
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

environments:

os                           x86_64
ubuntu                    22.04
python                    3.8.18
cengal                    4.0.3
cengal-light              4.0.3

The log points out that offset is None in this line

Cengal/cengal/hardware/memory/shared_memory/versions/v_0/shared_memory.py

Lines 277 to 278 in 9a2e728

    
           offset, real_size = shared_memory.malloc(ObjectType.tbytes, bs * len(BytesOffsets) + bs * data_size) 
        
           write_uint64(shared_memory.base_address, offset + bs * len(BaseObjOffsets) + bs * BytesOffsets.data_size, data_size)

I can help to fix but not sure if I can understand the logic behind malloc function quickly

Trouble loading ASharedMemoryContextManager.

Hello,

Using Cengal 4.4.0, tested with python 3.11 and 3.10:

Following the shared_objects__example__sender.py, I am doing this:

    sso: dict = {"block_1": "TEST"}

    # Create shared memory manager
    memory = SharedMemory('shared_objects', create=True, size=16 * 1024 ** 3)  # 16GB
    memory_manager = ASharedMemoryManager(memory)

    async with memory_manager as asmm:
        print('Sender is ready.')

        # Each coroutine should get its own context manager (ASharedMemoryContextManager). Either `asmm` or `ashared_memory_manager` can be used
        memory_context_manager: ASharedMemoryContextManager = asmm()

        async with memory_context_manager as context_manager:
            with MeasureTimeTraceLine('Preparing the shared object and sending it to the receiver') as mt:
                sso_mapped: str = context_manager.value.put_message(sso)  # by default, for custom classes including dataclass, `sso` is `sso_mapped`

    printval('sso', sso)

Running the code with asyncio.run, what happens is that the 16Gs of memory space is allocated, but we never advance past memory_context_manager: ASharedMemoryContextManager = asmm()

Any suggestions / advice on how to trace the cause are welcome!

fi-mihej / cengal Goto Github PK

cengal's People

Contributors

Stargazers

Watchers

Forkers

cengal's Issues

Whitespace in filename causing Bazel Python build rules to break

[BUG][shared_memory] Error when transfer numpy array

Trouble loading ASharedMemoryContextManager.

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	offset, real_size = shared_memory.malloc(ObjectType.tbytes, bs * len(BytesOffsets) + bs * data_size)
	write_uint64(shared_memory.base_address, offset + bs * len(BaseObjOffsets) + bs * BytesOffsets.data_size, data_size)