portugueslab / arrayqueues Goto Github PK

If I try to create a queue of 30 frames for a video that has a resolution of 1080p60 (1920x1080), the code never runs. I'm assuming that's stuck trying to allocate that much memory (I tried running this with SimpleQueue originally and had the same issue).

how can i put dict objects into array?

I was curious to know, if the items that needs to be added are already known in advance (shape and type) would it be technically possible to somehow utilize this information to increase the speed of the data transfer ? (If I tweak the code or whatnot)

Thanks

Hello!
I wanted to know, when adding an element to a queue that is full, like in this test:

def test():
    shape = (100, 100)
    num_elements = 10

    data = np.random.random(size=shape)
    mbytes = data.nbytes/1_000_000*num_elements

    queue = ArrayQueue(max_mbytes=mbytes)
    for i in range(2*num_elements):
        print(i)
        queue.put(data)

Instead of throwing an error :

Traceback (most recent call last):
  File "D:/Thomas/Python/treequeues/test_treequeues.py", line 99, in <module>
    test()
  File "D:/Thomas/Python/treequeues/test_treequeues.py", line 77, in test
    queue.put(data)
  File "D:\Thomas\Python\treequeues\venv\lib\site-packages\arrayqueues\shared_arrays.py", line 87, in put
    self.check_full()
  File "D:\Thomas\Python\treequeues\venv\lib\site-packages\arrayqueues\shared_arrays.py", line 73, in check_full
    raise Full(
queue.Full: Queue of length 10 full when trying to insert 0, last item read was 0

Would it be possible to be able to hang like multprocessing queue ?

Hi! I think you're project is very convenient for putting numpy arrays into threads.

However, I have noticed a bug that is slightly annoying. Calling ArrayQueue.get() does not truly clear the array out. Trying to insert an array of the same size back into a "cleared" array results in a memory error. Check out the code below: ArrayQueue.empty() returns true, even though the array is not truly cleared.

from arrayqueues.shared_arrays import ArrayQueue
import threading
import numpy as np
import time

class ReadProcess(threading.Thread):
    def __init__(self, queue):
        super().__init__()
        self.queue = queue
      
    def run(self):
        a = self.queue.get()
        print("Is queue empty (from child thread): {}".format(queue.empty()))

if __name__ == "__main__":
    create_start = time.time()
    queue = ArrayQueue(8/1000) # intitialises an ArrayQueue which can hold 8 KB of data
    numbers = np.random.uniform(10000, size = 1000)
    numbers_copy = np.copy(numbers)
    queue.put(numbers_copy)
    thread = ReadProcess(queue)
    thread.start()
    thread.join()
    print("Is queue empty(from main thread): {}".format(queue.empty()))
    queue.put(numbers_copy)
    queue.get()
    create_end = time.time()
    print("execution time: {}".format(create_end - create_start))

Here is the output of the error:

Is queue empty (from child thread): True
Is queue empty(from main thread): True
Traceback (most recent call last):
File "array_queue_bug.py", line 25, in
queue.put(numbers_copy)
File "C:\Users\jjones\AppData\Local\Programs\Python\Python37\lib\site-packages\arrayqueues\shared_arrays.py", line 71, in put
self.check_full()
File "C:\Users\jjones\AppData\Local\Programs\Python\Python37\lib\site-packages\arrayqueues\shared_arrays.py", line 63, in check_full
self.view.i_item, self.last_item))
queue.Full: Queue of length 1 full when trying to insert 0, last item read was 0