Multi threading about btfxwss HOT 5 CLOSED

Let's see if I understand you correctly:
You'd like to process the data that arrives in the btfxwss class in a parallel way, meaning simultaneously, correct?
In that case you'll only need a single btfxwss class. Then, start a Process (from the multiprocessing module) and give it a function that pulls data from the btfxwss properties and processes it.

from btfxwss.

Yes, you are right, that is exactly what i want.

I've tried the following:

import time
from multiprocessing import Process
from btfxwss import BtfxWss

key = 'abc'
secret = 'abc'
pair = 'BCHUSD'


def process_order_book(raw_books):
    while True:
        raw_orders = raw_books.get()
        print(raw_orders)


if __name__ == '__main__':
    wss = BtfxWss(key=key, secret=secret, log_level='DEBUG')
    wss.start()
    time.sleep(2)
    wss.authenticate()
    time.sleep(2)
    wss.subscribe_to_raw_order_book(pair=pair, prec='R0', len='100')
    time.sleep(2)

    raw_books = wss.raw_books(pair=pair)

    p1 = Process(target=process_order_book, kwargs={'raw_books': raw_books})
    p1.start()
    p1.join()

As you can see, I just print the raw books in new process. It works, but it prints only few first orders (messages from queue) and then it stucks on raw_books.get(). What am I doing wrong?

UPD:

But if I try to add the process manager:

import time
from multiprocessing import Process
from btfxwss import BtfxWss
from multiprocessing.managers import BaseManager

key = 'abc',
secret = 'abc'
pair = 'BCHUSD'


def process_order_book(_wss):
    raw_books = _wss.raw_books(pair=pair)
    while True:
        raw_orders = raw_books.get()
        print(raw_orders)


if __name__ == '__main__':
    BaseManager.register('BtfxWss', BtfxWss)
    manager = BaseManager()
    manager.start()

    wss = manager.BtfxWss(key=key, secret=secret, log_level='DEBUG')
    wss.start()
    time.sleep(2)
    wss.authenticate()
    time.sleep(2)
    wss.subscribe_to_raw_order_book(pair=pair, prec='R0', len='100')
    time.sleep(2)

    p1 = Process(target=process_order_book, kwargs={'_wss': wss})
    p1.start()
    p1.join()

It gives me the userializable object Error:

Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/tmp.py", line 12, in process_order_book
raw_books = _wss.raw_books(pair=pair)
File "", line 2, in raw_books
File "/usr/lib/python3.5/multiprocessing/managers.py", line 732, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:

Unserializable message: ('#RETURN', <queue.Queue object at 0x7fd74fdf2518>)

from btfxwss.

UPD:

Ok, I got it.
After using multithreading, but NOT multiprocessing all works fine.
I think it's because of that the BtfxWss (and its properties) are not so easy serialazable and Python can not share objects between my processes. But threads have the common memory space, that's why threads work:

import time
import threading
from btfxwss import BtfxWss

key = 'abs',
secret = 'abs'
pair = 'BCHUSD'


def process_order_book(raw_books):

    while True:
        raw_orders = raw_books.get()
        print(raw_orders)

threads = []
if __name__ == '__main__':

    wss = BtfxWss(key=key, secret=secret, log_level='DEBUG')
    wss.start()
    time.sleep(2)
    wss.authenticate()
    time.sleep(2)
    wss.subscribe_to_raw_order_book(pair=pair, prec='R0', len='100')
    time.sleep(2)

    raw_books = wss.raw_books(pair=pair)

    t = threading.Thread(target=process_order_book, args=(raw_books,))
    threads.append(t)
    t.start()

from btfxwss.

Well, the issue with using the threading module is that it doesn't actually process data in a parallel manner.
I could make use of multiprocessing.Queue instead of queue.Queue in a future release, that way you can pass the Queue object accessed via the properties to truly parallel processes. Otherwise, I'm not convinced you'd gain much from using threading in this context.
But do let me know your findings!

from btfxwss.

I've spent some time yesterday to understand the difference between processes and threads and yes, you are right, it'll be not so much odds using a threads.
And I think, multiprocessing queues will be a really great feature =)

from btfxwss.