pengutronix / aiohttp-json-rpc Goto Github PK

Looks very nice.

What about TLS? Is it supported and how (some example would be great to have).

When implementing the PasswdAuthBackend from passwd_example.py exactly from the example, I get the following error when calling one of the auth_backend methods, such as login() from a client connection:

Task exception was never retrieved
future: <Task finished name='Task-7' coro=<JsonRpc._handle_rpc_msg() done, defined at c:\users\tomlindm\appdata\local\programs\python\python38-32\lib\
site-packages\aiohttp_json_rpc\rpc.py:280> exception=AttributeError("'function' object has no attribute 'method'")>
Traceback (most recent call last):
  File "c:\users\tomlindm\appdata\local\programs\python\python38-32\lib\site-packages\aiohttp_json_rpc\rpc.py", line 307, in _handle_rpc_msg
    http_request.methods[msg.data['method']].method,
AttributeError: 'function' object has no attribute 'method'

The ping() method that is added to the rpc host works fine, but the methods instantiated in PasswdAuthBackend fails with the above error.

The JsonRpcClient does not make provision to pass an ssl_context to its .connect() method. As a workaround I patch the .connect() and .connect_url() methods to pass the ssl argument on to aiohttp. It would be useful if you could consider adding the ssl argument to the two methods described below.

import asyncio, aiohttp
from aiohttp_json_rpc import JsonRpcClient
from yarl import URL

#--- Patch JsonRpcClient to allow SSL context assignment ---
# Patch 1/2:
async def connect(self, host, port, url='/', protocol='ws', cookies=None, ssl=None): # Added ssl argument to method definition
    if not ssl is None: protocol += 's'                                              # Is this line necessary?
    url = URL.build(scheme=protocol, host=host, port=port, path=url)
    await self.connect_url(url, cookies=cookies, ssl=ssl)                            # Added ssl argument to method call
JsonRpcClient.connect = connect

# Patch 2/2:
async def connect_url(self, url, cookies=None, ssl=None):                            # Added ssl argument to method definition
    url = URL(url)
    self._session = aiohttp.ClientSession(cookies=cookies, loop=self._loop)

    self._logger.debug('#%s: ws connect...', self._id)
    self._ws = None
    try:
        self._ws = await self._session.ws_connect(url,ssl=ssl)                       # Added ssl argument to method call
    finally:
        if self._ws is None:
            # Ensure session is closed when connection failed
            await self._session.close()
    self._logger.debug('#%s: ws connected', self._id)

    self._message_worker = asyncio.ensure_future(self._handle_msgs())
JsonRpcClient.connect_url = connect_url
#--- END Patch JsonRpcClient to allow SSL connections ---

Usage is then:

rpc_client = JsonRpcClient()
ssl_context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ssl_context.load_cert_chain('server.crt','server.key')
try:
    await asyncio.wait_for(rpc_client.connect(127.0.0.1, 8080, ssl=ssl_context), timeout = 10)
except (asyncio.TimeoutError, ClientConnectorError, ConnectionRefusedError, ):
    return -1
except aiohttp.ClientSSLError:
    #Deal with SSL specific errors
    return -1

So, I was happy to have the RPC working on my code, so I tested more complex things and I broke the lib again.

I have the following method:

async def connect(self, token: str, name: str):
    # do random things...
    return "some text"

Notice it has two arguments.

When I try to call it with the websockets debug tool, I'm getting the following error (debug enabled):

DEBUG:aiohttp-json-rpc.server:raw msg received: {"jsonrpc": "2.0", "id": 0, "method": "MINECRAFTSRV__CONNECT", "params": ["hello", "world"]}
DEBUG:aiohttp-json-rpc.server:waiting for messages
DEBUG:aiohttp-json-rpc.server:message decoded: JsonRpcMsg(type=10, data={'jsonrpc': '2.0', 'id': 0, 'method': 'MINECRAFTSRV__CONNECT', 'params': ['hello', 'world']})
DEBUG:aiohttp-json-rpc.server:msg gets handled as request
ERROR:root:connect() missing 2 required positional arguments: 'token' and 'name'
Traceback (most recent call last):
  File "/Users/laggron/Library/Python/3.7/lib/python/site-packages/aiohttp_json_rpc/rpc.py", line 316, in _handle_rpc_msg
    msg=msg,
  File "/Users/laggron/Library/Python/3.7/lib/python/site-packages/aiohttp_json_rpc/rpc.py", line 146, in __call__
    return await self.method(**method_params)
TypeError: connect() missing 2 required positional arguments: 'token' and 'name'

Looks like the code isn't giving the arguments, so I searched a bit inside the source code and found where the arguments were processed. I launched the program with the Visual Studio Code debugger tool and added some breakpoints in JsonRpcMethod.__call__.

I noticed the following problem: self.required_args is equal to [], it didn't find the positional arguments. When I edit this value to ["token", "name"], everything works.

So the problem must come from this part:

# required args
self.required_args = copy(self.args)

if not (len(self.args) == 1 and self.defaults is None):
    self.required_args = [
        i for i in self.args[:-len(self.defaults or ())]
        if i not in self.CREDENTIAL_KEYS
    ]

I tried fixing this part of code to make a PR, but I just can't understand how it works. Like, you're getting the lenght of a boolean?? The or statement from len(self.defaults or ()) always return True or False, this is probably the problem, but I don't know how to fix that and what you wanted it to do... So I'm posting this issue, I tried but I can't fix this myself, I hope my research can be useful to fix the problem.

the snippet below

async def _ws_send_str(self, client, string):
    if client.ws._writer.transport.is_closing():
        self.clients.remove(client)
        await client.ws.close()

cause the error below
self.clients.remove(client)
ValueError: list.remove(x): x not in list

I'm trying to create an RPC server + client with your lib, so I tried with the README's example for the server:

rpc.py:18: DeprecationWarning: loop argument is deprecated
  app = Application(loop=loop)
rpc.py:21: DeprecationWarning: Application.make_handler(...) is deprecated, use AppRunner API instead
  handler = app.make_handler()

Not great to have outdated code, so I tried with examples/base_example.py and I got this:

rpc.py:23: DeprecationWarning: loop argument is deprecated
  app = Application(loop=loop)
Traceback (most recent call last):
  File "rpc.py", line 24, in <module>
    app.router.add_route('*', '/', rpc.handle_request)
AttributeError: 'JsonRpc' object has no attribute 'handle_request'

The library is up-to-date

$ pip install -U aiohttp-json-rpc
Requirement already up-to-date: aiohttp-json-rpc in ./Library/Python/3.7/lib/python/site-packages (0.11.2)

Please correct me if I'm wrong, but it doesn't appear possible to pass in (and use) an alternate loop other than the one returned by asyncio.get_event_loop(). It seems that JsonRpcClient.connect() and JsonRpcClient.connect_url() would need to accept a loop keyword parameter so that this could be passed to the aiohttp.ClientSession() constructor. Since that's not possible it appears only the default one will be chosen for the JSON-RPC client. Would it be possible to add this additional keyword parameter so an outside loop can be substituted?

Finally, just to confirm, it appears this library only supports JSON-RPC over websockets and not HTTP. Is this a correct assumption? If so, is this something (HTTP support) that could easily be added?

Thanks . . .

@fscherf hi, there's a few improvements to make, CI-wise:

1. Run tests against master on daily basis
2. Migrate integration into new platform travis-ci.com

myrpc should be rpc, as here: https://github.com/pengutronix/aiohttp-json-rpc/blob/master/examples/base_example.py#L24

Please fix this in project's README.md file.

currently lib takes no actual care and does not really "knows" whatever connection is alive, also
it has very poor interface to inform user of such problem. handling connection recovery becomes a mess.

If connection becomes a mess, first we could receive something from the depths of aiohttp and asyncio such as RuntimeError("Transport closed..."),

if we try to connect to nonexisting host (name not resolved) we receive:

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host [Name or service not known]

at first, but when comes the worst, after an unsuccessful attempt to connect
JsonRpcClient object is left in such state what it does not tries to autoconnect and instead throws:

 AttributeError: 'NoneType' object has no attribute 'send_str'

because attribute _ws is not dropped upon previous connection failure.

Please consider making consistent exception strategy and try not to leave objects in non recoverable state. Taming such interface is a pain.

Hello, thank you for your package. Do you have some particular problems with the latest aiohttp?

JSON-RPC 2.0 spec:

6 Batch
To send several Request objects at the same time, the Client MAY send an Array filled with Request objects.

This currently (aiohttp-json-rpc 0.12.1) fails as:

future: <Task finished coro=<JsonRpc._handle_rpc_msg() done, defined at venv/lib/python3.6/site-packages/aiohttp_json_rpc/rpc.py:280> exception=AttributeError("'list' object has no attribute 'get'",)>
Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/aiohttp_json_rpc/rpc.py", line 282, in _handle_rpc_msg
    msg = decode_msg(raw_msg.data)
  File "venv/lib/python3.6/site-packages/aiohttp_json_rpc/protocol.py", line 70, in decode_msg
    raise RpcInvalidRequestError(msg_id=msg_data.get('id', None))
AttributeError: 'list' object has no attribute 'get'

I'm trying to use the lib and I pasted the server and client examples:

rpc_server.py

from aiohttp.web import Application, run_app
from aiohttp_json_rpc import JsonRpc
import asyncio


async def ping(request):
    return 'pong'


if __name__ == '__main__':
    loop = asyncio.get_event_loop()

    rpc = JsonRpc()
    rpc.add_methods(
        ('', ping),
    )

    app = Application(loop=loop)
    app.router.add_route('*', '/', rpc.handle_request)

    run_app(app, host='0.0.0.0', port=8080)

rpc_client.py

import asyncio
from aiohttp_json_rpc import JsonRpcClient


async def ping_json_rpc():
    """Connect to ws://localhost:8080/rpc, call ping() and disconnect."""
    rpc_client = JsonRpcClient()
    try:
        await rpc_client.connect('localhost', 8080, '/rpc')
        call_result = await rpc_client.call('ping')
        print(call_result)  # prints 'pong' (if that's return val of ping)
    finally:
        await rpc_client.disconnect()


asyncio.get_event_loop().run_until_complete(ping_json_rpc())

When I execute rpc_client.py, I get the following error:

Traceback (most recent call last):
  File "rpc_client.py", line 9, in ping_json_rpc
    await rpc_client.connect('localhost', 8080, '/rpc')
  File "/usr/local/lib/python3.7/site-packages/aiohttp_json_rpc/client.py", line 136, in connect
    await self.connect_url(url, cookies=cookies)
  File "/usr/local/lib/python3.7/site-packages/aiohttp_json_rpc/client.py", line 125, in connect_url
    self._ws = await self._session.ws_connect(url)
  File "/Users/laggron/Library/Python/3.7/lib/python/site-packages/aiohttp/client.py", line 615, in _ws_connect
    headers=resp.headers)
aiohttp.client_exceptions.WSServerHandshakeError: 404, message='Invalid response status'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rpc_client.py", line 16, in <module>
    asyncio.get_event_loop().run_until_complete(ping_json_rpc())
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "rpc_client.py", line 13, in ping_json_rpc
    await rpc_client.disconnect()
  File "/usr/local/lib/python3.7/site-packages/aiohttp_json_rpc/client.py", line 152, in disconnect
    await self._ws.close()
AttributeError: 'NoneType' object has no attribute 'close'

The server is running, because I get Cannot connect to host localhost:8080 when the server is off.
I tried calling the ping method with curl to see if the problem was from the client or the server, and this is what I got

$ curl -v -H "content-type: application/json-rpc" --data '{"jsonrpc": "2.0", "method": "ping", "params": {}, id: 0}' http://0.0.0.0:8080
* Rebuilt URL to: http://0.0.0.0:8080/
*   Trying 0.0.0.0...
* TCP_NODELAY set
* Connected to 0.0.0.0 (127.0.0.1) port 8080 (#0)
> POST / HTTP/1.1
> Host: 0.0.0.0:8080
> User-Agent: curl/7.54.0
> Accept: */*
> content-type: application/json-rpc
> Content-Length: 55
>
* upload completely sent off: 55 out of 55 bytes
< HTTP/1.1 405 Method Not Allowed
< Content-Length: 0
< Content-Type: application/octet-stream
< Date: Thu, 14 Feb 2019 13:15:35 GMT
< Server: Python/3.7 aiohttp/3.4.4
<
* Connection #0 to host 0.0.0.0 left intact

This time, I get 405 Method Not Allowed, so the problem is from the server. I don't get what's wrong with what I'm doing...

how do we supply timeout parameter so the i eleiminate this annoying error
Traceback (most recent call last):
File "server.py", line 47, in handle_conversation
answer = await Engine.get_answer(decoded_data["Task"], decoded_data["content"])
File "/home/netlabsug/PycharmProjects/untitled1/mak_polls/Engine.py", line 193, in get_answer
answer = await function(content)
File "/home/netlabsug/PycharmProjects/untitled1/mak_polls/Engine.py", line 66, in post
method_res = await jrpc.post(doc)
File "/home/netlabsug/PycharmProjects/makpolls/venv/lib/python3.7/site-packages/aiohttp_json_rpc/client.py", line 174, in call
await asyncio.wait_for(self._pending[id], timeout=timeout)
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 449, in wait_for
raise futures.TimeoutError()
concurrent.futures._base.TimeoutError

When used within the context of an asyncio coroutine/future, it's impossible to cleanly "cancel" the tasks associated with the JsonRpcClient. In my application which uses this module, exceptions can occur when processing the main coroutine. This is illustrated below:

def run():
    """Entry point for console_scripts
    """
    # Register a signal handler to handle Ctrl-C
    signal.signal(signal.SIGINT, signal_handler)

    # Get the default loop
    loop = asyncio.get_event_loop()
    try:
        # Run the main program until it exits (or Ctrl-C is hit) 
        loop.run_until_complete(main(loop, sys.argv[1:]))
    except Exception as e: 
        print("Caught an error: {}".format(e))
    except SystemExit:
        # Catch the system exit and ignore
        pass
    finally:
        # We need to cancel any scheduled coroutines/tasks
        # that are still active in order to exit cleanly
        shutdown(loop)

The shutdown() routine is as follows:

def shutdown(loop):
    """Shutdown and cancel all pending coroutines/tasks."""
    tasks = asyncio.gather(*asyncio.Task.all_tasks(loop=loop),
                            loop=loop, return_exceptions=True)
    tasks.add_done_callback(lambda t: loop.stop())
    tasks.cancel()

    while not tasks.done() and not loop.is_closed():
        try:
            loop.run_forever()
        except asyncio.CancelledError:
            pass
    loop.close()

Basically, any pending task/future is cancelled and the loop stopped when all of these tasks are "done". The problem, unfortunately, is that loop.run_forever() never returns because the asyncio.CancelledError exception (raised by tasks.cancel()) is being eaten inside of client.py (JsonRpcClient). Specifically, inside async def _handle_msgs(self) the

while not self._ws.close():

loop will not exit when the asyncio.CancelledError is triggered during the call to

raw_msg = await self._ws.receive()

All of the code inside the while loop is wrapped in a try/except which merely logs the exception. I'd suggest adding an additional

except asyncio.CancelledError:
    raise

To re-raise this specific exception so the while loop will be exited (or even break instead of raise from the while loop). If this isn't done the loop.forever() in a client's clean-up code will never exit. The only alternative is to explicitly close the associated JsonRpcClient web-socket prior to invoking a similar shutdown() function as shown above so the while loop exits normally. This may be difficult to do depending on the client code and whether this web-socket is easily accessible via the client.disconnect() call.

Properly handling a asyncio.CancelledError exception within a Python future/coroutine could really help clients safely and cleanly cancel their JSON-RPC clients. In this case, just re-raising this error will allow the event loop to exit properly.

Please consider making this change . . . Thanks!

So, there's setuptools_scm package making use of git tags for determining the version of distribution package + there's a deployment provider in Travis CI helping to publish package to PYPI.

@fscherf would you accept PR implementing above? The result be the following: to release new version of dist to PYPI you'd have to push a new tag to github and everything else is done automatically.

It looks like exceptions.py does not handle implementation defined server errors (e.g. JSON-RPC errors in the range -32000 to -32099). On line exceptions.py:67

return _EXCEPTION_LOOKUP_TABLE[error_code]

There is no mapping for error codes in the range -32000 to -32099. One of the services I communicate with over JSON-RPC return implementation defined server errors and my application throws a KeyError when such an error is returned. Perhaps bit of logic could be added to exceptions.py to map these implementation defined server errors to a new generic RpcServerDefinedError?

the code below some times ti cause timeoutError randomly

if timeout:
        await asyncio.wait_for(self._pending[id], timeout=timeout)

the error is
concurrent.futures._base.TimeoutError

This is essential URL data structure used by aiohttp (and invented alongside with it), so all of the machinery of current package should support it (and probably use internally).
This may be addressed via #17 or separate PR.

Are there any plans to add support for HTTPS json-rpc for the client or server or both?

May I know where the 'send_notification' method is defined in rpc.py line 256?

request.ws.send_notification(topic, self.state[topic])

Thank you!

In latest version, C functions can not be used as RPC methods anymore. Builtin functions are C functions.
For no reason we use builtins as RPC methods. But, we do want to use C extensions. Wrapping C extensions within python functions is not good idea if there are many extensions.

Python 3.6.3 (default, Oct  3 2017, 21:45:48)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import inspect
>>> inspect.getfullargspec(min)
Traceback (most recent call last):
  File "/usr/lib/python3.6/inspect.py", line 1116, in getfullargspec
    sigcls=Signature)
  File "/usr/lib/python3.6/inspect.py", line 2262, in _signature_from_callable
    skip_bound_arg=skip_bound_arg)
  File "/usr/lib/python3.6/inspect.py", line 2087, in _signature_from_builtin
    raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in function min>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/inspect.py", line 1122, in getfullargspec
    raise TypeError('unsupported callable') from ex
TypeError: unsupported callable

I also tried a Cython compiled module with source as:

def func1(a, b):
    return a + b

It is the same.

Python 3.6.3 (default, Oct  3 2017, 21:45:48)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import inspect
>>> import t
>>> inspect.getfullargspec(t.func1)
Traceback (most recent call last):
  File "/usr/lib/python3.6/inspect.py", line 1116, in getfullargspec
    sigcls=Signature)
  File "/usr/lib/python3.6/inspect.py", line 2262, in _signature_from_callable
    skip_bound_arg=skip_bound_arg)
  File "/usr/lib/python3.6/inspect.py", line 2087, in _signature_from_builtin
    raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <built-in function func1>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/inspect.py", line 1122, in getfullargspec
    raise TypeError('unsupported callable') from ex
TypeError: unsupported callable
>>> t.func1(1, 2)
3
>>>

Incase the user have big json data, the future will just through timeout error instead
Of informing that user that actually the message size is exceeded, it would be
Nice if the Web socket response class can be supplied with max_msg_size incse the user wants to adjust that size.

Hi. I am looking for a library to implement json-rpc api with aiohttp. This one looks nice but I need POST requests. Are you going to implement them in the near future?

In setup.py aiohttp version is currently limited by <3.5.

aiohttp 3.5 was released in the late December. Do you have plans to support it?

I've noticed (supposingly) a design issue with RPC handlers.

I've got an app, where I attach some interfaces to globally accessible app object. I need to access app from within the handler therefore.
But I get only request as an input and that's it.
I guess, it is possible to retrieve app via request.http_request.app, but that's too verbose and annoying to repeat over and over in every each handler. Also using @unpack_request_args hides access to request at all.

Currently I'm thinking about introducing some kind of Class-Based "views" concept.

@fscherf do you have any additional ideas regarding this?