tcalmant / jsonrpclib Goto Github PK

Got the following traceback:

  File "/opt/redislabs/lib/python3.9/site-packages/jsonrpclib/jsonrpc.py", line 778, in __call__
    return self.__send(self.__name, kwargs)
  File "/opt/redislabs/lib/python3.9/site-packages/jsonrpclib/jsonrpc.py", line 652, in _request
    return response["result"]
TypeError: 'NoneType' object is not subscriptable

from what I could see ServerProxy._request performs self._run_request(request), which can return None, in which case check_for_errors returns the same None, and that is accessed.

when u read documenetation u talk about Asynchronous JSON-RPC Server but its not included in actual code

There are two print statements in the UJsonHandler class in jsonlib.py. These get executed at load-time.

Traceback (most recent call last):
File "", line 1, in
File "/root/redacted/venv/lib/python3.4/site-packages/jsonrpclib/jsonrpc.py", line 639, in call
return self.__send(self.__name, args)
File "/root/redacted/venv/lib/python3.4/site-packages/jsonrpclib/jsonrpc.py", line 523, in _request
response = self._run_request(request)
File "/root/redacted/venv/lib/python3.4/site-packages/jsonrpclib/jsonrpc.py", line 556, in _run_request
verbose=self.__verbose
File "/usr/lib/python3.4/xmlrpc/client.py", line 1131, in request
return self.single_request(host, handler, request_body, verbose)
File "/root/redacted/venv/lib/python3.4/site-packages/jsonrpclib/jsonrpc.py", line 364, in single_request
response = connection.getresponse()
File "/usr/lib/python3.4/http/client.py", line 1172, in getresponse
response.begin()
File "/usr/lib/python3.4/http/client.py", line 351, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.4/http/client.py", line 333, in _read_status
raise BadStatusLine(line)
http.client.BadStatusLine: {"result":[{"REDACTED REAL DATA"}],"id":"de3f4da8-3df0-4c07-8074-a9cde173b81b","jsonrpc":"2.0"}

I'm performing the call, and it does seem to be connecting to the server, performing the method, and retrieving that data I want, but it raises the BadStatusLine error.

Any info on what may be causing this?

I'm running Python 2.7.8 and I get this issue after running:
import jsonrpclib
server = jsonrpclib.ServerProxy('https://<server>:<port>')
print(server.derp)

and this is the result.

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/tux/code/xmsso/venv/local/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 557, in __call__
return self.__send(self.__name, kwargs)
File "/home/tux/code/xmsso/venv/local/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 434, in _request
response = self._run_request(request)
File "/home/tux/code/xmsso/venv/local/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 467, in _run_request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1293, in single_request
h = self.make_connection(host)
File "/usr/lib/python2.7/xmlrpclib.py", line 1509, in make_connection
self._connection = host, HTTPS(chost, None, context=self.context, **(x509 or {}))
AttributeError: 'SafeTransport' object has no attribute 'context'

Hello, just a little confusing, when using version 0.4.0 as a client with query string (e.g: 'http://192.168.1.3/XXX?auth=12345') its all okay when sending jsonrpc request.. But when updating to version 0.4.2, the query string (?auth=12345) is been abandoned before sending the request....

Glancing at code, seems lack of handling query property after calling urlparse ?

Hoping for your feedback, thanks a lot!

Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 651, in init
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
    self.wfile.close()
  File "/usr/lib/python2.7/socket.py", line 279, in close
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

after this happened server not responsed..
any ideas?

When using PooledJSONRPCServer, there appears to be a memory leak. To see the effect, you can run:

from jsonrpclib.SimpleJSONRPCServer import PooledJSONRPCServer
import objgraph
import gc

def poll():
    gc.collect()
    objgraph.show_growth()
    return "Hello world!"

server = PooledJSONRPCServer(('', 8080))
server.register_function(poll, 'poll')
server.serve_forever()

And poll repeatedly with:

from jsonrpclib import ServerProxy
import time

server = ServerProxy('http://host:port')

while True:
    response = server.poll()
    print(response)
    time.sleep(1)

Every iteration these extra objects are created and never cleared by the garbage collector:

dict +3
builtin_function_or_method +2
weakref +1
Event +1
Condition +1
deque +1
Thread +1

Over time, with a lot of requests, this is causing applications based on the library to crash.

This issue does not affect the SimpleJSONRPCServer.

It's an excellent library otherwise. I hope you are able to find a fix, as we've been using it for some time. Please let me know if more information is needed.

Thanks.

A Config instance should be created for each server and each proxy instances.
If not given, a Config instance (or a singleton) with a default configuration should be used.

This would allow to contact multiple servers with particular configurations.

Traffic capture of ServerProxy() to SimpleJSONRPCServer() shows multiple Content-Type headers:
POST / HTTP/1.1
Host: localhost:8000
Accept-Encoding: gzip
Content-Type: text/xml
User-Agent: jsonrpclib/0.2.5 (Python3.4.2)
Content-Type: application/json-rpc
Content-Length: 105

A 3rd party server that inspects the Content-Type header sees the "text/xml" one, and rejects the request. Seems to be sneaking in from the underlying XMLRPC libraries.
Thanks.

In method SimpleJSONRPCRequestHandler.do_POST size_remaining is lowered by len(chunk[-1]) instead of len(raw_chunk). size_remaining is a byte count while len(chunk[-1]) is a string length. Having a content that contains multi-byte characters the while loop hangs.

The thread pool provides an easy way to have a well behaving JSON-RPC server in most situations, with some control of resources consumption.

The obvious issue here is the GIL which, depending on the server tasks, might prevent to gain as much parallelism as we would expect.

For that reason, it might be a good idea to implement a process-based pool, with a similar interface to the thread pool.

This won't work for servers keeping an internal state, but could increase performances for those communicating with backend servers.

You can vote for or against this issue.

class myRequestHandler(SimpleJSONRPCRequestHandler):
   rpc_paths=('/adWolf-accApi')

   def do_OPTIONS(self):
      self.send_response(200)
      self.end_headers()

   def do_GET(self):
      self.send_response(200)
      self.end_headers()

   def do_POST(self):
      SimpleJSONRPCRequestHandler.do_POST(self)

   def end_headers(self):
      self.send_header('Access-Control-Allow-Headers', 'Origin, Authorization, X-Requested-With, Content-Type, Accept')
      self.send_header('Access-Control-Allow-Origin', '*')
      self.send_header('Access-Control-Max-Age', '0')
      self.send_header('Access-Control-Allow-Methods', 'GET, PUT, POST, DELETE, OPTIONS')
      SimpleJSONRPCRequestHandler.end_headers(self)

how to do this right?

I've started writing a WSGI app for for this library ( https://github.com/dgilman/jsonrpclib/tree/wsgi ) and I've got it working fine under 2.7. I'd like some feedback from you as I've never worked with 3.x.

The python3 json.dumps seems to be returning a python3 str and using that internally for lots of things. WSGI wants a bytestring for its data content (and they're 100% correct in their demands there). I can encode() the output of _marshaled_dispatch but there are still plenty of exceptions kicking up from the server and client side of things relating to a mismatch of bytestrings and str strings within jsonrpc.py.

So my question is 'am I on the right track? Do I need to dig through the JSONRPC server code and force it to use bytestrings everywhere in order to get WSGI on 3 working? Would you, the maintainer, be accepting of the change?'

Import's section

from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCServer
from jsonrpclib.SimpleJSONRPCServer import PooledJSONRPCServer
from jsonrpclib.threadpool import ThreadPool
from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCRequestHandler
import SocketServer

First try

   server=SimpleJSONRPCServer(("0.0.0.0", 8099), requestHandler=myRequestHandler, logRequests=False)
   server.register_instance(mySharedMethods())
   server.serve_forever()

work fine

Second try

   class ThreadedJSONRPCServer(SocketServer.ThreadingMixIn, SimpleJSONRPCServer):
      pass
   server=ThreadedJSONRPCServer(("0.0.0.0", 8099), requestHandler=myRequestHandler, logRequests=False)
   server.register_instance(mySharedMethods())
   server.serve_forever()

work fine too

And third try

   request_pool=ThreadPool(max_threads=50, min_threads=2)
   request_pool.start()
   server=PooledJSONRPCServer(("0.0.0.0", 8099), requestHandler=myRequestHandler, thread_pool=request_pool, logRequests=False)
   server.register_instance(mySharedMethods())
   server.serve_forever()

starting without error, but i can't connect to server. Any request wait forever. When i kill request, i have stack every time in one place. Maybe it's help..

/usr/local/lib/python2.7/dist-packages/jsonrpclib/jsonrpc.pyc in __call__(self, *args, **kwargs)
    565             return self.__send(self.__name, args)
    566         else:
--> 567             return self.__send(self.__name, kwargs)
    568 
    569     def __getattr__(self, name):

/usr/local/lib/python2.7/dist-packages/jsonrpclib/jsonrpc.pyc in _request(self, methodname, params, rpcid)
    442                         rpcid=rpcid, version=self.__version,
    443                         config=self._config)
--> 444         response = self._run_request(request)
    445         check_for_errors(response)
    446         return response['result']

/usr/local/lib/python2.7/dist-packages/jsonrpclib/jsonrpc.pyc in _run_request(self, request, notify)
    475             self.__handler,
    476             request,
--> 477             verbose=self.__verbose
    478         )
    479 

/usr/lib/python2.7/xmlrpclib.pyc in request(self, host, handler, request_body, verbose)
   1262         for i in (0, 1):
   1263             try:
-> 1264                 return self.single_request(host, handler, request_body, verbose)
   1265             except socket.error, e:
   1266                 if i or e.errno not in (errno.ECONNRESET, errno.ECONNABORTED, errno.EPIPE):

/usr/lib/python2.7/xmlrpclib.pyc in single_request(self, host, handler, request_body, verbose)
   1292             self.send_content(h, request_body)
   1293 
-> 1294             response = h.getresponse(buffering=True)
   1295             if response.status == 200:
   1296                 self.verbose = verbose

/usr/lib/python2.7/httplib.pyc in getresponse(self, buffering)
   1043         response = self.response_class(*args, **kwds)
   1044 
-> 1045         response.begin()
   1046         assert response.will_close != _UNKNOWN
   1047         self.__state = _CS_IDLE

/usr/lib/python2.7/httplib.pyc in begin(self)
    407         # read until we get a non-100 response
    408         while True:
--> 409             version, status, reason = self._read_status()
    410             if status != CONTINUE:
    411                 break

/usr/lib/python2.7/httplib.pyc in _read_status(self)
    363     def _read_status(self):
    364         # Initialize with Simple-Response defaults
--> 365         line = self.fp.readline(_MAXLINE + 1)
    366         if len(line) > _MAXLINE:
    367             raise LineTooLong("header line")

/usr/lib/python2.7/socket.pyc in readline(self, size)
    474             while True:
    475                 try:
--> 476                     data = self._sock.recv(self._rbufsize)
    477                 except error, e:
    478                     if e.args[0] == EINTR:

KeyboardInterrupt: 

It's not a big problem, but strange..

from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCServer
from jsonrpclib.SimpleJSONRPCServer import PooledJSONRPCServer
from jsonrpclib.threadpool import ThreadPool
from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCRequestHandler
import jsonrpclib
import jsonrpclib.utils
try: import xmlrpc.server as xmlrpcserver
except ImportError: import SimpleXMLRPCServer as xmlrpcserver

from multiprocessing import Process

class myRequestHandler(SimpleJSONRPCRequestHandler):
   rpc_paths=('/adWolf-redirectApi')

class mySharedMethods:
   def ping(self): return 'pong'

if __name__=='__main__':
   print 'Running api..'
   # server=SimpleJSONRPCServer(("0.0.0.0", 8014), requestHandler=myRequestHandler, logRequests=False)
   request_pool=ThreadPool(max_threads=50, min_threads=10)
   request_pool.start()
   server=PooledJSONRPCServer(("0.0.0.0", 8014), thread_pool=request_pool, requestHandler=myRequestHandler, logRequests=False)
   server.register_instance(mySharedMethods())
   # server.serve_forever()
   Process(target=server.serve_forever).start()

This code work without errors, but i can't connect to server.
But if i change PooledJSONRPCServer to SimpleJSONRPCServer, all work good.

• PooledJSONRPCServer definition must be fixed
• Unit tests must check the time taken by queries to run

Discussed in #54

Me again with another strange problem: We use the raven package to report runtime errors to a sentry instance. When the raven client serializes objects it encounters on the stack, it probes them for a special __sentry__ property using __getattribute__. If the object in question happens to be a ServerProxy this goes wrong because the xmlrpc.ServerProxy base class is an old-style class and thus doesn't have a __getattribute__ method. This is particularly annoying in the (JSON)RPC case because unknown attributes are assumed to be RPC functions and thus called, leading to weird errors from the RPC service.

There are some obvious ways of hacking around this. You can just catch __getattribute__ (or something broader) in ServerProxy.__getattr__ and raise AttributeError. Or you could manually add a __getattribute__ method to ServerProxy. None of that seems particularly pretty, though.

Any ideas? Maybe ditch the xmlrpclib base? 😄

It seems in forked version using basic authentication isn't possible, I used tcpdump to capture packets and jsonrpclib-plex didn't send Authorization header

I have a python file which has this imported as a module, on python2 it runs fine but on python3 it throws this error. How to fix for python3?

now i'm do this:

from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCServer
from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCRequestHandler

class myRequestHandler(SimpleJSONRPCRequestHandler):
   rpc_paths=('/adWolf-accApi')
   clientIp=None #неясно как такой способ работает при многопоточности

   def do_POST(self):
      global clientIp
      clientIp, clientPort=self.client_address
      SimpleJSONRPCRequestHandler.do_POST(self)

class mySharedMethods:
   def test(self): print clientIp

if __name__=='__main__':
   server=SimpleJSONRPCServer(("0.0.0.0", 8013), requestHandler=myRequestHandler, logRequests=False)
   server.register_instance(mySharedMethods())
   server.serve_forever()

It's ugly and i'm think it's doesn't work properly with threaded version of server (when more then one clients per time connected, global variable "clientIp" will be shared and contain IP of last client only).

More elegant way?

Sorry, but i again...

Simple server (dispathers do only adding and editing global dict, no mysql or some IO or any special).
About 40-50 connections per time.
PooledJSONRPCServer(min=10, max=100) as server.

After 3-4 hours server don't response to connection.
In log no any strange.

I have no ideas how to catch problem..
Now i add logging to every dispatcher (in begin and in end of function). But i don't think this help to locate problem.

Ran a test on python2.6 and I'm getting errors relating to the underlying xmlrpc not having a call method, raising an exception in tearDown() of TestCompatibility on line self.client('close').

I've got the source for 2.6 and 2.7 and it seems that the call was only implemented in 2.7's xmlrpclib for the proxy class. So I imagine it's not just me, it's broken for everyone else and they're just not running tox. Or am I just crazy?

If it is broken for others I'd suggest just taking py26 out of the tox file and officially dropping support - 2.6 is on its way out and it's almost not worth the conditional in tearDown.

Currently, the documentation is only the README file.
It would be nice to have a more complete documentation, with various examples.

Source of server:

import sys, re, os, random, math, imp, threading, gc, datetime, time
import jsonrpclib
from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCServer
from jsonrpclib.SimpleJSONRPCServer import SimpleJSONRPCRequestHandler
import SocketServer
import logging

class ThreadedJSONRPCServer(SocketServer.ThreadingMixIn, SimpleJSONRPCServer):
   pass

class myRequestHandler(SimpleJSONRPCRequestHandler):
    rpc_paths=('/test')
    clientIp=None

    def do_OPTIONS(self):
        self.send_response(200)
        self.end_headers()

    def do_GET(self):
        self.send_response(200)
        self.end_headers()

    def do_POST(self):
        # global clientIp
        # clientIp, clientPort=self.client_address
        SimpleJSONRPCRequestHandler.do_POST(self)

    def end_headers(self):
        self.send_header('Access-Control-Allow-Headers', 'Origin, Authorization, X-Requested-With, Content-Type, Accept')
        self.send_header('Access-Control-Allow-Origin', '*')
        self.send_header('Access-Control-Max-Age', '0')
        self.send_header('Access-Control-Allow-Methods', 'GET, PUT, POST, DELETE, OPTIONS')
        SimpleJSONRPCRequestHandler.end_headers(self)

class mySharedMethods:
    def test(self):
        # this method do nothing!
        return time.sleep(2) # without sleep i'm need waiting more time before problem happened

if __name__=='__main__':
    print 'Running Api...'
    logging.getLogger('jsonrpclib').setLevel(logging.DEBUG)
    server=ThreadedJSONRPCServer(("0.0.0.0", 8099), requestHandler=myRequestHandler,logRequests=False)
    server.register_instance(mySharedMethods())
    server.serve_forever()

on this server i'm give high load (near 5-15 connections per second).
Because i'm use ThreadedJSONRPCServer(), for every incoming connection creating new thread. In htop with tree view i'm see them very well. Threads added and removed, all work well. Every thread eat ~0.1% of cpu.

BUT..
after 10-60 minutes one(sometimes two or three) threads not removed. I'm name them 'Ghosts'.
Each Ghost eat near 50% cpu. And it never stop this and it never removed.
After first Ghost happened, process start eating memory.
After 2-3 hours process takes 6-7 gigabytes of RAM.
Peak is 22 gigabytes.

I'm think i'm do something wrong but i can't understand what...

p.s. If i'm use SimpleJSONRPCServer() instead ThreadedJSONRPCServer() all work so sloooow. And so many timeouted connections...

When an exception occurs on the server side, log the exception on the server side and return the trace to the client.

As shown in pull request #5, the jsonclass.dump() method can be easily enhanced to support more not-so-special cases.
The current (merged) version supports objects with dict and slots members, looking directly into them.

It might be more efficient to use a more generic approach, like using inspect.getmembers() to find all bean members (i.e. inherited ones too).

The __iter__ method in MultiCallIterator is a generator, using the yield statement to produce results. As such it should simply return in the end, not raise any exception. Just remove the raise StopIteration line to correct this bug.

In addition, the entire method is unnecessary, as iter() can derive it from __len__ and __getitem__. Outright removing __iter__ therefore also fixes the bug.

I haven’t had time to go through the library yet, but how hard would you estimate that it would be to add HTTP basic auth support?

is it possible to add another "notify" method, for calling methods without waiting for completion?
it will be useful for some specific things, like pushing data to server (with long processing of it, for example adding to db).
I'm know, it's break specification, but can be useful like optional extended future.

Is it possible to get the ip address of someone making an rpc call in the registered function?

ex:

def add(a, b):
    print("IP of caller")
    return a + b

if __name__ == '__main__':
    server = PooledJSONRPCServer(("0.0.0.0", 1234))
    server.register_function(add)
    server.serve_forever()

It would be interesting to implement an asyncio-based version of the library, as it undeniably increases performances in most I/O cases.

This would require a big refactoring of the project, in order to share as most code as possible between the sync & async versions of the transport and protocol layers.

It is necessary to decide if the project will depend on a third-party library to handle this or if it will use the Python 3.4+ API & syntax.
Note: the library has to stay compatible with Python 2 (at least for some years), which means all of the Python 3 syntax would be in a specific module, loaded when possible.

You can vote for or against this proposal.

If I run the following code on Python 2.7.13 against a server with an invalid certificate then it raises an ssl.CertificateError exception:

from jsonrpclib.jsonrpc import ServerProxy
import ssl

j = ServerProxy('https://<ip>', context=ssl._create_unverified_context())
j.status()

It should not give a CertificateError exception.

The problem occurs because a context parameter was added to xmlrpclib.SafeTransport in python/cpython@efa3cf8, so when the line at https://github.com/tcalmant/jsonrpclib/blob/3bfd2f3/jsonrpclib/jsonrpc.py#L467 calls TransportMixin.__init__(self, config, context) (which sets self.context), the line XMLSafeTransport.__init__(self) resets self.context to None.

From code inspection, it looks likely to be broken in Python 3 too.

It seems that Enum classes can't be unmarshalled with jsonrpclib.
A special treatment should be applied to reload them.

Sample error:

Error instantiating StatusCode: __call__() missing 1 required positional argument: 'value'

Sample JSON-RPC snippet:

{
  "id": "ae15d720-030c-4bd6-b0fa-aa04e71e56ed",
  "method": "service_14.update_status",
  "jsonrpc": "2.0",
  "params": [
    "f72c4f36-3b70-4ea1-85a6-ee6fe482d0d7",
    "bf206cd3-bcd9-4645-8371-0fe871aeef8a",
    {"__jsonclass__": ["beans.StatusCode", []], "_name_": "ERROR", "_value_": 2}
  ]
}

Support for Python 2.6 forces to use ugly tricks to ensure compatibility with this version.
It also causes a lot of trouble to correct bugs, as each correction must then be rewritten to work with Python 2.6.

Therefore, jsonrpclib 0.2.5 is the last to support Python 2.6, the next release will get rid of it.