pylons / webob Goto Github PK

View Code? Open in Web Editor NEW

429.0 20.0 186.0 4.42 MB

WSGI request and response objects

Home Page: https://webob.org/

Python 100.00%

webob's Introduction

WebOb

WebOb provides objects for HTTP requests and responses. Specifically it does this by wrapping the WSGI request environment and response status/headers/app_iter(body).

The request and response objects provide many conveniences for parsing HTTP request and forming HTTP responses. Both objects are read/write: as a result, WebOb is also a nice way to create HTTP requests and parse HTTP responses.

Support and Documentation

See the WebOb Documentation website to view documentation, report bugs, and obtain support.

License

WebOb is offered under the MIT-license.

Authors

WebOb was authored by Ian Bicking and is currently maintained by the Pylons Project and a team of contributors.

webob's People

Contributors

Stargazers

Watchers

Forkers

cwebber smal mitchellrj almet aodag nkunal squirro jinty mvidner mpirnat lmctv xpahos proppy hayate gordonschmidt lrowe craga89 google-feinberg bukzor kthguru mrmuxl pubkraal batterii didia amotl tbkraf08 danielholmstrom dirkmueller lukecyca creamcoderz metalman saschpe meteogrid mereostr moriyoshi michilu echlebek fashtimedotcom maisano qq40660 domenkozar ivoz msabramo chris-morgan cdellin jturmel dobesv amol- sunyi00 koansys minddistrict zuiwanting wenxueliu nphilipp young001 htang2013 lasson sontek ltvolks natim lieryan fortara dairiki sileht joulez faassen py-web jonatasoliveira jd openprocurement dstufft perey quantum-omega narendasan jimbaker madhawa1206 nickstenning lakshmi-kannan k-hasan-19 ryanpetrello chimobb cdunklau codonaut e42s simudream greghaynes melon-li yonilavi ryd994 shashankchakelam sigmavirus24 mateuszklimek tc1989tc deviora adamchainz rylz chrismorales miohtama martinth jeremydw

webob's Issues

question: multipart form requests with files

I saw in the release notes that the request object supports multipart form and thus file uploads.

I didn't see any example in the docs though. From looking at the source code it looks like this is the way.

from webob import Request
r = Request.blank("http://someappengineapp/upload_to_blobstore", POST=dict(file=("my.jpg", open("./my.jpg").read())))
res = r.send()

And it uploads correctly, but the mime type doesn't seem to be detected properly at least on appengine. Is there an additional header to pass maybe? Using the equivalent webtest.TestApp request makes it happen. I've compared both webob and webtest and nothing jumps out at me as being missing.

multidict.from_fieldstorage ignores content-type charset

When converting cgi.FieldStorage list using multidict.from_fieldstorage webob decodes each field value using utf8 regarless of the charset specified in the content type.

See:
https://github.com/Pylons/webob/blob/master/webob/multidict.py#L67

NilAccept.best_match([], default_match="foo") broken

Calling NilAccept.best_match([], default_match="foo") is broken.

I've created a pull request here to fix the issue:

You might argue that this is a feature or that it's at least known... the previous test had a comment along the lines of:

default_match has no effect on NilAccept class

However it seems to me that looking from the code that this is a bug (why assign to the otherwise unused best_match? and any call of seemingly fitting arguments, such as an empty sequence for offers and something that actually exists for default_match, shouldn't raise an error as it was previously with an UnboundLocalError exception...). I think this fix is reasonable, even urgent... the recent release logs of webob noted:

"Accept.best_matches() is gone; use list(request.accept) or request.accept.best_match(..) instead (applies to all Accept-* headers) or similar with request.accept_language."

So this seems to encourage using request.accept.best_match(request.accept_language, 'en') or similar. Indeed, I did this in two of my projects as a fix and it wasn't until I ran the tests that I noticed that this means that if your browser isn't passing in a Accept-Language argument, your application will break with the solution offered by the recent release notes!

I think this patch should be applied and hopefully a 1.2b3 can be released as a stopgap fix..

Request.decode tries to read from an already consumed stream

When building a request multiple times (for example different WSGI middlewares) it is not possible to call decode on the built requests apart from the first one.

Can be replicated with something like:

from io import BytesIO
from webob import Request

body = 'something'.encode('latin-1')
environ = {'wsgi.input': BytesIO(body), 
           'CONTENT_TYPE': 'application/x-www-form-urlencoded; charset=latin-1',
           'CONTENT_LENGTH': len(body),
           'REQUEST_METHOD': 'POST'}

r = Request(environ)
r = r.decode('latin-1')

r = Request(environ)
r = r.decode('latin-1')

The second call to decode will crash due to the fact that it tries to read again the wsgi.input.
This is due to the fact that Request.decode performs:

if content_type == 'application/x-www-form-urlencoded':
            r.body = bytes_(t.transcode_query(native_(r.body)))

which causes setup of the request body from scratch while it already got parsed by the request that we are decoding. Is this expected? Changing it to use self.body in place of r.body seems to solve the issue by using the already read body:

if content_type == 'application/x-www-form-urlencoded':
            r.body = bytes_(t.transcode_query(native_(self.body)))

Issue with unicode post parameters?

I'm not sure this is a real issue, but while using the WSGI version of zope.testbrowser to post a form with a unicode field I got this with Python 2.7:

  File "webob/request.py", line 1411, in readinto
    data = bytes_(url_encode(self.vars), 'utf8')
  File "/usr/lib/python2.7/urllib.py", line 1311, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xc9' in position 13: ordinal not in range(128)

The test browser code is something like:

input = browser.getControl(name="some-form-input")
input.value = u"1-001A1B2C3D4\xc95F6".encode("utf-8")
browser.getControl(name="some-form-button").click()

It turns out that Request.POST does the following:

    else:
        fs = cgi.FieldStorage(
            fp=self.body_file,
            environ=fs_environ,
            keep_blank_values=True)
        vars = MultiDict.from_fieldstorage(fs)

    ctype = self._content_type_raw or 'application/x-www-form-urlencoded'
    f = FakeCGIBody(vars, ctype)

where FieldStorage holds str values but MultiDict converts them to unicode, so now vars has unicode values. But FakeCGIBody.readinto then eventually runs:

        if self.content_type.startswith(
            'application/x-www-form-urlencoded'):
            # TODO: check if bytes_ is necessary
            data = bytes_(url_encode(self.vars), 'utf8')

where url_encode is urllib.urlencode, which breaks with unicode values.

Any take on this?

Thanks!

json ImportError

When try to use with the environment where is json library missed receive ImportError exception.

File "/home/buildbot/slave/full/build/env/lib/python2.6/site-packages/webob/request.py", line 8, in
import json
ImportError: No module named json

Fast fix here - smal@4f5b253#diff-0

tests.test_request.TestRequest_functional.test_headers2() fails with Python 3.3

Python 3.3 has hash randomization enabled by default, so sometimes test suite fails. You might need to run test suite multiple times to reproduce this issue.

$ nosetests-3.3
.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F........................................................................................................................................................................................................
======================================================================
FAIL: test_headers2 (tests.test_request.TestRequest_functional)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/webob/tests/test_request.py", line 2665, in test_headers2
    self.assertEqual(list(r.environ.keys()), ['a',  'HTTP_SERVER'])
AssertionError: Lists differ: ['HTTP_SERVER', 'a'] != ['a', 'HTTP_SERVER']

First differing element 0:
HTTP_SERVER
a

- ['HTTP_SERVER', 'a']
?               -----

+ ['a', 'HTTP_SERVER']
?  +++++

    """Fail immediately, with the given message."""
>>  raise self.failureException("Lists differ: ['HTTP_SERVER', 'a'] != ['a', 'HTTP_SERVER']\n\nFirst differing element 0:\nHTTP_SERVER\na\n\n- ['HTTP_SERVER', 'a']\n?               -----\n\n+ ['a', 'HTTP_SERVER']\n?  +++++\n")


----------------------------------------------------------------------
Ran 998 tests in 4.799s

FAILED (failures=1)

deprecated_property message shows `None` as the attribute name

DeprecationWarning: The attribute None is deprecated: disabled starting WebOb 1.2, use cookies instead

It should use name instead of attr when calling warn_deprecation

uploading (large) files using HTTP POST forces me to write them twice

i am doing api for uploading many files. Some of them are really large (even 40 GB).

i would love to use POST (as i have many other fields to send along)...
That stores temporary files on disk -> and forces me to read it back and write it elsewhere.

It would be sweet if i could just move the file...
When stored on the same filesystem, this will be much more efficient.

Error when running as CGI script

The standard wsgiref package offers CGIHandler helper class to convert a WSGI application to a CGI script. However, converting a WebOb callback to a WSGI app with "wsgify", then to a CGI script, then executing it, produces an error. Executing the CGI script below results in a server error under Apache.

#!/usr/bin/env python

from webob.dec import wsgify
from wsgiref.handlers import CGIHandler

@wsgify
def app(req):
    return req.path_info

CGIHandler().run(app)

webob removes CONTENT_LENGTH when setting body_file, which some middleware treats as 0

I recently ran into an issue using a modified Turbogears 2 setup. BaseRequest.POST sets body_file to a FakeCGIBody, which has the side effect of setting content_length to None, which removes CONTENT_LENGTH from the wsgi environment. The FriendlyFormPlugin uses paste.request's parse_formvars to get POST data, which assumes a missing CONTENT_LENGTH means a CONTENT_LENGTH of 0, and thus never sees any post data.

My quick fix was to set content_length to -1 instead of None, but presumably setting it to the actual length would be better.

Thanks!

an escaped url as part of the route gets unescaped.

I'm working on trying to allow an escaped url to be part of a route in Pyramid, however webob is unescaping it so when it hits routes it splits on the / and causes it to break apart.

r = webob.Request.blank('/readable/http%3A%2F%2Fgoogle.com%2Fintl%2Fen%2Fabout%2Findex.html')

r.url
http://localhost/readable/http://google.com/intl/en/about/index.html

malformed header from script. Bad header=200 OK

The apache server returns this error: "malformed header from script. Bad header = 200 OK"

The header returned is '200 OK ' but should be ' HTTP/1.1 200 OK\r '. The same header should be for the other states.

Exception with 1.1.1 when using wsgify

We recently updated from version 0.9.8 to 1.1.1 and we have found the following error in our logs. The only thing we import from dec is wsgify.

[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] mod_wsgi (pid=32391): Exception occurred processing WSGI script '/usr/cachelogic/lib/billingapi.py'.
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] Traceback (most recent call last):
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] File "/usr/lib/python2.6/site-packages/webob/dec.py", line 142, in call
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] req.response.request = req
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] File "/usr/lib/python2.6/site-packages/webob/response.py", line 881, in _request__set
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] _warn_req()
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] File "/usr/lib/python2.6/site-packages/webob/response.py", line 1196, in _warn_req
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] warn_deprecation("Response.request and Response.environ are deprecated", '1.2', 3)
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] File "/usr/lib/python2.6/site-packages/webob/util.py", line 48, in warn_deprecation
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] warnings.warn(text, cls, stacklevel=stacklevel+1)
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] File "/usr/lib64/python2.6/warnings.py", line 29, in _show_warning
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] file.write(formatwarning(message, category, filename, lineno, line))
[Thu Feb 02 08:04:02 2012] [error] [client 10.18.48.10] AttributeError: 'module' object has no attribute 'write'

OS: Ubuntu (2.6.32.25)
Python: Python 2.6.6 64bit
WebOb: 1.1.1

Incorrect headers when cache_expires() is called multiple times

If response.cache_expires(0) is called and later response.cache_expires(123) is called later for the same response, then cache_control is screwed up. Response headers are like:
Pragma: no-cache
Cache-Control: max-age=600, no-cache

It looks like cache_expires() can be called multiple times with non-zero values of seconds param, but after it's called with seconds=0, then any further calls with non-zero seconds will cause incorrect response headers.

Fix tests under Python 3.3 and update tox and trove classifiers

2 tests currently fail under Python 3.3.

Turn the "file serving" example into usable code

WebOb documentation includes an example on how to serve a static file through WebOb: http://docs.webob.org/en/latest/file-example.html

Would it be possible to include this code directly into WebOb itself, so that it can be importable and people can reliably use it to serve a file without having to copy/paste the example each time?
It would also be possible to improve the code base, by using the wsgi.file_wrapper, and everybody could benefit from it.

If there's no objection for this change, I'm willing to propose a pull request with the imported code and tests.

Accessing request.body after request.params raises UnicodeEncodeError if params contains non-ascii characters.

>>> import webob
>>> import urllib
>>> req = webob.Request.blank('/', POST=urllib.urlencode({'test': u'тест'.encode('utf-8')}))
>>> print req.body
test=%D1%82%D0%B5%D1%81%D1%82
>>> print req.params
NestedMultiDict([(u'test', u'\u0442\u0435\u0441\u0442')])
>>> print req.body
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "webob/request.py", line 619, in _body__get
    self.make_body_seekable() # we need this to have content_length
  File "webob/request.py", line 829, in make_body_seekable
    self.copy_body()
  File "webob/request.py", line 845, in copy_body
    self.body = self.body_file_raw.read()
  File "webob/request.py", line 1461, in readinto
    data = bytes_(url_encode(self.vars), 'utf8')
  File "/usr/lib/python2.7/urllib.py", line 1311, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)
>>>

urllib.urlencode requires str, not unicode, but accessing request.params sets body_file to FakeCGIBody with vars containing unicode strings. Possible fix - encode all unicode values to utf8 before passing them to urllib.urlencode:

@@ -1455,7 +1455,10 @@ class FakeCGIBody(io.RawIOBase):
             if self.content_type.startswith(
                 'application/x-www-form-urlencoded'):
                 # TODO: check if bytes_ is necessary
-                data = bytes_(url_encode(self.vars), 'utf8')
+                keys = [key.encode('utf-8') for key in self.vars.keys() if isinstance(key, unicode)]
+                values = [value.encode('utf-8') for value in self.vars.values() if isinstance(value, unicode)]
+                vars = dict(zip(keys, values))
+                data = bytes_(url_encode(vars), 'utf8')
                 self.file = io.BytesIO(data)
             elif self.content_type.startswith('multipart/form-data'):
                 self.file = _encode_multipart(

Not sure if this is correct way to fix it though. Also, looks like bytes_ is not needed in this case :)

webob response time is thery long

I try to execute the following code

req = Request(env)
req.POST['_form']

When the server receives a few large images through the form, this code runs about 60 seconds, and the CPU load to 100%

Could it be my fault, or is it the normal time of the script.

headers.clear() breaks headers in Response

If you call resp.headers.clear() on a Response object, resp.headers and resp.headerlist get out of sync.

Example:

>>> import webob
>>> resp = webob.Response()
>>> resp.headers
ResponseHeaders([('Content-Type', 'text/html; charset=UTF-8'), ('Content-Length', '0')])
>>> resp.headerlist
[('Content-Type', 'text/html; charset=UTF-8'), ('Content-Length', '0')]
>>> resp.headers.clear()
>>> resp.headerlist
[('Content-Type', 'text/html; charset=UTF-8'), ('Content-Length', '0')]
>>> resp.headers
ResponseHeaders([])

webob1.2b2 breaks appengine SDK 1.6.0 (python 2.7 runtime)

dev_appserver_blobstore.py is referencing webob.byterange.Range.parse_bytes which I guess was removed in webob1.2?

So here's my concern. Not sure if you agree or disagree.

the python2.7 runtime is currently experimental but everyone is hoping to use it when it "goes gold" I'm already stuck on pyramid 1.0 on the 2.5 runtime due to appengine only providing and depending on web0.9

getting on 2.7 gets me back in line with the latest and greatest pyramid, until pyramid 1.3 when it requires >webob1.2dev. It's going to suck, I want to play with the cool kids.

I would think it would suck for pyramid to have to state that "we just released another cool version, but you can't use it on appengine, you have to stick with 1.2"

Anyway, honestly I think this is an appengine bug, but I bet it would take them longer to decide whether to reply to a bug report than it would for you add parse_bytes back in.

thanks for reading.

Multiple range request support

I've run an issue with openstack swift's partial http range support:
https://answers.launchpad.net/swift/+question/184299
so openstack's support depends on its webob's support.

I suggest full http range support in webob framework including multiple ranges.

It would be beneficial for openstack swift project.

Yet Another Encoding Error

I was peacefully using paste when you start taliking about encoding in #11 issue :)

Here how the webserver is talking to me now.
Anybody can help ?
Regards.

File "/home/digy/temp/pyenf/lib/python2.7/site-packages/paste/
httpserver.py", line 1068, in process_request_in_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in
finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/SocketServer.py", line 639, in init
self.handle()
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/paste/
httpserver.py", line 442, in handle
BaseHTTPRequestHandler.handle(self)
File "/usr/lib/python2.7/BaseHTTPServer.py", line 343, in handle
self.handle_one_request()
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/paste/
httpserver.py", line 437, in handle_one_request
self.wsgi_execute()
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/paste/
httpserver.py", line 287, in wsgi_execute
self.wsgi_start_response)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/paste/
cascade.py", line 130, in call
return self.apps[-1](environ, start_response)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/paste/
registry.py", line 379, in call
app_iter = self.application(environ, start_response)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/pylons/
middleware.py", line 163, in call
self.app, new_environ, catch_exc_info=True)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/pylons/
util.py", line 48, in call_wsgi_application
app_iter = application(environ, start_response)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/weberror/
errormiddleware.py", line 156, in call
return self.application(environ, start_response)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/tg/
configuration.py", line 825, in remover
return app(environ, start_response)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/repoze/tm/
init.py", line 24, in call
result = self.application(environ, save_status_and_headers)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/repoze/who/
middleware.py", line 65, in call
ids = self.identify(environ, classification)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/repoze/who/
middleware.py", line 162, in identify
identity = plugin.identify(environ)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/repoze/who/
plugins/friendlyform.py", line 122, in identify
request = Request(environ, charset=self.charset)
File "/home/digy/temp/pyenf/lib/python2.7/site-packages/webob/
request.py", line 124, in init
"req.decode(charset)" % charset DeprecationWarning: You passed charset='iso-8859-1' to the Request constructor. As of WebOb 1.2, if your application needs a non-UTF-8 request charset, please construct the request without a charset or with a charset of 'None', then usereq = req.decode(charset)``

EnvironHeaders.getitem()

I think, should change EnvironHeaders.getitem() in module headers
(line 139) return self.environ[_trans_name(hname)]
on
return self.environ.get(_trans_name(hname))
to avoid KeyError

Processing bad url

Sometimes our application come the bad URL, and we could do redirect to the correct URL but can not get request.path:
File '/usr/lib64/python2.7/site-packages/paste/registry.py', line 137 in getattr
return getattr(self.current_obj(), attr)
File '/usr/lib64/python2.7/site-packages/webob/request.py', line 482 in path
bpath = bytes(self.path_info, self.url_encoding)
File '/usr/lib64/python2.7/site-packages/webob/descriptors.py', line 68 in fget
return req.encget(key, encattr=encattr)
File '/usr/lib64/python2.7/site-packages/webob/request.py', line 174 in encget
return val.decode(encoding)
File '/usr/lib64/python2.7/encodings/utf_8.py', line 16 in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc3 in position 39: invalid continuation byte

now we use hack: return val.decode(encoding, errors='replace') but it is not good. We would like to control the parameter "errors" for our project.

Example bad url and good redirect: http://actualidad.rt.com/ciencias/view/45060-La-inmortalidad,-m%C3!A1s-accesible-que-nunca

app_iter close method is not called when you have two WebOb middlewares on HEAD requests

Hello,

There is an issue with the class EmptyResponse of response.py (line 1221).
The first if in the constructor is formulated like this:

        if app_iter and hasattr(app_iter, 'close'):

And should be more likely something like:

        if app_iter is not None and hasattr(app_iter, 'close'):

The bug I encounter is that my main application uses the close method, and I have at least two middlewares using WebOb in my WSGI stack. Now when a HEAD request occurs, WebOb returns this EmptyResponse iterator instead of the one of the application, still mapping the close method of the original iterator to this new one. However it defines as well the __len__ method to return zero. Now if you have two middlewares using WebOb, the first middleware will replace the application iterator with this EmptyResponse, and the second will do the same thing, however the if used to map the close method will fail (because of the __len__ method), and the close method of the application will never be mapped, and called.

This bug is really a blocking point for me, and I would greatly appreciate to have it fixed quickly.

Thank you,

ssl.SSLError: EOF occurred in violation of protocol as a result of zlib compression.

Hello All,

I stumbled across this bug when using webob in a wsgiref server, to serve gzip encoded content. I think I've found the problem and I'll issue a patch and pull request shortly (with tests). Bear in mind that this is the first time I've parsed this code, so please correct my reasoning below if necessary.

I am currently running Python3.2 in Ubuntu 12.03. I haven't tested in any other versions. I'm running webob inside a wsgiref server that is serving https using this recipe:

http://tibit.com/code/wsgissl.py

I'm currently using SSLV3 in the SSL version and it does not seem to like when the _app_iter property of the Response object yields an empty byte string; it's a violation of protocol. In response.py we have this bit of code:

yield _gzip_header
for item in app_iter:
    size += len(item)
    crc = zlib.crc32(item, crc) & 0xffffffff
    yield compress.compress(item)
yield compress.flush()
yield struct.pack("<2L", crc, size & 0xffffffff)

Unfortunately, both compress.compress(item) and yield compress.flush() may yield zero byte strings, depending on the input. I dug into the zlib module to find out exactly what it's doing. The compression object uses this function call to compress the input:

err = deflate(&(self->zst), Z_NO_FLUSH);

I looked at the documentation for deflate and found this:

Normally the parameter flush is set to Z_NO_FLUSH, which allows deflate to
decide how much data to accumulate before producing output, in order to
maximize compression.

It seems the compression algorithm accumulates data, and when the buffer is full, yields data to the caller. Otherwise it yields an empty byte string. Here's a simple example illustrating this behaviour:

import zlib
compress = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS,
                            zlib.DEF_MEM_LEVEL, 0)

test = ['this is a test']

for item in test:
    result = compress.compress(item)         
    if not result:
        print('compress yielded no data')

result = compress.flush()
if not result:
    print('flush yielded no data')

The patch I've come up with is along the lines of:

yield _gzip_header
for item in app_iter:
    size += len(item)
    crc = zlib.crc32(item, crc) & 0xffffffff
    result = compress.compress(item)
    if result:
        yield result
result = compress.flush()
if result:
    yield result
yield struct.pack("<2L", crc, size & 0xffffffff)

Does this issue seem legitimate?

ToscaWidget failures using WebOb 1.2

I think all of these are due to:

request.script_name/request.path_info are now Unicode on Py2
removal of response.request/response.environ attrs

This isn't really an "issue" I suppose, but are we sure we want to break everybody like this?


======================================================================
ERROR: test_require_once_rendering (tests.test_resources.TestRequiredOnceJavascript)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/ToscaWidgets/tests/test_resources.py", line 664, in test_require_once_rendering
    res = app.get("/toscawidgets/resources/tw_require_once/SomeJavascript.js")
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 757, in get
    expect_errors=expect_errors)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 1001, in do_request
    res = req.get_response(app, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1122, in get_response
    application, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/lint.py", line 173, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/tw/core/registry.py", line 361, in __call__
    app_iter = self.application(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 46, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 68, in wsgi_app
    resp = req.get_response(resources_app)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resources.py", line 179, in __call__
    resp = self.serve_file(req)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resources.py", line 184, in serve_file
    stream, ct, enc = self.get_stream_type_encoding(req)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resources.py", line 218, in get_stream_type_encoding
    stream = _JavascriptFileIter(modname, relative_path, require_once, stream, self.bufsize)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resources.py", line 267, in __init__
    self.marker_name = self._marker_name(modname, path)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resources.py", line 286, in _marker_name
    return "_".join((cls.escape(modname), cls.escape(path)))
  File "/home/chrism/projects/ToscaWidgets/tw/core/resources.py", line 295, in escape
    return s.translate(cls.TRANSLATION_TABLE)
TypeError: character mapping must return integer, None or unicode

======================================================================
ERROR: test_full_response_content (tests.test_wsgiapps.TestWSGIApp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/ToscaWidgets/tests/test_wsgiapps.py", line 28, in test_full_response_content
    response = self.app.get('/')
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 757, in get
    expect_errors=expect_errors)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 1001, in do_request
    res = req.get_response(app, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1122, in get_response
    application, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/lint.py", line 173, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/opt/Python-2.7.2/lib/python2.7/wsgiref/validate.py", line 176, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/tw/core/registry.py", line 361, in __call__
    app_iter = self.application(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 46, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 72, in wsgi_app
    resp = req.get_response(self.application)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resource_injector.py", line 70, in _injector
    resp = req.get_response(app)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/opt/Python-2.7.2/lib/python2.7/wsgiref/validate.py", line 176, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/examples/wsgi_app.py", line 46, in app
    resp = Response(request=req, content_type="text/html; charset=UTF8")
  File "/home/chrism/projects/webob/webob/response.py", line 140, in __init__
    setattr(self, name, value)
  File "/home/chrism/projects/webob/webob/descriptors.py", line 89, in fset
    warn()
  File "/home/chrism/projects/webob/webob/descriptors.py", line 83, in warn
    3
  File "/home/chrism/projects/webob/webob/util.py", line 51, in warn_deprecation
    raise DeprecationWarning(text)
DeprecationWarning: The attribute request is deprecated: Response.request will be removed completely in 1.4

======================================================================
ERROR: test_links_are_fetchable (tests.test_wsgiapps.TestWSGIApp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/ToscaWidgets/tests/test_wsgiapps.py", line 39, in test_links_are_fetchable
    response = self.app.get('/')
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 757, in get
    expect_errors=expect_errors)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 1001, in do_request
    res = req.get_response(app, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1122, in get_response
    application, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/lint.py", line 173, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/opt/Python-2.7.2/lib/python2.7/wsgiref/validate.py", line 176, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/tw/core/registry.py", line 361, in __call__
    app_iter = self.application(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 46, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 72, in wsgi_app
    resp = req.get_response(self.application)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resource_injector.py", line 70, in _injector
    resp = req.get_response(app)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/opt/Python-2.7.2/lib/python2.7/wsgiref/validate.py", line 176, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/examples/wsgi_app.py", line 46, in app
    resp = Response(request=req, content_type="text/html; charset=UTF8")
  File "/home/chrism/projects/webob/webob/response.py", line 140, in __init__
    setattr(self, name, value)
  File "/home/chrism/projects/webob/webob/descriptors.py", line 89, in fset
    warn()
  File "/home/chrism/projects/webob/webob/descriptors.py", line 83, in warn
    3
  File "/home/chrism/projects/webob/webob/util.py", line 51, in warn_deprecation
    raise DeprecationWarning(text)
DeprecationWarning: The attribute request is deprecated: Response.request will be removed completely in 1.4

======================================================================
ERROR: test_resources_included (tests.test_wsgiapps.TestWSGIApp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/ToscaWidgets/tests/test_wsgiapps.py", line 23, in test_resources_included
    response = self.app.get('/')
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 757, in get
    expect_errors=expect_errors)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 1001, in do_request
    res = req.get_response(app, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1122, in get_response
    application, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/lint.py", line 173, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/opt/Python-2.7.2/lib/python2.7/wsgiref/validate.py", line 176, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/tw/core/registry.py", line 361, in __call__
    app_iter = self.application(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 46, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 72, in wsgi_app
    resp = req.get_response(self.application)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resource_injector.py", line 70, in _injector
    resp = req.get_response(app)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/opt/Python-2.7.2/lib/python2.7/wsgiref/validate.py", line 176, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/examples/wsgi_app.py", line 46, in app
    resp = Response(request=req, content_type="text/html; charset=UTF8")
  File "/home/chrism/projects/webob/webob/response.py", line 140, in __init__
    setattr(self, name, value)
  File "/home/chrism/projects/webob/webob/descriptors.py", line 89, in fset
    warn()
  File "/home/chrism/projects/webob/webob/descriptors.py", line 83, in warn
    3
  File "/home/chrism/projects/webob/webob/util.py", line 51, in warn_deprecation
    raise DeprecationWarning(text)
DeprecationWarning: The attribute request is deprecated: Response.request will be removed completely in 1.4

======================================================================
ERROR: test_scripts_are_fetchable (tests.test_wsgiapps.TestWSGIApp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/ToscaWidgets/tests/test_wsgiapps.py", line 32, in test_scripts_are_fetchable
    response = self.app.get('/')
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 757, in get
    expect_errors=expect_errors)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 1001, in do_request
    res = req.get_response(app, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1122, in get_response
    application, catch_exc_info=True)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/lint.py", line 173, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/opt/Python-2.7.2/lib/python2.7/wsgiref/validate.py", line 176, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/tw/core/registry.py", line 361, in __call__
    app_iter = self.application(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 46, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/middleware.py", line 72, in wsgi_app
    resp = req.get_response(self.application)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/projects/ToscaWidgets/tw/core/resource_injector.py", line 70, in _injector
    resp = req.get_response(app)
  File "/home/chrism/projects/webob/webob/request.py", line 1126, in get_response
    application, catch_exc_info=False)
  File "/home/chrism/projects/webob/webob/request.py", line 1095, in call_application
    app_iter = application(self.environ, start_response)
  File "/home/chrism/opt/Python-2.7.2/lib/python2.7/wsgiref/validate.py", line 176, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/home/chrism/projects/ToscaWidgets/examples/wsgi_app.py", line 46, in app
    resp = Response(request=req, content_type="text/html; charset=UTF8")
  File "/home/chrism/projects/webob/webob/response.py", line 140, in __init__
    setattr(self, name, value)
  File "/home/chrism/projects/webob/webob/descriptors.py", line 89, in fset
    warn()
  File "/home/chrism/projects/webob/webob/descriptors.py", line 83, in warn
    3
  File "/home/chrism/projects/webob/webob/util.py", line 51, in warn_deprecation
    raise DeprecationWarning(text)
DeprecationWarning: The attribute request is deprecated: Response.request will be removed completely in 1.4

======================================================================
FAIL: test_callback_authorization (tests.test_serverside_callbacks.TestServersideCallbacks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/ToscaWidgets/tests/test_serverside_callbacks.py", line 107, in test_callback_authorization
    urls = self.app.get("/").body
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 757, in get
    expect_errors=expect_errors)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 1011, in do_request
    res.body
  File "/home/chrism/projects/webob/webob/response.py", line 322, in _body__get
    body = b''.join(app_iter)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/lint.py", line 284, in next
    % (self.iterator, v))
AssertionError: Iterator <listiterator object at 0x2c864d0> returned a non-str object: u'/toscawidgets/resources/__callback__/allowed_widget/test_callback,/toscawidgets/resources/__callback__/denied_widget/test_callback'

======================================================================
FAIL: test_callback_parameter (tests.test_serverside_callbacks.TestServersideCallbacks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/ToscaWidgets/tests/test_serverside_callbacks.py", line 140, in test_callback_parameter
    allowed_url = self.app.get("/").body
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 757, in get
    expect_errors=expect_errors)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 1011, in do_request
    res.body
  File "/home/chrism/projects/webob/webob/response.py", line 322, in _body__get
    body = b''.join(app_iter)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/lint.py", line 284, in next
    % (self.iterator, v))
AssertionError: Iterator <listiterator object at 0x2cf8e10> returned a non-str object: u'/toscawidgets/resources/__callback__/allowed_widget/test_callback'

======================================================================
FAIL: test_callback_registry (tests.test_serverside_callbacks.TestServersideCallbacks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/ToscaWidgets/tests/test_serverside_callbacks.py", line 60, in test_callback_registry
    url = self.app.get("/").body
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 757, in get
    expect_errors=expect_errors)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/app.py", line 1011, in do_request
    res.body
  File "/home/chrism/projects/webob/webob/response.py", line 322, in _body__get
    body = b''.join(app_iter)
  File "/home/chrism/projects/ToscaWidgets/WebTest-1.3.1-py2.7.egg/webtest/lint.py", line 284, in next
    % (self.iterator, v))
AssertionError: Iterator <listiterator object at 0x2cf8fd0> returned a non-str object: u'/toscawidgets/resources/__callback__/test_widget/test_callback'

----------------------------------------------------------------------
Ran 281 tests in 2.396s

FAILED (failures=3, errors=5)

request.params clears request.body

>>>  request.body
'{"owner": "salmon"}'
>>> request.params
NestedMultiDict([])
>>> request.body
''

Something like this fixes the problem, but it's not correct solution:

diff --git a/webob/request.py b/webob/request.py
index 9a4132b..2e09a73 100644
--- a/webob/request.py
+++ b/webob/request.py
@@ -644,10 +644,10 @@ class BaseRequest(object):


         #ctype = self.content_type or 'application/x-www-form-urlencoded'
-        ctype = self._content_type_raw or 'application/x-www-form-urlencoded'
-        f = FakeCGIBody(vars, ctype)
-        self.body_file = io.BufferedReader(f)
-        env['webob._parsed_post_vars'] = (vars, self.body_file_raw)
+        #ctype = self._content_type_raw or 'application/x-www-form-urlencoded'
+        #f = FakeCGIBody(vars, ctype)
+        #self.body_file = io.BufferedReader(f)
+        #env['webob._parsed_post_vars'] = (vars, self.body_file_raw)
         return vars

     @property

feature requests: host_port and client_addr

I'd like these features as methods of the request. What do you think Sergey?

    @property
    def host_port(self):
        """
        The port in the URL
        """
        e = self.environ
        if e.get('HTTP_HOST'):
            host = e['HTTP_HOST']
            if ':' in host:
                host, port = host.split(':', 1)
            else:
                port = '80'
        else:
            port = e['SERVER_PORT']
        return port

and

    @property
    def client_addr(self):
        e = self.environ
        xff = e.get('HTTP_X_FORWARDED_FOR')
        if xff is not None:
            addr = xff.split(',')[0].strip()
        else:
            addr = e.get('REMOTE_ADDR')
        return addr

Quotes in Cookie are wrongly handled

When pyramid server send cookie to client, it enclosed the cookie value with quotes:

Set-Cookie:   session="ed40dd3f394c486cf048e029a40ff6b600889e19gAJKcCExT0dB08xIXC6f3H1xAShVCHVzZXJuYW1lcQJYDgAAADNAbGlhbmdzdW4ub3JncKNVA3VpZHEESwN1h3EFLg\075\075"; Path=/

But some clients (like apache's httpclient) remove the quotes, and in the following communications, the cookie value without quotes are sent to pyramid server:

Cookie: session=ed40dd3f394c486cf048e029a40ff6b600889e19gAJKcCExT0dB08xIXC6f3H1xAShVCHVzZXJuYW1lcQJYDgAAADNAbGlhbmdzdW4ub3JncKNVA3VpZHEESwN1h3EFLg\075\075

According to RFC 2109 http://www.ietf.org/rfc/rfc2109.txt, the above two kinds of cookie values should both be supported.
But the pyramid server can't identify the cookie value without quotes and generate a new session id automatically.

A possible problem with request.client_addr when not using a reverse proxy

There is a problem with recently added request.client_addr property as it assumes that the Python application is behind a properly configured reverse proxy.

This is a security vulnerability for a case where an application/middleware author uses client_addr for anything related to authentication/authorization and is not aware of deployment details. It may also be a problem if the application is using this property for other purposes.

When the WSGI server is not behind a trusted proxy it is possible to put someone else's IP or just any string in X_FORWARDED_FOR as it is a normal header and can be modified using even browser plugins. Forward proxies can also provide incorrect values (private IP addresses etc).

I'm not providing a patch as it is hard to decide how to fix this. The simplest solution would be to put a warning in the docstring but it would only cause application/middleware developers to not use it at all (their code may be used in all possible deployment configurations) and write their own similar functions.

The proper solution would require some kind of mechanism for people installing the application (not authors of the application) to set a global flag IS_BEHIND_TRUSTED_PROXY (or maybe a list of trusted proxy IPs to be completely secure?). Framework authors would be able then to expose this flag in the main configuration file of applications.

client_addr does not handle bogus data in X-Forwarded-For

I have seen X-Forwarded-For headers in the wild with Unknown values. For example Unknown, 9.10.11.12. It would be nice if WebOb's client_addr property would silently skip this.

For reference this is code I have been using for a few years:

IPV4_ADDRESS = re.compile(r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}')
IPV6_ADDRESS = re.compile(r'[0-9a-f]+[0-9a-f:]+')

def remote_address(request):
    environ = request.environ
    proxies = environ.get('HTTP_X_FORWARDED_FOR')
    if proxies is not None:
        proxies = proxies.replace(' ', '')
        addresses = [a for a in proxies.split(',')
                     if IPV4_ADDRESS.match(a) or IPV6_ADDRESS.match(a)]
        if addresses:
            return addresses[0]

    return request.environ.get('REMOTE_ADDR')

Python 3.2 MultiDict problem

I'm porting webhelpers to python 3.2 and found bug in webob.

This code doesn't work in python 3.2:

from webob.multidict import MultiDict

foo = MultiDict({'foo', 'bar'})
foo.add('foo2', 'bar2')

Patch:

--- multidict.py~       2012-06-07 19:08:13.000000000 +0400
+++ multidict.py        2012-06-07 19:08:34.000000000 +0400
@@ -32,7 +32,7 @@ class MultiDict(MutableMapping):
             if hasattr(args[0], 'iteritems'):
                 items = list(args[0].iteritems())
             elif hasattr(args[0], 'items'):
-                items = args[0].items()
+                items = list(args[0].items())
             else:
                 items = list(args[0])
             self._items = items

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 112-115: invalid data

2012-02-08 17:01:31,058 ..... user_agent=Mozilla%2F5.0+%28Linux%3B+U%3B+Android+2.3.4%3B+es-us%3B+X500+Build%2FGINGERBREAD%29+AppleWebKit%2F533.1+%28KHTML%2C+como+Gecko%29+versi%F3n%2F4.0+Mobile+Safari%2F533.1
Traceback (most recent call last):
File "/opt/runenv/lib/python2.6/site-packages/pyramid_exclog-0.5-py2.6.egg/pyramid_exclog/init.py", line 40, in exclog_tween
return handler(request)
File "/opt/runenv/lib/python2.6/site-packages/pyramid-1.2.1-py2.6.egg/pyramid/tweens.py", line 17, in excview_tween
response = handler(request)
File "/opt/runenv/lib/python2.6/site-packages/pyramid_tm-0.3-py2.6.egg/pyramid_tm/init.py", line 61, in tm_tween
response = handler(request)
File "/opt/runenv/lib/python2.6/site-packages/pyramid-1.2.1-py2.6.egg/pyramid/router.py", line 153, in handle_request
response = view_callable(context, request)
File "/opt/runenv/lib/python2.6/site-packages/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 266, in attr_view
return view(context, request)
File "/opt/runenv/lib/python2.6/site-packages/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 239, in predicate_wrapper
return view(context, request)
File "/opt/runenv/lib/python2.6/site-packages/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 292, in rendered_view
result = view(context, request)
File "/opt/runenv/lib/python2.6/site-packages/pyramid-1.2.1-py2.6.egg/pyramid/config/views.py", line 402, in _requestonly_view
response = view(request)
File "/opt/runenv/lib/python2.6/site-packages/offerservice-0.1-py2.6.egg/offerservice/views/adserver.py", line 233, in pick_banner
q = request_logging.convert_multidict_to_dict(request.params)
File "build/bdist.linux-x86_64/egg/webob/request.py", line 794, in params
params = NestedMultiDict(self.GET, self.POST)
File "build/bdist.linux-x86_64/egg/webob/request.py", line 776, in GET
vars = GetDict(data, env)
File "build/bdist.linux-x86_64/egg/webob/multidict.py", line 273, in init
MultiDict.init(self, data)
File "build/bdist.linux-x86_64/egg/webob/multidict.py", line 37, in init
items = list(args[0])
File "build/bdist.linux-x86_64/egg/webob/compat.py", line 124, in parse_qsl_text
yield (x.decode(encoding), y.decode(encoding))
File "/opt/runenv/lib/python2.6/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 112-115: invalid data

don't try to encode unicode as latin1

The fix for http://trac.pythonpaste.org/pythonpaste/ticket/251 introduced the following code:

    if isinstance(value, text_type) and not PY3:
        value = value.encode('latin-1')

As unicode >> latin1 (latin1 does not even contain a simple €) and I don't understand why headers should be latin-1 anyways (my understanding is they have to be ascii), the following code breaks badly:

>>> from webob import Response
>>> resp = Response()
>>> resp.etag = u'€'
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "webob/descriptors.py", line 138, in fset
    hset(r, val)
  File "webob/descriptors.py", line 112, in fset
    value = value.encode('latin-1')
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u20ac' in position 1: ordinal not in range(256)

I would suggest value = value.encode('ascii', 'ignore') here (just to be safe, noone should actually pass non-ascii here, but you never know).

Python 3.2 MultiDict problem

I'm porting webhelpers to python 3.2 and found bug in webob.

This code doesn't work in python 3.2:

from webob.multidict import MultiDict

foo = MultiDict({'foo', 'bar'})
foo.add('foo2', 'bar2')

Patch:

--- multidict.py~       2012-06-07 19:08:13.000000000 +0400
+++ multidict.py        2012-06-07 19:08:34.000000000 +0400
@@ -32,7 +32,7 @@ class MultiDict(MutableMapping):
             if hasattr(args[0], 'iteritems'):
                 items = list(args[0].iteritems())
             elif hasattr(args[0], 'items'):
-                items = args[0].items()
+                items = list(args[0].items())
             else:
                 items = list(args[0])
             self._items = items

If content_type is provided in Response() then a charset=utf-8 is always upended to content_type

response=Response(request=request, status=200, content_type='text/plain;charset=ISO-8859-1')
print response
200 OK
Content-Type: text/plain;charset=ISO-8859-1; charset=UTF-8

However if content_type is provided later then the extra charset is not added.

response=Response(request=request, status=200)

response.content_type = 'text/plain;charset=ISO-8859-1'
print response
200 OK
Content-Length: 0
Content-Type: text/plain;charset=ISO-8859-1

Request.charset defaults to UTF-8

Request.charset defaults to UTF-8 if no charset is specified. This seems like incorrect behavior.

See http://www.ietf.org/rfc/rfc2854.txt,:
"""
6. Charset default rules

The use of an explicit charset parameter is strongly recommended.
While [MIME] specifies "The default character set, which must be
assumed in the absence of a charset parameter, is US-ASCII." [HTTP]
Section 3.7.1, defines that "media subtypes of the 'text' type are
defined to have a default charset value of 'ISO-8859-1'". Section
19.3 of [HTTP] gives additional guidelines. Using an explicit
charset parameter will help avoid confusion.

Using an explicit charset parameter also takes into account that the
overwhelming majority of deployed browsers are set to use something
else than 'ISO-8859-1' as the default; the actual default is either a
corporate character encoding or character encodings widely deployed
in a certain national or regional community. For further
considerations, please also see Section 5.2 of [HTML40].
"""

It might be better to refuse to guess if no explicit charset is provided and let the client code deal with it.

Unicode failures with mod_wsgi + WebOb (1.1 and 1.2.1)

I am currently using WebOb 1.1 in production and I'm having some problems with unicode in query strings. Also, I tried upgrading to WebOb 1.2.1 and had additional unicode problems.

Could this be a bug in mod_wsgi or WebOb, or am I misunderstanding something about how WebOb handles unicode, or what the expectation is of me, the programmer, if I expect to get unicode requests?

Thanks,
Patrick

The Environment

Software:

python 2.6
mod_wsgi 3.3

Sample WSGI application I used for testing:

def application(environ, start_response):
    request = webob.Request(environ)
    response = webob.Response(request=request)
    response.text = request.GET['loc']
    return response(environ, start_response)

Path and query string from the browser, each with a properly percent-encoded valid UTF-8 character, 'é':
/montr%E9al-qc?loc=montr%E9al-qc

Environment dict from mod_wsgi:

{'DOCUMENT_ROOT': <snip>,
 'GATEWAY_INTERFACE': 'CGI/1.1',
 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
 'HTTP_ACCEPT_ENCODING': 'gzip,deflate,sdch',
 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8',
 'HTTP_CACHE_CONTROL': 'max-age=0',
 'HTTP_CONNECTION': 'close',
 'HTTP_COOKIE': <snip>,
 'HTTP_HOST': <snip>,
 'HTTP_USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.57 Safari/537.1',
 'NODEHEADER': <snip>,
 'PATH_INFO': '/montr\xe9al-qc',
 'PATH_TRANSLATED': '<snip>/montr\xe9al-qc',
 'QUERY_STRING': 'loc=montr%E9al-qc',
 'REMOTE_ADDR': <snip>,
 'REMOTE_PORT': <snip>,
 'REQUEST_METHOD': 'GET',
 'REQUEST_URI': '/montr%E9al-qc?loc=montr%E9al-qc',
 'SCRIPT_FILENAME': <snip>,
 'SCRIPT_NAME': '',
 'SCRIPT_URI': '<snip>/montr\xe9al-qc',
 'SCRIPT_URL': '/montr\xe9al-qc',
 'SERVER_ADDR': <snip>,
 'SERVER_ADMIN': <snip>,
 'SERVER_NAME': <snip>,
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.1',
 'SERVER_SIGNATURE': '',
 'SERVER_SOFTWARE': 'Apache',
 'mod_wsgi.application_group': '',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': <snip>,
 'mod_wsgi.listener_port': <snip>,
 'mod_wsgi.process_group': '',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '0',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0x7f2e83aafeb0>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f2e83aa5300>,
 'wsgi.input': <mod_wsgi.Input object at 0x7f2e83aaf6b0>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}

Unicode Errors

With WebOb 1.1, I get a UnicodeDecodeError when trying to access the UnicodeMultiDict request.GET.

Traceback:

Traceback (most recent call last):
  File "<snip>", line 39, in application
    response.text = request.GET['loc']
  File "/usr/lib/pymodules/python2.6/webob/multidict.py", line 312, in __getitem__
    return self._decode_value(self.multi.__getitem__(self._encode_key(key)))
  File "/usr/lib/pymodules/python2.6/webob/multidict.py", line 301, in _decode_value
    value = value.decode(self.encoding, self.errors)
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 5: invalid continuation byte

With WebOb 1.2.1, I get an additional UnicodeDecodeError attempting to read request.path_qs.

Traceback:

Traceback (most recent call last):
  File "<snip>", line 39, in application
    response.text = request.path_qs
  File "<snip>/vendor/webob/request.py", line 494, in path_qs
    path = self.path
  File "<snip>/vendor/webob/request.py", line 486, in path
    bpath = bytes_(self.path_info, self.url_encoding)
  File "<snip>/vendor/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "<snip>/vendor/webob/request.py", line 178, in encget
    return val.decode(encoding)
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 6: invalid continuation byte

When argument to request.retry_after() is a long, gets treated as string.

When you pass an integer to request.retry_after(int(number)), the Retry-After header is set to that number, which can be used to indicate that the client should retry after number of seconds. However, if you pass it a long(number), the argument is treated as a unix timestamp and interpreted as seconds after epoch. This doesn't seem like the right thing to do. Also, the WebOb documentation on this method is pretty sparse. Something listing the arguments and supported types (and expected result) would be more useful and pythonic. Thanks.

AttributeErrors during generation of documentation

docs/modules/client.txt should be modified to not reference nonexistent send_request.
docs/reference.txt, docs/modules/webob.txt and docs/test_request.txt should be modified to not reference nonexistent UnicodeMultiDict.

$ python2.7 setup.py build_sphinx
running build_sphinx
creating build
creating build/sphinx
creating build/sphinx/doctrees
creating build/sphinx/html
Running Sphinx v1.1.3
loading pickled environment... not yet created
building [html]: targets for 19 source files that are out of date
updating environment: 19 added, 0 changed, 0 removed
Traceback (most recent call last):client                                                                                                                     
  File "/usr/lib64/python2.7/site-packages/sphinx/ext/autodoc.py", line 326, in import_object
    obj = self.get_attr(obj, part)
  File "/usr/lib64/python2.7/site-packages/sphinx/ext/autodoc.py", line 232, in get_attr
    return safe_getattr(obj, name, *defargs)
  File "/usr/lib64/python2.7/site-packages/sphinx/util/inspect.py", line 70, in safe_getattr
    raise AttributeError(name)
AttributeError: send_request
Traceback (most recent call last):webob                                                                                                                      
  File "/usr/lib64/python2.7/site-packages/sphinx/ext/autodoc.py", line 326, in import_object
    obj = self.get_attr(obj, part)
  File "/usr/lib64/python2.7/site-packages/sphinx/ext/autodoc.py", line 232, in get_attr
    return safe_getattr(obj, name, *defargs)
  File "/usr/lib64/python2.7/site-packages/sphinx/util/inspect.py", line 70, in safe_getattr
    raise AttributeError(name)
AttributeError: UnicodeMultiDict
reading sources... [100%] wiki-example                                                                                                                       
/tmp/webob/docs/modules/client.txt:12: WARNING: autodoc can't import/find class 'webob.client.send_request', it reported error: "send_request", please check your spelling and sys.path
/tmp/webob/webob/static.py:docstring of webob.static.DirectoryApp:1: WARNING: Inline interpreted text or phrase reference start-string without end-string.
/tmp/webob/docs/modules/webob.txt:77: WARNING: autodoc can't import/find class 'webob.multidict.UnicodeMultiDict', it reported error: "UnicodeMultiDict", please check your spelling and sys.path
looking for now-outdated files... none found
pickling environment... done
checking consistency... /tmp/webob/docs/modules/client.txt:: WARNING: document isn't included in any toctree
/tmp/webob/docs/pycon2011/pycon-py3k-sprint.txt:: WARNING: document isn't included in any toctree
/tmp/webob/docs/test-file.txt:: WARNING: document isn't included in any toctree
/tmp/webob/docs/test_dec.txt:: WARNING: document isn't included in any toctree
/tmp/webob/docs/test_request.txt:: WARNING: document isn't included in any toctree
/tmp/webob/docs/test_response.txt:: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] wiki-example                                                                                                                        
writing additional files... genindex py-modindex search
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded, 9 warnings.

wrong argument of wsgify.middleware

sample in document::

        @wsgify.middleware
        def all_caps(app, req):

i wrote very simple dumb middleware::

@wsgify.middleware
def middleware(app, req):
    return req.get_response(app)

but i tried simple middleware causes error::

Traceback (most recent call last):
File ".\test_middleware.py", line 14, in
request.get_response(app)
File "C:\Users\aodag\Documents\works\wsgiwiki\lib\site-packages\webob\request.py", line 1149, in get_response
application, catch_exc_info=False)
File "C:\Users\aodag\Documents\works\wsgiwiki\lib\site-packages\webob\request.py", line 1118, in call_application
app_iter = application(self.environ, start_response)
File "C:\Users\aodag\Documents\works\wsgiwiki\lib\site-packages\webob\dec.py", line 132, in call
resp = self.call_func(req, _args, *_self.kwargs)
File "C:\Users\aodag\Documents\works\wsgiwiki\lib\site-packages\webob\dec.py", line 197, in call_func
return self.func(req, _args, *_kwargs)
File ".\test_middleware.py", line 6, in middleware
return req.get_response(app)
AttributeError: 'wsgify' object has no attribute 'get_response'

the order of arguments is wrong. in the fact first argument is Request, and second argument is wsgiapp.

OS: Windows 7 64bit
Python: 3.2.2 64bit
WebOb: 1.2b2

request methods that use path_info are broke under Python 3

Because PEP-3333-WSGI uses bytes-tunneled-in-latin-1 as PATH_INFO, the following methods of a WebOb request will produce unexpected results on Python 3:

@property
def path_url(self):
    """
    The URL including SCRIPT_NAME and PATH_INFO, but not QUERY_STRING
    """
    return self.application_url + url_quote(
        self.environ.get('PATH_INFO', ''), PATH_SAFE)


@property
def path(self):
    """
    The path of the request, without host or query string
    """
    return (url_quote(self.script_name, PATH_SAFE) +
            url_quote(self.path_info, PATH_SAFE))

Also, Request.path_info_pop and Request.path_info_peek and environ_from_url (and methods/functions that use any of these) will return odd results on Python 3 when PATH_INFO contains nonascii characters.

Bleh. I don't know how to fix this without introducing backwards incompatibilities.

webob/request.py should try to use simplejson

webob/response.py contains:

try:
    import simplejson as json
except ImportError:
    import json

webob/request.py should contain the same simplejson / json handling instead of import json line.

Import fails in compat.py on python 2.5

Context

I'm writing an appengine project on python 2.5.4. Running the appengine local dev environment works fine. When I try to run unittests with 1.2b2 it fails in the following way.

Problem

When importing compat.py, on python 2.5 somehow we end up on line 89 as in the following build error:

File "/home/ccomb/buildout-eggs/WebOb-1.2b2-py2.5.egg/webob/compat.py", line 89
    return b''
             ^
SyntaxError: invalid syntax

A full log is here:
http://buildbot.afpy.org/formalchemy/builders/FormAlchemy%20Python2.5.5%2064bit%20Linux/builds/405/steps/test/logs/stdio/text
(not mine, but it's the same error in the end)

It's really weird, because afaict the "if PY3:" block that the def is in should mean we never get there.

Workaround

Reverting to 1.1.1 with easy_install fixes the problem for me.

WSGIHTTPException overwrites headers of an empty-bodied HEAD response

The bug is triggered in OpenStack Swift (Object Storage), see also their bug report for context: https://bugs.launchpad.net/swift/+bug/920197

We have a webob.exc.HTTPOk response to a HEAD request. That response proxies another response from a backend server, so it has headers reflecting a non-empty body and an empty body.

Commit 2e23e22 "make sure WSGIHTTPException return the same headers for HEAD and GET requests" chenged the behavior in such a case, resulting in the templated exception body being generated, and then discarded because of HEAD, but replacing the original headers with those of the template.

DeprecationWarnings and ResourceWarnings with Python >=3.2

When warnings are enabled (e.g. using PYTHONWARNINGS="d" environmental variable), then some DeprecationWarnings and ResourceWarnings occur with Python >=3.2. I use WebOb 1.2.2.

$ PYTHONWARNINGS="d" nosetests-3.2
.........................................................................................................../usr/lib64/python3.2/site-packages/nose/case.py:198: ResourceWarning: unclosed <socket.socket object, fd=3, family=2, type=1, proto=0>
  self.test(*self.arg)
.../tmp/WebOb-1.2.2/webob/request.py:1260: ResourceWarning: unclosed <socket.socket object, fd=4, family=2, type=2049, proto=6>
  app_iter = application(self.environ, start_response)
.........................................................../tmp/WebOb-1.2.2/tests/test_dec.py:160: DeprecationWarning: Please use assertTrue instead.
  self.assert_(set_urlvar.__class__ is _MiddlewareFactory)
/tmp/WebOb-1.2.2/tests/test_dec.py:162: DeprecationWarning: Please use assertTrue instead.
  self.assert_('set_urlvar' in r)
../tmp/WebOb-1.2.2/tests/test_dec.py:179: DeprecationWarning: Please use assertTrue instead.
  self.assert_(unbound.__class__ is _UnboundMiddleware)
/tmp/WebOb-1.2.2/tests/test_dec.py:184: DeprecationWarning: Please use assertTrue instead.
  self.assert_(middle.__class__ is wsgify)
./tmp/WebOb-1.2.2/tests/test_dec.py:190: DeprecationWarning: Please use assertTrue instead.
  self.assert_(unbound.__class__ is _UnboundMiddleware)
........./tmp/WebOb-1.2.2/tests/test_dec.py:72: DeprecationWarning: Please use assertTrue instead.
  self.assert_(test_app.__get__(test_app) is test_app)
.../tmp/WebOb-1.2.2/tests/test_dec.py:59: DeprecationWarning: Please use assertTrue instead.
  self.assert_(resp.body.startswith(b'400 Bad Request'))
../tmp/WebOb-1.2.2/tests/test_dec.py:137: DeprecationWarning: Please use assertTrue instead.
  self.assert_(wrapped_test_app.undecorated is test_app)
.........................................................................................................................................................................................................../usr/lib64/python3.2/unittest/case.py:557: ResourceWarning: unclosed file <_io.BufferedRandom name=4>
  callableObj(*args, **kwargs)
/usr/lib64/python3.2/unittest/case.py:557: ResourceWarning: unclosed file <_io.BufferedRandom name=5>
  callableObj(*args, **kwargs)
................................................................................................................................................................................................................................................................................................................................................................................................/tmp/WebOb-1.2.2/webob/descriptors.py:44: ResourceWarning: unclosed file <_io.BufferedRandom name=3>
  req.environ[key] = val
......../tmp/WebOb-1.2.2/tests/test_request.py:2529: ResourceWarning: unclosed file <_io.BufferedRandom name=3>
  body_file=UnseekableInput(b'0123456789'), content_length=10)
......................................../tmp/WebOb-1.2.2/tests/test_request.py:3413: DeprecationWarning: Please use assertTrue instead.
  self.assert_("audio/mpeg" in request.body.decode('ascii'), str(request))
/tmp/WebOb-1.2.2/tests/test_request.py:3414: DeprecationWarning: Please use assertTrue instead.
  self.assert_('application/x-foo' in request.body.decode('ascii'), str(request))
........../usr/lib64/python3.2/unittest/case.py:557: ResourceWarning: unclosed file <_io.BufferedRandom name=3>
  callableObj(*args, **kwargs)
.../usr/lib64/python3.2/site-packages/nose/case.py:198: ResourceWarning: unclosed file <_io.BufferedRandom name=3>
  self.test(*self.arg)
...................................................................................................................................../usr/lib64/python3.2/unittest/case.py:370: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/tmpo6rpps/bar'>
  function()
...../tmp/WebOb-1.2.2/tests/test_static.py:66: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/tmpttldz9.py'>
  self.assertEqual(200, resp(method='GET').status_code)
/tmp/WebOb-1.2.2/webob/request.py:1260: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/tmpttldz9.py'>
  app_iter = application(self.environ, start_response)
../usr/lib64/python3.2/unittest/case.py:370: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/tmppav3pz.py'>
  function()
../usr/lib64/python3.2/unittest/case.py:370: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/tmp67r0pi.py'>
  function()
................
----------------------------------------------------------------------
Ran 992 tests in 4.771s

OK

_abs_headerlist should die

Can we kill off the mutation of the headerlist by response._abs_headerlist? It doesn't have enough information to be able to create a correct request url (it can't know the url encoding), and I'm not really even sure of its ultimate utility; it seems like most clients cope just fine with relative Location headers.

Python 3 error in file-example

I was following this example with python 3:

http://docs.webob.org/en/latest/file-example.html

And I got this error:

Traceback (most recent call last):
  File "/usr/lib/python3.2/wsgiref/handlers.py", line 138, in run
    self.finish_response()
  File "/usr/lib/python3.2/wsgiref/handlers.py", line 178, in finish_response
    for data in self.result:
TypeError: iter() returned non-iterator of type 'FileIterator'

I found that the iter interface for python 3 has changed. An iterator needs a __next__ method in python 3, where in python 2, it needs a next method.

Can we change the example to include:

    def next(self):
        chunk = self.fileobj.read(self.chunk_size)
        if not chunk:
            raise StopIteration
        return chunk

+   __next__ = next

so that it works in both python 2 and 3.