http = urllib3.PoolManager()
resp = http.urlopen('POST', 'www.opda.com.cn')
print resp.data
'\r\n<title>400 Bad Request</title>\r\n\r\n

400 Bad Request

But it can be open....

from urllib3.

But it's response code is 200,and html code is 400 bad request

from urllib3.

This is the same problem as you posted on #77.

The server hosting www.opda.com.cn doesn't like it when you request full URLs in the GET field. So you'll need to strip the path yourself and make the request yourself and handle redirection yourself.

Here's a basic recipe:

import urlparse
import urllib3

http = urllib3.PoolManager()

def request(method, url, conn=None):
    if conn:
        # Request within the current host connection (used for redirect handling)
        if not url.startswith('/'):
            url = '/' + url
        r = conn.request(method, url, redirect=False, assert_same_host=False)
    else:
        p = urlparse.urlparse(url)
        conn = http.connection_from_host(p.hostname, p.port, p.scheme)
        r = conn.request(method, p.path, redirect=False, assert_same_host=False)

    is_redirect = r.get_redirect_location()
    if not is_redirect:
        return r
    print "Redirecting: %s" % is_redirect

    if '://' not in is_redirect:
        # Redirect to same host
        return request('GET', is_redirect, conn)

    return request('GET', is_redirect)

r = request('GET', 'http://www.opda.com.cn')
r.status # -> 200

from urllib3.

Thank you for your answer.

Sent from my Windows Phone
发件人: Andrey Petrov
发送时间: 2012/6/23 10:50
收件人: sbitpdc
主题: Re: [urllib3] 301 problems (#82)
This is the same problem as you posted on #77.

The server hosting www.opda.com.cn doesn't like it when you request
full URLs in the GET field. So you'll need to strip the path yourself
and make the request yourself and handle redirection yourself.

Here's a basic recipe:

import urlparse
import urllib3

http = urllib3.PoolManager()

def request(method, url, conn=None):
    if conn:
        # Request within the current host connection (used for
redirect handling)
        if not url.startswith('/'):
            url = '/' + url
        r = conn.request(method, url, redirect=False, assert_same_host=False)
    else:
        p = urlparse.urlparse(url)
        conn = http.connection_from_host(p.hostname, p.port, p.scheme)
        r = conn.request(method, p.path, redirect=False, assert_same_host=False)

    is_redirect = r.get_redirect_location()
    if not is_redirect:
        return r
    print "Redirecting: %s" % is_redirect

    if '://' not in is_redirect:
        # Redirect to same host
        return request('GET', is_redirect, conn)

    return request('GET', is_redirect)

r = request('GET', 'http://www.opda.com.cn')
r.status # -> 200

Reply to this email directly or view it on GitHub:
#82 (comment)

from urllib3.

Hi, I was confused by this short example.
Should the urllib3 handle the redirect or just we do it ourselves?
But since header-HOST is forced in HTTP/1.1, I think urllib3 should consider this.
In urllib3/poolmanager.py PoolManager.urlopen():
if response.status == 303:
method = 'GET'

• if '://' not in redirect_location:
  

•     redirect_location = conn.scheme + '://' + conn.host + '/' + redirect_location
  log.info("Redirecting %s -> %s" % (url, redirect_location))
  

  Or some smarter way to handle this?

from urllib3.

Sorry,
In urllib3/poolmanager.py PoolManager.urlopen()

def urlopen():
    ...
    if response.status == 303:
        method = 'GET'
    if '://' not in redirect_location:                            # Resolve the path and host separation
        redirect_location = conn.scheme + '://' + conn.host + '/' + redirect_location
    log.info("Redirecting %s -> %s" % (url, redirect_location))
    ...

from urllib3.

@CipherChen urllib3 no longer has the behaviour described in this issue.

What problem are you having? Do you have a specific URL that isn't working?

from urllib3.

I just try this:

>>> conn = urllib3.PoolManager()
>>> rsp = conn.urlopen('GET', 'http://www.opda.com.cn')
send: 'GET / HTTP/1.1\r\nHost: www.opda.com.cn\r\nAccept-Encoding: identity\r\n\r\n'
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Server: nginx/1.0.12
header: Date: Wed, 15 May 2013 09:30:18 GMT
header: Content-Type: text/html
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Location: /m/
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 141, in urlopen
    return self.urlopen(method, redirect_location, **kw)
  File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 121, in urlopen
    conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
  File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 89, in connection_from_host
    pool_cls = pool_classes_by_scheme[scheme]
KeyError: None

And as the traceback mentioned, the Location is '/m/', which without any scheme or host.
So I thought urlopen() redirect handlling doesn't include this situation.

from urllib3.

Ah, this is a different bug: #178

There is already a pull request: #179, we're just waiting for some tests to be written. :)

If you're interested in helping, writing some tests for this fix would be appreciated.

from urllib3.