juliaweb / requests.jl Goto Github PK

Recent changes to the semantics of .data has broken Quand.jll and I'm not sure what version to use as the ceiling.

Now that @malmaud is the hero JuliaWeb needs, I thought I'd put in a request (pun intended).

In the past, trying to use Requests.get was excruciatingly slow when trying to download large files. Below is a quick benchmark on a 8.6 zipped CSV file (publicly available). The difference isn't huge, but I'm also wondering if the slowness I've experience in the past is due to https and having to go through GnuTLS. I'll try to figure out another benchmark to show an https call.

julia> @time Requests.get("http://seanlahman.com/files/database/lahman-csv_2015-01-24.zip")
  7.701232 seconds (5.64 M allocations: 1.770 GB, 3.43% gc time)
Response(200 OK, 12 Headers, 8614880 Bytes in Body)

julia> @time readall(`curl -sS http://seanlahman.com/files/database/lahman-csv_2015-01-24.zip`);
  4.748406 seconds (9.60 k allocations: 33.613 MB, 0.42% gc time)

PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their tests (if available) on both the stable version of Julia (0.3) and the nightly build of the unstable version (0.4). The results of this script are used to generate a package listing enhanced with testing results.

On Julia 0.4

• On 2014-08-26 the testing status was Tests pass.
• On 2014-08-27 the testing status changed to Tests fail, but package loads.

Tests pass. means that PackageEvaluator found the tests for your package, executed them, and they all passed.

Tests fail, but package loads. means that PackageEvaluator found the tests for your package, executed them, and they didn't pass. However, trying to load your package with using worked.

This issue was filed because your testing status became worse. No additional issues will be filed if your package remains in this state, and no issue will be filed if it improves. If you'd like to opt-out of these status-change messages, reply to this message saying you'd like to and @IainNZ will add an exception. If you'd like to discuss PackageEvaluator.jl please file an issue at the repository. For example, your package may be untestable on the test machine due to a dependency - an exception can be added.

Test log:

>>> 'Pkg.add("Requests")' log
INFO: Installing BinDeps v0.3.3
INFO: Installing Codecs v0.1.2
INFO: Installing Dates v0.4.3
INFO: Installing GnuTLS v0.0.1
INFO: Installing HttpCommon v0.0.6
INFO: Installing HttpParser v0.0.9
INFO: Installing JSON v0.3.7
INFO: Installing Nettle v0.1.4
INFO: Installing Requests v0.0.5
INFO: Installing SHA v0.0.2
INFO: Installing URIParser v0.0.2
INFO: Building Nettle
INFO: Building GnuTLS
INFO: Building HttpParser
INFO: Package database updated
 - Library "libhttp_parser"


>>> 'using Requests' log
Warning: import of Dates.Dates into Dates conflicts with an existing identifier; ignored.
Julia Version 0.4.0-dev+376
Commit f522f8f (2014-08-27 03:03 UTC)
Platform Info:
  System: Linux (x86_64-unknown-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

>>> test log
Warning: import of Dates.Dates into Dates conflicts with an existing identifier; ignored.

ERROR: test error in expression: get("http://httpbin.org/get").status === 200
format not defined
 in process_response at /home/idunning/pkgtest/.julia/v0.4/Requests/src/Requests.jl:225
 in do_request at /home/idunning/pkgtest/.julia/v0.4/Requests/src/Requests.jl:563
 in get at /home/idunning/pkgtest/.julia/v0.4/Requests/src/Requests.jl:577
 in get at /home/idunning/pkgtest/.julia/v0.4/Requests/src/Requests.jl:576
 in anonymous at test.jl:81
 in do_test at test.jl:47
 in include at ./boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at ./client.jl:285
 in _start at ./client.jl:354
 in _start_3B_3593 at /home/idunning/julia04/usr/bin/../lib/julia/sys.so
while loading /home/idunning/pkgtest/.julia/v0.4/Requests/test/runtests.jl, in expression starting on line 9

>>> end of log

When you pass in data, no matter the form, "Content-Type" is overridden. We need to support other formats.

I have trying somtehing like that :

Requests.get("http://web.something.com/users"; headers = {"Accept-Encoding" => "gzip,deflate"} )

It does not seem to compress.

How to upgrade the GnuTLS in win ?
Paul.
_
_ _ ()_ | A fresh approach to technical computing
() | () () | Documentation: http://docs.julialang.org
_ _ | | __ _ | Type "help()" for help.
| | | | | | |/ ` | |
| | || | | | (| | | Version 0.3.6 (2015-02-17 22:12 UTC)
/ |_'|||__'| | Official http://julialang.org/ release
|__/ | x86_64-w64-mingw32

julia> Pkg.update()
INFO: Initializing package repository C:\Users\SAMSUNG2.julia\v0.3
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl
INFO: Updating METADATA...
INFO: Computing changes...
INFO: No packages to install, update or remove

julia> Pkg.add("WinRPM")
INFO: Cloning cache of BinDeps from git://github.com/JuliaLang/BinDeps.jl.git
INFO: Cloning cache of Compat from git://github.com/JuliaLang/Compat.jl.git
INFO: Cloning cache of LibExpat from git://github.com/amitmurthy/LibExpat.jl.git
INFO: Cloning cache of SHA from git://github.com/staticfloat/SHA.jl.git
INFO: Cloning cache of URIParser from git://github.com/JuliaWeb/URIParser.jl.git
INFO: Cloning cache of WinRPM from git://github.com/JuliaLang/WinRPM.jl.git
INFO: Cloning cache of Zlib from git://github.com/dcjones/Zlib.jl.git
INFO: Installing BinDeps v0.3.7
INFO: Installing Compat v0.2.12
INFO: Installing LibExpat v0.0.6
INFO: Installing SHA v0.0.3
INFO: Installing URIParser v0.0.4
INFO: Installing WinRPM v0.1.6
INFO: Installing Zlib v0.1.7
INFO: Building WinRPM
WARNING: skipping repodata/repomd.xml, not in cache -- call WinRPM.update() to download
WARNING: skipping repodata/repomd.xml, not in cache -- call WinRPM.update() to download
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1//repodata/repomd.x
ml
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1//repodata/f322c990
9de4fe377ec69678ced90b9b9ab17757d24d63ed94b36477a93e84f9-primary.xml.gz
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.1//repodata/repomd.x
ml
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.1//repodata/65d3ceb5
78b1243de965b54166774344236de22759295f438bb3302cbb42c5e1-primary.xml.gz
INFO: Packages to install: libgcc_s_seh1, libssp0, libstdc++6, libquadmath0, libgfortran3
INFO: Downloading: libgcc_s_seh1
INFO: Extracting: libgcc_s_seh1
INFO: Downloading: libssp0
INFO: Extracting: libssp0
INFO: Downloading: libstdc++6
INFO: Extracting: libstdc++6
INFO: Downloading: libquadmath0
INFO: Extracting: libquadmath0
INFO: Downloading: libgfortran3
INFO: Extracting: libgfortran3
INFO: Complete
INFO: Updated Julia's gcc dlls, you may need to restart Julia for some WinRPM packages to work.
INFO: Package database updated

julia> WinRPM.update()
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1//repodata/repomd.x
ml
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.1//repodata/repomd.x
ml

julia> Pkg.update()
INFO: Updating METADATA...
INFO: Computing changes...
INFO: No packages to install, update or remove

julia> using WinRPM

julia> WinRPM.search("Requests")
WinRPM Package Set:

julia> Pkg.add("Requests")
INFO: Cloning cache of Codecs from git://github.com/dcjones/Codecs.jl.git
INFO: Cloning cache of Dates from git://github.com/quinnj/Dates.jl.git
INFO: Cloning cache of GnuTLS from git://github.com/JuliaWeb/GnuTLS.jl.git
INFO: Cloning cache of HttpCommon from git://github.com/JuliaWeb/HttpCommon.jl.git
INFO: Cloning cache of HttpParser from git://github.com/JuliaWeb/HttpParser.jl.git
INFO: Cloning cache of JSON from git://github.com/JuliaLang/JSON.jl.git
INFO: Cloning cache of Nettle from git://github.com/staticfloat/Nettle.jl.git
INFO: Cloning cache of Requests from git://github.com/JuliaWeb/Requests.jl.git
INFO: Installing Codecs v0.1.3
INFO: Installing Dates v0.3.2
INFO: Installing GnuTLS v0.0.3
INFO: Installing HttpCommon v0.0.12
INFO: Installing HttpParser v0.0.10
INFO: Installing JSON v0.4.1
INFO: Installing Nettle v0.1.8
INFO: Installing Requests v0.0.8
INFO: Building WinRPM
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1//repodata/repomd.x
ml
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.1//repodata/repomd.x
ml
INFO: Building Nettle
INFO: Updating WinRPM package list
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.1//repodata/repomd.x
ml
INFO: Downloading http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.1//repodata/repomd.x
ml
INFO: Packages to install: libnettle-4-6
INFO: Downloading: libnettle-4-6
INFO: Extracting: libnettle-4-6
INFO: Complete
INFO: Building GnuTLS
INFO: Packages to install: libffi6, libtasn1-6, libp11-kit0, zlib1, libgmp10, libhogweed-2-4, gnutls-lang, libgnut
ls28, libintl8, gnutls
INFO: Downloading: libffi6
INFO: Extracting: libffi6
INFO: Downloading: libtasn1-6
INFO: Extracting: libtasn1-6
INFO: Downloading: libp11-kit0
INFO: Extracting: libp11-kit0
INFO: Downloading: zlib1
INFO: Extracting: zlib1
INFO: Downloading: libgmp10
INFO: Extracting: libgmp10
INFO: Downloading: libhogweed-2-4
INFO: Extracting: libhogweed-2-4
INFO: Downloading: gnutls-lang
INFO: Extracting: gnutls-lang
INFO: Downloading: libgnutls28
INFO: Extracting: libgnutls28
INFO: Downloading: libintl8
INFO: Extracting: libintl8
INFO: Downloading: gnutls
INFO: Extracting: gnutls
INFO: Complete
INFO: Building HttpParser
INFO: Attempting to Create directory C:\Users\SAMSUNG2.julia\v0.3\HttpParser\deps\downloads
INFO: Downloading file https://julialang.s3.amazonaws.com/bin/winnt/extras/libhttp_parser.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 77835 100 77835 0 0 44939 0 0:00:01 0:00:01 --:--:-- 44939
INFO: Done downloading file https://julialang.s3.amazonaws.com/bin/winnt/extras/libhttp_parser.zip
INFO: Attempting to Create directory C:\Users\SAMSUNG2.julia\v0.3\HttpParser
INFO: Directory C:\Users\SAMSUNG2.julia\v0.3\HttpParser already created
INFO: Attempting to Create directory C:\Users\SAMSUNG2.julia\v0.3\HttpParser\deps\usr

7-Zip [64] 9.20 Copyright (c) 1999-2010 Igor Pavlov 2010-11-18

Processing archive: C:\Users\SAMSUNG2.julia\v0.3\HttpParser\deps\downloads\libhttp_parser.zip

Extracting usr\lib
Extracting usr\lib\libhttp_parser32.dll
Extracting usr\lib\libhttp_parser64.dll

Everything is Ok

Folders: 1
Files: 2
Size: 222605
Compressed: 77835
INFO: Package database updated

julia> WinRPM.search("Requests")
WinRPM Package Set:

julia> using Requests
WARNING: This version of the GnuTLS library (3.2.15) is deprecated
and contains known security vulnerabilities. Please upgrade to a
more recent version.

julia>

I'm attempting to connect to a web service interface that can produce results as sparql-results+json (i.e. "Accept" => "sparql-results+json"). The same code works fine with other accept content types, it is though "sparql-results+json" is not considered valid and either removed or altered.

cred = "Basic $(base64("anonymous:anonymous"))"

heads = { "Authorization" => cred
          #, "Accept" => "text/turtle"  # works
          #, "Accept" => "application/rdf+xml"  # works
          , "Accept" => "application/sparql-results+json" # fails
        }

result = post(URI(url); data = query, headers = heads )
println("Headers: $(heads)")
println("Result headers: $(result.headers)")

Sample result: application/sparql-results+json (fails)

Headers: {"Accept"=>"application/sparql-results+json","Authorization"=>"Basic YW5vbnltb3VzOmFub255bW91cw=="}
Result headers: ["http_major"=>"1","Content-Length"=>"0","Content-Type"=>"text/html; charset=utf-8","Date"=>"Mon, 09 Mar 2015 21:45:22 GMT","Content-Language"=>"en","http_minor"=>"1","Keep-Alive"=>"1","Server"=>"Julia/0.3.4","status_code"=>"406"]

Sample result if with "application/rdf+xml" (success)

Headers: {"Accept"=>"application/rdf+xml","Authorization"=>"Basic YW5vbnltb3VzOmFub255bW91cw=="}
Result headers: ["Transfer-Encoding"=>"chunked","http_major"=>"1","Content-Type"=>"application/rdf+xml","Date"=>"Mon, 09 Mar 2015 21:40:20 GMT","Content-Language"=>"en","http_minor"=>"1","Keep-Alive"=>"1","Server"=>"Julia/0.3.4","status_code"=>"200"]

I'm getting, after updating to latest version:

Requests.get("http://httpbin.org/get")

ERROR: nowutc not defined
 in process_response at /Users/sacmac/.julia/v0.3/Requests/src/Requests.jl:225
 in do_request at /Users/sacmac/.julia/v0.3/Requests/src/Requests.jl:563
 in get at /Users/sacmac/.julia/v0.3/Requests/src/Requests.jl:577
 in get at /Users/sacmac/.julia/v0.3/Requests/src/Requests.jl:576

I'm sure Im missing something obvious here.

And imbed the button :)

WARNING: deprecated syntax "[a=>b, ...]" at /Users/Administrator/.julia/v0.4/Requests/src/Requests.jl:31.
Use "Dict(a=>b, ...)" instead.

WARNING: deprecated syntax "(String=>String)[a=>b, ...]" at /Users/Administrator/.julia/v0.4/Requests/src/Requests.jl:34.
Use "Dict{String,String}(a=>b, ...)" instead.

milktrader/Quandl.jl#61

If I checkout master I get the following errors:

julia> get("http://www.encodeproject.org")
ERROR: getaddrinfo callback: unknown node or service (EAI_NONAME)
 in getaddrinfo at ./socket.jl:597
 in open_stream at /Users/slund1/.julia/v0.4/Requests/src/streaming.jl:180
 in do_stream_request at /Users/slund1/.julia/v0.4/Requests/src/Requests.jl:334
 in do_stream_request at /Users/slund1/.julia/v0.4/Requests/src/Requests.jl:371 (repeats 3 times)
 in do_request at /Users/slund1/.julia/v0.4/Requests/src/Requests.jl:274
 in get at /Users/slund1/.julia/v0.4/Requests/src/Requests.jl:395
 in get at /Users/slund1/.julia/v0.4/Requests/src/Requests.jl:394
julia> get("http://www.google.com")
Response(200 OK, 15 headers, 157785 bytes in body)

but with the latest tagged release I get:

julia> get("http://www.encodeproject.org")
Response(200 OK, 18 headers, 30885 bytes in body)

I then added a println(uri.host) to streaming.jl at line 180 and got:

julia> get("https://www.encodeproject.org/search/?format=json")
www.encodeproject.org
www.encodeproject.org
www.encodeproject.orgwww.encodeproject.org
ERROR: getaddrinfo callback: no address (EAI_NODATA)
 in getaddrinfo at ./socket.jl:597
 in open_stream at /homes/gws/slund1/.julia/v0.4/Requests/src/streaming.jl:180
 in do_stream_request at /homes/gws/slund1/.julia/v0.4/Requests/src/Requests.jl:334
 in do_stream_request at /homes/gws/slund1/.julia/v0.4/Requests/src/Requests.jl:371 (repeats 2 times)
 in do_request at /homes/gws/slund1/.julia/v0.4/Requests/src/Requests.jl:274
 in get at /homes/gws/slund1/.julia/v0.4/Requests/src/Requests.jl:395
 in get at /homes/gws/slund1/.julia/v0.4/Requests/src/Requests.jl:394

Not sure what recent change this is related to, but I wanted to report it since it seems to only effect the resolution of certain domains and so might be missed.

Is there a way to periodically write the results to a file? I'm trying to use the Twitter streaming API, so there's no real "end" to the result set. I tried the following, with no luck:

#Function that does properly formatted OAuth Twitter call
function post_statuses_filter(; options = Dict())
    r = post_oauth("https://stream.twitter.com/1.1/statuses/filter.json", options)
end

#Open a file connection
tweettest = open("/Users/randyzwitch/Desktop/tweets.txt", "w")

#Write data to file
write(tweettest, post_statuses_filter(; options = {"track" => "#oscars"}))

I guess what I'm looking for is either a way to specify how long to keep the connection open and write to an in-memory object such as r above or a way to continuously write to text file as the data streams in.

Not sure how much it's an issue with Requests.jl, but it seems to be the only package that I can reproduce it with.

First of all, test code:

using Requests
using URIParser
using Gadfly

function fetch_urls(urls; timeout=Inf)
    statuses = -ones(length(urls))
    times = -ones(length(urls))
    @sync for i=1:length(urls)
        @async begin
            try
                t = @elapsed resp = get(urls[i]; timeout=timeout)
                statuses[i] = resp.status
                times[i] = t
                print("OK\n")
            catch
                print("Error\n")
            end
        end
    end
    return (statuses, times)
end

function main()
    @time urls = [strip(l) for l in
                  open(readlines, expanduser("~/Dropbox/ips.txt"))][1:100]
    @time (statuses, times) = fetch_urls(urls)
    plot(x=[i for i=1:length(times)], y=times)
end

This code uses current state of master of Requests.jl and file "ips.txt" available here.

Basically, here we take 100 URLs with already resolved IPs and try to fetch them, each in a separate task. Then we collect response statuses and request run times for analysis, and visualize later in Gadfly.

When I run this code, about 90% of requests get executed very quickly, altogether taking about 1 second. Obviously, even though Julia code is single-threaded, IO operations are executed in parallel for these requests.

However there's some little number of slow requests, each taking 5-15 seconds (and these are actually slow URLs, so even 15 seconds is believable here):

But the most interesting thing is that the whole function invocation (together with slow requests) takes more than 2 minutes! So it looks like fast requests are being executed in parallel, while slow ones start executing one by one or close to it.

I also added timeout by specifying timeout=2 in fetch_urls. As expected, long requests are truncated (-1 in the graph):

But total time is still 2 minutes!

So I don't know whether problem is in Requests.jl, Julia tasks or just my code, but any suggestions why it takes so much time are welcome.

PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their tests (if available) on both the stable version of Julia (0.3) and the nightly build of the unstable version (0.4). The results of this script are used to generate a package listing enhanced with testing results.

On Julia 0.4

• On 2014-08-15 the testing status was Tests pass.
• On 2014-08-16 the testing status changed to Package doesn't load.

Tests pass. means that PackageEvaluator found the tests for your package, executed them, and they all passed.

Package doesn't load. means that PackageEvaluator did not find tests for your package. Additionally, trying to load your package with using failed.

This issue was filed because your testing status became worse. No additional issues will be filed if your package remains in this state, and no issue will be filed if it improves. If you'd like to opt-out of these status-change messages, reply to this message saying you'd like to and @IainNZ will add an exception. If you'd like to discuss PackageEvaluator.jl please file an issue at the repository. For example, your package may be untestable on the test machine due to a dependency - an exception can be added.

Test log:

>>> 'Pkg.add("Requests")' log
 - Library "libhttp_parser"

INFO: Installing BinDeps v0.3.0
INFO: Installing Calendar v0.4.2
INFO: Installing Codecs v0.1.2
INFO: Installing FactCheck v0.1.2
INFO: Installing GnuTLS v0.0.1
INFO: Installing HttpCommon v0.0.5
INFO: Installing HttpParser v0.0.9
INFO: Installing ICU v0.4.1
INFO: Installing JSON v0.3.7
INFO: Installing Nettle v0.1.4
INFO: Installing Requests v0.0.5
INFO: Installing SHA v0.0.2
INFO: Installing URIParser v0.0.2
INFO: Building ICU
=================================[ ERROR: ICU ]=================================

None of the selected providers can install dependency icu.
Use BinDeps.debug(package_name) to see available providers

while loading /home/idunning/pkgtest/.julia/v0.4/ICU/deps/build.jl, in expression starting on line 28

================================================================================
INFO: Building Nettle
INFO: Building GnuTLS
INFO: Building HttpParser

================================[ BUILD ERRORS ]================================

WARNING: ICU had build errors.

 - packages with build errors remain installed in /home/idunning/pkgtest/.julia/v0.4
 - build a package and all its dependencies with `Pkg.build(pkg)`
 - build a single package by running its `deps/build.jl` script

================================================================================
INFO: Package database updated
INFO: METADATA is out-of-date a you may not have the latest version of Requests
INFO: Use `Pkg.update()` to get the latest versions of your packages

>>> 'using Requests' log

ERROR: could not open file /home/idunning/pkgtest/.julia/v0.4/ICU/src/../deps/deps.jl
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:54
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:54
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:54
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:54
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:51
 in include at ./boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at ./client.jl:285
 in _start at ./client.jl:354
 in _start_3B_3500 at /home/idunning/julia04/usr/bin/../lib/julia/sys.so
while loading /home/idunning/pkgtest/.julia/v0.4/ICU/src/ICU.jl, in expression starting on line 54
while loading /home/idunning/pkgtest/.julia/v0.4/Calendar/src/Calendar.jl, in expression starting on line 1
while loading /home/idunning/pkgtest/.julia/v0.4/HttpCommon/src/HttpCommon.jl, in expression starting on line 3
while loading /home/idunning/pkgtest/.julia/v0.4/HttpParser/src/HttpParser.jl, in expression starting on line 9
while loading /home/idunning/pkgtest/.julia/v0.4/Requests/src/Requests.jl, in expression starting on line 6
while loading /home/idunning/pkgtest/.julia/v0.4/Requests/testusing.jl, in expression starting on line 1

>>> test log
no tests to run
>>> end of log

I'm getting precompile failures with GeoIP, and I think they're in the Requests precompilation. From travis:

https://travis-ci.org/JuliaWeb/GeoIP.jl/jobs/87342701#L838-L856

Also ref fhs/ZipFile.jl#22

I have performed a clean build of the latest release-v0.4 branch of Julia. Afterwards I have added the requests library by using Pkg.add("Requests"). Testing the library (as well as using basic functionality, such as get()) causes errors on my system (Red Hat enterprise linux 7):

julia> Pkg.init()
INFO: Initializing package repository /local/.julia/v0.4
INFO: Cloning METADATA from git://github.com/JuliaLang/METADATA.jl

julia> Pkg.add("Requests")
INFO: Installing BinDeps v0.3.15
INFO: Installing Codecs v0.1.4
INFO: Installing Compat v0.7.0
INFO: Installing HttpCommon v0.2.3
INFO: Installing HttpParser v0.1.0
INFO: Installing JSON v0.4.5
INFO: Installing MbedTLS v0.1.3
INFO: Installing Requests v0.3.1
INFO: Installing SHA v0.1.1
INFO: Installing URIParser v0.1.0
INFO: Installing Zlib v0.1.9
INFO: Building HttpParser
INFO: Precompiling module Compat...
INFO: Precompiling module URIParser...
INFO: Precompiling module SHA...
INFO: Building MbedTLS
INFO: Attempting to Create directory /local/.julia/v0.4/MbedTLS/deps/downloads
INFO: Downloading file https://tls.mbed.org/download/mbedtls-2.1.0-apache.tgz
....... ommitted a lot of compile information from MbedTLS .....
[100%] Building C object tests/CMakeFiles/test_suite_xtea.dir/test_suite_xtea.c.o
Linking C executable test_suite_xtea
[100%] Built target test_suite_xtea
INFO: Package database updated

julia> Pkg.test("Requests")
INFO: Testing Requests
ERROR (unhandled task failure): ArgumentError: cannot convert NULL to string
 in on_header_value at /local/.julia/v0.4/Requests/src/parsing.jl:92

The tests do not continue after the error message. I get the same error when trying to use basic HTTP functionality, e.g. get("https://www.google.com/").

Any idea what could cause this?

My issue is with this site. Specifically, I need to use the ajaxauth/login address for logging in, and I can't get this to work:

resp = post("https://www.space-track.org/ajaxauth/login",data="identity=username&password=password")
Response(401 Unauthorized, 11 Headers, 69 Bytes in Body)

I tried this with the Python requests package, and it worked as expected:

cred = {'identity':'username' , 'password':'password'}
r = requests.post("https://www.space-track.org/ajaxauth/login",data = cred)

(result is Response 200)

the body of the request is the same (except that password comes before username):

julia>ASCIIString(resp.requests[1].data)
"identity=username&password=password"

Python:

>>>r.request.body
'password=password&identity=username'

I also tried switching the order of password and identity, to no avail. I am not even sure this is an issue with Requests.jl, but I figured I'd start here.

Thanks.

The packaged GnuTLS seems to be of an old version, so I can't complete requests to a website that uses the latest TLS SNI technology (e.g. any website protected by cloudflare). Do you have advice on getting this to work?

Julia Version 0.4.0-dev+4137
Commit d61c7e4* (2015-04-02 13:30 UTC)
Platform Info:
System: Darwin (x86_64-apple-darwin14.1.1)
CPU: Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT NO_AFFINITY HASWELL)
LAPACK: libopenblas
LIBM: libopenlibm
LLVM: libLLVM-3.3

ERROR: LoadError: test error in expression: get("http://httpbin.org/get").status === 200
MethodError: `convert` has no method matching convert(::Type{Ptr{UInt8}}, ::ASCIIString)
This may have arisen from a call to the constructor Ptr{UInt8}(...),
since type constructors fall back to convert methods.
Closest candidates are:
  convert{T}(::Type{Ptr{T}}, ::UInt64)
  convert{T}(::Type{Ptr{T}}, ::Int64)
  convert{T}(::Type{Ptr{T}}, ::Ptr{T})
  ...
 in http_parser_execute at /Users/seth/.julia/v0.4/HttpParser/src/HttpParser.jl:106
 in process_response at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:237
 in do_request at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:569
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:583
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:582
 in anonymous at test.jl:87
 in do_test at test.jl:47
 in include at /usr/local/julia-latest/lib/julia/sys.dylib
 in include_from_node1 at loading.jl:129
 in process_options at /usr/local/julia-latest/lib/julia/sys.dylib
 in _start at /usr/local/julia-latest/lib/julia/sys.dylib
while loading /Users/seth/.julia/v0.4/Requests/test/runtests.jl, in expression starting on line 10

This also impacts get():

julia> z = get("http://httpbin.org/get")
WARNING: [a,b,...] concatenation is deprecated; use [a;b;...] instead
 in depwarn at /usr/local/julia-latest/lib/julia/sys.dylib
 in oldstyle_vcat_warning at /usr/local/julia-latest/lib/julia/sys.dylib
 in vect at abstractarray.jl:35
 in render at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:23
 in open_stream at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:224
 in open_stream at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:207
 in do_request at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:569
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:583
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:582
WARNING: convert{T}(p::Type{Ptr{T}},a::Array) is deprecated, use convert(p,pointer(a)) instead.
 in depwarn at /usr/local/julia-latest/lib/julia/sys.dylib
 in convert at deprecated.jl:29
 in http_parser_execute at /Users/seth/.julia/v0.4/HttpParser/src/HttpParser.jl:106
 in process_response at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:234
 in do_request at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:569
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:583
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:582
WARNING: int(x) is deprecated, use Int(x) instead.
 in depwarn at /usr/local/julia-latest/lib/julia/sys.dylib
 in int at /usr/local/julia-latest/lib/julia/sys.dylib
 in on_header_value at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:122
 in http_parser_execute at /Users/seth/.julia/v0.4/HttpParser/src/HttpParser.jl:106
 in process_response at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:234
 in do_request at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:569
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:583
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:582
ERROR: MethodError: `convert` has no method matching convert(::Type{Ptr{UInt8}}, ::ASCIIString)
This may have arisen from a call to the constructor Ptr{UInt8}(...),
since type constructors fall back to convert methods.
Closest candidates are:
  convert{T}(::Type{Ptr{T}}, ::UInt64)
  convert{T}(::Type{Ptr{T}}, ::Int64)
  convert{T}(::Type{Ptr{T}}, ::Ptr{T})
  ...
 in http_parser_execute at /Users/seth/.julia/v0.4/HttpParser/src/HttpParser.jl:106
 in process_response at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:237
 in do_request at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:569
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:583
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:582

I'm investigating version 0.4, just spotted this - do I have to update something else?:

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.0-dev+6991 (2015-08-26 04:02 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 811a977 (0 days old master)
|__/                   |  x86_64-apple-darwin13.4.0

julia> using Requests
WARNING: could not import Base.put into Requests

julia> Pkg.status()
2 required packages:
 - JSON                          0.4.5
 - Requests                      0.0.8
11 additional packages:
 - BinDeps                       0.3.15
 - Codecs                        0.1.4
 - Compat                        0.6.0
 - Dates                         0.4.4
 - GnuTLS                        0.0.4
 - Homebrew                      0.1.16
 - HttpCommon                    0.0.12
 - HttpParser                    0.0.11
 - Nettle                        0.1.10
 - SHA                           0.1.1
 - URIParser                     0.0.5

By the way, these warnings (in case you've not seen them recently):

 WARNING: [a,b,...] concatenation is deprecated; use [a;b;...] instead
 in depwarn at /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib
 in oldstyle_vcat_warning at /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib
 in vect at abstractarray.jl:38

 WARNING: int(x) is deprecated, use Int(x) instead.
 in depwarn at /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib
 in int at /Applications/Julia-0.4.0.app/Contents/Resources/julia/lib/julia/sys.dylib

I'm cross-posting from https://groups.google.com/forum/#!topic/julia-users/M96DSCArIds

Basically, I call

curl -X POST -d 'grant_type=password&username=[MYNAME]&password=[MYPASS]' --user 'CLIENTID:CLIENTSECRET' https://www.reddit.com/api/v1/access_token

from my terminal. This gives me an OAuth token, so I can interact with the Reddit API. It works fine. I am using it in a Reddit.jl-script/interface in Julia, so I wanted to just use Julia to fetch the token as well.

The problem is that

enc = bytestring(encode(Base64, "CLIENTID:CLIENTSECRET"))
post(URI("https://www.reddit.com/api/v1/access_token"), "grant_type=password&username=[MYNAME]&password=[MYPASS]"; headers = Dict("Authorization" => "Basic $enc", "User-Agent" => UserAgent))
% UserAgent is a string here

Does not do the same thing as the curl-call from above. Am I doing something wrong here? Maybe I've just misunderstood the Requests.jl syntax.

I wanted to debug my code, but I could not find a way to print my POST request before sending it. Is this possible?

Thanks!

Yesterday all worked as expected. The following code would return 'object' as a JSON array.
Today line 3 fails with: 'type: typeassert: expected String, got Array{Uint8,1}'

I just wanted to check whether there's any change at this end. My url appears to be returning the same data when run in a browser.

edit: I'm running in juliabox.org and this occurs with both a fresh session and an updated session (with Pkg.update() )

resp = get(url)
println(resp.data)
object = JSON.parse(resp.data::String; ordered=true);

See JuliaIO/JSON.jl#74 for instructions on AppVeyor (you may need to refresh AppVeyor to reflect your changed github handle?)

@garborg and I have been playing with GitHub.jl and we noticed that put requests to the GitHub API seem fail with the below error unless data is passed along in the request.

ERROR: stream is closed or unusable
 in check_open at stream.jl:294
 in write at stream.jl:729
 in close at /Users/westley/.julia/v0.3/GnuTLS/src/GnuTLS.jl:96
 in on_message_complete at /Users/westley/.julia/v0.3/Requests/src/Requests.jl:151
 in http_parser_execute at /Users/westley/.julia/v0.3/HttpParser/src/HttpParser.jl:106
 in process_response at /Users/westley/.julia/v0.3/Requests/src/Requests.jl:228
 in do_request at /Users/westley/.julia/v0.3/Requests/src/Requests.jl:562
 in put at /Users/westley/.julia/v0.3/Requests/src/Requests.jl:576
 in star at /Users/westley/.julia/v0.3/GitHub/src/starring.jl:54
 in star at /Users/westley/.julia/v0.3/GitHub/src/starring.jl:45

Probing further, we've found that making the request using Postman works just fine (to test, try making a PUT request to https://api.github.com/user/starred/$YOUR_USER/github.jl with the "Authorization" header set to "token ***YOUR_TOKEN***").

That leads me to believe the problem is somewhere in Requests.jl or the underlying http libraries. To repo using just Reqests.jl:

# will work just fine
put("https://api.github.com/user/starred/$YOUR_USER/github.jl"; data = "{}", headers = {"Authorization"=>"token ***YOUR_TOKEN***"})

# will hang and eventually error
put("https://api.github.com/user/starred/$YOUR_USER/github.jl"; headers = {"Authorization"=>"token ***YOUR_TOKEN***"})

I accidentally updated all packages (Pkg.update()) and I think that the latest version of Requests.jl might not be working, with errors such as:

ERROR: getaddrinfo callback: unknown node or service (EAI_NONAME)
  in getaddrinfo at socket.jl:538

The new version contains this changed function:

function render(request::Request)
    join(@compat [
        request.method*" "*(isempty(request.resource)?"/":request.resource)*" HTTP/1.1";
        map(h->(h*": "*request.headers[h]),collect(keys(request.headers)));
        "";
        request.data],CRLF)
end

which replaces the old version:

function render(request::Request)
   join([
       request.method*" "*(isempty(request.resource)?"/":request.resource)*" HTTP/1.1",
       map(h->(h*": "*request.headers[h]),collect(keys(request.headers))),
      "",
       request.data],CRLF)
end

If I swap the old version of the function back in, it works again.

I wish I understood what was going on, then I'd be able to suggest a fix, rather than just hard-wire my version to work again... :)

Can https requests be made using Requests.jl?

For the reasons oblivious to me, handshake doesn't occur for https requests.

julia> get("http://github.com")
get("http://github.com")
Response(301 Moved Permanently, 11 Headers, 0 Bytes in Body)

julia> get("https://github.com")
get("https://github.com")
ERROR: read! not defined
 in read_ptr at c:\Users\User\.julia\v0.2\GnuTLS\src\GnuTLS.jl:285
 in handshake! at c:\Users\User\.julia\v0.2\GnuTLS\src\GnuTLS.jl:302
 in open_stream at c:\Users\User\.julia\v0.2\Requests\src\Requests.jl:217

I'm not behind any proxy. And, works with Python's requests library

import requests
requests.get('https://github.com').status_code
200

will fix #83

Requests.head appears to hang and never come back, I''ll take a look, but is this a known issue ?

Should submit a PR, I know, but no time atm

I just got this warning...

WARNING: convert{T}(p::Type{Ptr{T}},a::Array) is deprecated, use convert(p,pointer(a)) instead.
 in convert at deprecated.jl:26
 in Parser at /Users/randyzwitch/.julia/v0.3/HttpParser/src/HttpParser.jl:48
 in ResponseParser at /Users/randyzwitch/.julia/v0.3/Requests/src/Requests.jl:54
 in process_response at /Users/randyzwitch/.julia/v0.3/Requests/src/Requests.jl:221
 in get at /Users/randyzwitch/.julia/v0.3/Requests/src/Requests.jl:260
 in get_oauth at /Users/randyzwitch/.julia/v0.3/Twitter/src/general.jl:89
 in get_direct_messages at /Users/randyzwitch/.julia/v0.3/Twitter/src/dm.jl:9
 in include_string at loading.jl:89
 in execute_request_0x535c5df2 at /Users/randyzwitch/.julia/v0.3/IJulia/src/execute_request.jl:139
 in eventloop at /Users/randyzwitch/.julia/v0.3/IJulia/src/IJulia.jl:68
 in anonymous at task.jl:329

We can't use a dict to represent headers, as a server can send back multiples of headers, such as Set-Cookie, which need to get merged together, possibly by higher logic. E.g. github.com sets:

Set-Cookie: user_session=JuN1cxdfprx8gJvQOOIYN6_DTuMGOBcmRF-BH7D-Zp8yIxtQU__NSZeOWLU2SvUew98aOqu3mHrnwPvh; path=/; expires=Fri, 12-Sep-2014 00:46:01 GMT; secure; HttpOnly
Set-Cookie: logged_in=yes; domain=.github.com; path=/; expires=Tue, 29-Aug-2034 00:46:01 GMT; secure; HttpOnly
Set-Cookie: dotcom_user=staticfloat; domain=.github.com; path=/; expires=Tue, 29-Aug-2034 00:46:01 GMT; secure; HttpOnly
Set-Cookie: _gh_sess=cyJsYXN0X3dyaXRlIjoxNDA5MjczMTYxNzU5LCJzZXNzaW9uX2lkIjoiYmZhNDdlM2U2NWQ2M2ZhODkxMjNiM2YyOWIxN2U5YjgifQ%3D%3D--9a25437789d4646087d3b682468db8047f3e2533; path=/; secure; HttpOnly

And expects to get back in following requests:

Cookie: tz=America%2FNew_York; user_session=JuN1cxdfprx8gJvQOOIYN6_DTuMGOBcmRF-BH7D-Zp8yIxtQU__NSZeOWLU2SvUew98aOqu3mHrnwPvh; logged_in=yes; dotcom_user=staticfloat; _gh_sess=cyJsYXN0X3dyaXRlIjoxNDA5MjczMTYxNzU5LCJzZXNzaW9uX2lkIjoiYmZhNDdlM2U2NWQ2M2ZhODkxMjNiM2YyOWIxN2U5YjgifQ%3D%3D--9a25437789d4646087d3b682468db8047f3e2533

Relevant code pasted here for convenience:

    function process_response(stream)
        r = Response()
        rp = ResponseParser(r,stream)
        while isopen(stream)
            data = readavailable(stream)
            add_data(rp, data)
        end
        http_parser_execute(rp.parser,rp.settings,"") #EOF
        r
    end

This frequently (but not always!) results in a lockup:

julia> using Requests

julia> @time g=get("https://github.com")  <---- the REPL completely locks here.

However, when I put a 0.005-second sleep in before add_data, I get no lockup.

julia> @time g=get("https://github.com")
elapsed time: 4.772878302 seconds (262 MB allocated, 0.81% gc time in 12 pauses with 0 full sweep)
Response(200 OK, 25 Headers, 16965 Bytes in Body)

New versions of Requests require a new minimum version of HttpCommon, but Pkg.update() does not realize that HttpCommon needs to be upgraded.

Otherwise, users on old incompatible HttpCommon versions will get errors like

ERROR: LoadError: type Response has no field request
 in open_stream at /home/ubuntu/.julia/v0.5/Requests/src/streaming.jl:209
 in do_stream_request at /home/ubuntu/.julia/v0.5/Requests/src/Requests.jl:334
 in do_request at /home/ubuntu/.julia/v0.5/Requests/src/Requests.jl:274
 [inlined code] from /home/ubuntu/.julia/v0.5/Requests/src/Requests.jl:274
 in post at /home/ubuntu/.julia/v0.5/Requests/src/Requests.jl:395
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:392
 in process_options at ./client.jl:277
 in _start at ./client.jl:377
while loading /home/ubuntu/pingjuliaslack/report-slack.jl, in expression starting on line 39

It'd be great to have a way to understand what the request looks like instead of just sending it. For example, Twitter provides this example, but after receiving 403 errors as the response, I have no good way to validate that my POST request matches this:

POST /oauth2/token HTTP/1.1
Host: api.twitter.com
User-Agent: My Twitter App v1.0.23
Authorization: Basic eHZ6MWV2RlM0d0VFUFRHRUZQSEJvZzpMOHFxOVBaeVJn
                     NmllS0dFS2hab2xHQzB2SldMdzhpRUo4OERSZHlPZw==
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Content-Length: 29
Accept-Encoding: gzip

grant_type=client_credentials

julia> using Requests
ERROR: JSON not found
 in require at loading.jl:39
 in include at boot.jl:240
while loading /home/idunning/.julia/Requests/src/Requests.jl, in expression starting on line 10

METADATA wrong?

PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their tests (if available) on both the stable version of Julia (0.3) and the nightly build of the unstable version (0.4). The results of this script are used to generate a package listing enhanced with testing results.

On Julia 0.3

• On 2014-08-15 the testing status was Tests pass.
• On 2014-08-16 the testing status changed to Package doesn't load.

Tests pass. means that PackageEvaluator found the tests for your package, executed them, and they all passed.

Package doesn't load. means that PackageEvaluator did not find tests for your package. Additionally, trying to load your package with using failed.

This issue was filed because your testing status became worse. No additional issues will be filed if your package remains in this state, and no issue will be filed if it improves. If you'd like to opt-out of these status-change messages, reply to this message saying you'd like to and @IainNZ will add an exception. If you'd like to discuss PackageEvaluator.jl please file an issue at the repository. For example, your package may be untestable on the test machine due to a dependency - an exception can be added.

Test log:

>>> 'Pkg.add("Requests")' log
 - Library "libhttp_parser"

INFO: Installing BinDeps v0.3.0
INFO: Installing Calendar v0.4.2
INFO: Installing Codecs v0.1.2
INFO: Installing FactCheck v0.1.2
INFO: Installing GnuTLS v0.0.1
INFO: Installing HttpCommon v0.0.5
INFO: Installing HttpParser v0.0.9
INFO: Installing ICU v0.4.1
INFO: Installing JSON v0.3.7
INFO: Installing Nettle v0.1.4
INFO: Installing Requests v0.0.5
INFO: Installing SHA v0.0.2
INFO: Installing URIParser v0.0.2
INFO: Building ICU
=================================[ ERROR: ICU ]=================================

None of the selected providers can install dependency icu.
Use BinDeps.debug(package_name) to see available providers

while loading /home/idunning/pkgtest/.julia/v0.3/ICU/deps/build.jl, in expression starting on line 28

================================================================================
INFO: Building Nettle
INFO: Building GnuTLS
INFO: Building HttpParser

================================[ BUILD ERRORS ]================================

WARNING: ICU had build errors.

 - packages with build errors remain installed in /home/idunning/pkgtest/.julia/v0.3
 - build a package and all its dependencies with `Pkg.build(pkg)`
 - build a single package by running its `deps/build.jl` script

================================================================================
INFO: Package database updated
INFO: METADATA is out-of-date a you may not have the latest version of Requests
INFO: Use `Pkg.update()` to get the latest versions of your packages

>>> 'using Requests' log

ERROR: could not open file /home/idunning/pkgtest/.julia/v0.3/ICU/src/../deps/deps.jl
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:54
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:54
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:54
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:54
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in reload_path at loading.jl:152
 in _require at loading.jl:67
 in require at loading.jl:51
 in include at ./boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at ./client.jl:285
 in _start at ./client.jl:354
 in _start_3B_1699 at /home/idunning/julia03/usr/bin/../lib/julia/sys.so
while loading /home/idunning/pkgtest/.julia/v0.3/ICU/src/ICU.jl, in expression starting on line 54
while loading /home/idunning/pkgtest/.julia/v0.3/Calendar/src/Calendar.jl, in expression starting on line 1
while loading /home/idunning/pkgtest/.julia/v0.3/HttpCommon/src/HttpCommon.jl, in expression starting on line 3
while loading /home/idunning/pkgtest/.julia/v0.3/HttpParser/src/HttpParser.jl, in expression starting on line 9
while loading /home/idunning/pkgtest/.julia/v0.3/Requests/src/Requests.jl, in expression starting on line 6
while loading /home/idunning/pkgtest/.julia/v0.3/Requests/testusing.jl, in expression starting on line 1

>>> test log
no tests to run
>>> end of log

I don't have push access, didn't want to just submit the changes to metadata without making the tag here as well.

Every once in a while when I have some long-running post requests, I get the following

ERROR: write: broken pipe (EPIPE)
 in wait at ./task.jl:305
 in stream_wait at ./stream.jl:273
 in uv_write at stream.jl:754
 in buffer_or_write at stream.jl:767
 in write at stream.jl:814
 in write_ptr at /Users/jacobquinn/.julia/v0.4/GnuTLS/src/GnuTLS.jl:307
 in write at /Users/jacobquinn/.julia/v0.4/GnuTLS/src/GnuTLS.jl:373
 in write at utf8.jl:196
 in open_stream at /Users/jacobquinn/.julia/v0.4/Requests/src/Requests.jl:223
 in open_stream at /Users/jacobquinn/.julia/v0.4/Requests/src/Requests.jl:206
 in do_request at /Users/jacobquinn/.julia/v0.4/Requests/src/Requests.jl:568
 in post at /Users/jacobquinn/.julia/v0.4/Requests/src/Requests.jl:582
 in post at /Users/jacobquinn/.julia/v0.4/Requests/src/Requests.jl:581

Does anybody know if this is something we should try and handle in the Requests library? or is it a problem of the server I'm connecting to? @Keno @staticfloat

It looks like Requests doesn't look at the http_proxy or https_proxy environment variables, or have any way to pass these to it. Is there any way to include this?

Failure of getting the finished flag field:
e.g.

using Requests
z=Requests.get("http://valid url/")

ERROR: type Response has no field finished
in on_message_complete at /.julia/v0.4/Requests/src/Requests.jl:219

Regards Johan

I'm frequently and unpredictably getting this when calling get:

ERROR: write: broken pipe (EPIPE)
 in write at stream.jl:760
 in close at /Users/malmaud/.julia/GnuTLS/src/GnuTLS.jl:96
 in readtobuf at /Users/malmaud/.julia/GnuTLS/src/GnuTLS.jl:378
 in readavailable at /Users/malmaud/.julia/GnuTLS/src/GnuTLS.jl:401
 in process_response at /Users/malmaud/.julia/Requests/src/Requests.jl:221
 in get at /Users/malmaud/.julia/Requests/src/Requests.jl:258
 in get at /Users/malmaud/.julia/Requests/src/Requests.jl:249
 in anonymous at no file:13
 in include_from_node1 at loading.jl:120
while loading /Users/malmaud/Dropbox/Documents/code/julia_stats/scrape_issues.jl, in expression starting on line 9

using DataFrames currently does not generate any warnings but when used with Requests there is a wide range of operator ambiguities.

julia> using DataFrames, Requests
Warning: New definition
    +(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N}) at operators.jl:376
is ambiguous with:
    +(DataArray{T,N},AbstractArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:326.
To fix, define
    +(DataArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N})
before the new definition.
Warning: New definition
    +(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N}) at operators.jl:376
is ambiguous with:
    +(AbstractDataArray{T,N},AbstractArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:350.
To fix, define
    +(AbstractDataArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N})
before the new definition.
Warning: New definition
    +(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N}) at operators.jl:376
is ambiguous with:
    +(AbstractArray{T,N},DataArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:326.
To fix, define
    +(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},DataArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N})
before the new definition.
Warning: New definition
    +(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N}) at operators.jl:376
is ambiguous with:
    +(AbstractArray{T,N},AbstractDataArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:350.
To fix, define
    +(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractDataArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N})
before the new definition.
Warning: New definition
    -(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N}) at operators.jl:376
is ambiguous with:
    -(DataArray{T,N},AbstractArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:326.
To fix, define
    -(DataArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N})
before the new definition.
Warning: New definition
    -(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N}) at operators.jl:376
is ambiguous with:
    -(AbstractArray{T,N},DataArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:326.
To fix, define
    -(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},DataArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N})
before the new definition.
Warning: New definition
    -(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N}) at operators.jl:376
is ambiguous with:
    -(AbstractDataArray{T,N},AbstractArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:350.
To fix, define
    -(AbstractDataArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N})
before the new definition.
Warning: New definition
    -(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N}) at operators.jl:376
is ambiguous with:
    -(AbstractArray{T,N},AbstractDataArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:350.
To fix, define
    -(AbstractArray{T1<:Union(CalendarTime,AbstractCalendarDuration),N},AbstractDataArray{T2<:Union(CalendarTime,AbstractCalendarDuration),N})
before the new definition.
Warning: New definition
    -(AbstractArray{T1<:CalendarTime,N},AbstractArray{T2<:CalendarTime,N}) at operators.jl:376
is ambiguous with:
    -(DataArray{T,N},AbstractArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:326.
To fix, define
    -(DataArray{T1<:CalendarTime,N},AbstractArray{T2<:CalendarTime,N})
before the new definition.
Warning: New definition
    -(AbstractArray{T1<:CalendarTime,N},AbstractArray{T2<:CalendarTime,N}) at operators.jl:376
is ambiguous with:
    -(AbstractArray{T,N},DataArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:326.
To fix, define
    -(AbstractArray{T1<:CalendarTime,N},DataArray{T2<:CalendarTime,N})
before the new definition.
Warning: New definition
    -(AbstractArray{T1<:CalendarTime,N},AbstractArray{T2<:CalendarTime,N}) at operators.jl:376
is ambiguous with:
    -(AbstractDataArray{T,N},AbstractArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:350.
To fix, define
    -(AbstractDataArray{T1<:CalendarTime,N},AbstractArray{T2<:CalendarTime,N})
before the new definition.
Warning: New definition
    -(AbstractArray{T1<:CalendarTime,N},AbstractArray{T2<:CalendarTime,N}) at operators.jl:376
is ambiguous with:
    -(AbstractArray{T,N},AbstractDataArray{T,N}) at /Users/Administrator/.julia/v0.3/DataArrays/src/operators.jl:350.
To fix, define
    -(AbstractArray{T1<:CalendarTime,N},AbstractDataArray{T2<:CalendarTime,N})
before the new definition.

julia> Pkg.build("Requests")
INFO: Building HttpParser

julia> using Requests
ERROR: precompile not defined
in include at /Applications/Julia-0.3.11.app/Contents/Resources/julia/lib/julia/sys.dylib
in include_from_node1 at /Applications/Julia-0.3.11.app/Contents/Resources/julia/lib/julia/sys.dylib
in reload_path at loading.jl:152
in _require at loading.jl:67
in require at loading.jl:51
while loading /Users/jowanza/.julia/v0.3/Requests/src/Requests.jl, in expression starting on line 1

eg
files = {"json_file"=>data},

As discussed in JuliaLang/julia#10274, Response.data cannot always be guaranteed to return a valid UTF8 string since an HTTP request can return arbitrary binary data. Typing the result as a UTF8String can cause downstream errors due to having an invalid string.

julia> using Requests; get("https://avatars.githubusercontent.com/u/744556?v=3")
Response(200 OK, 28 Headers, 29365 Bytes in Body)

julia> ans.data
"\U1a01df90JFIF\0\x01\x01\0\0\x01\0\x01\0\0\U3bfba092EATOR: gd-jpeg v1.0 (using IJG JPEG v80), quality = 90\n\U18fc0f83\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\b\x05\x05\x04\x04\x05\n\a\a\x06\b\f\n\f\f\v\n\v\v\r\x0e\x12\x10\r\x0e\x11\x0e\v\v\x10\x16\x10\x11\x13\x14\x15\x15\x15\f\x0f\x17\x18\x16\x14\x18\x12\x14\x15\x14\U18fc0fc3\x04\x04\x05\x04\x05\t\x05\x05\t\x14\r\v\r\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\Ufdf8f181ʁʃ\x01\"\0\x02\x11\x01\x03\x11\x01\U1f9cf80\x01\x05\x01\x01\x01\x01\x01\x01\0\0\0\0\0\0\0\0\x01\x02\x03\x04\x05\x06\a\b\t\n\v\U2033380\x02\x01\x03\x03\x02\x04\x03\x05\x05\x04\x04\0\0\x01}\x01\x02\x03\0\x04\x11\x05\x12!1A\x06\x13Qa\a\"q\x142���\b#B�\Uffffffd5RҰ\$3br�\t\n\x16\x17\x18\x19\x1a%&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz�����������������������������������ÃŅLJ\u249˒\u514֖\u618ښ⣣妦詩벲\U135db6\U239eba\U1f9cfc0\x03\x01\x01\x01\x01\x01\x01\x01\x01\x01\0\0\0\0\0\0\x01\x02\x03\x04\x05\x06\a\b\t\n\v\U20333c0\x02\x01\x02\x04\x04\x03\x04\a\x05\x04\x04\0\x01\x02w\0\x01\x02\x03\x11\x04\x05!1\x06\x12AQ\aaq\x13\"2�\b\x14B���\Uffffffc9#3R\Ufff94872ϊ\x16\$4\Ufffff9b1\x17\x18\x19\x1a&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz������������������������������������ÃŅLJ\u249˒\u514֖\u618ښ㤤槧骪󴵵\U1f8e79\U4017f8c\x03\x01\0\x02\x11\x03\x11\0?\0???ү+�\Ue0429J��\f�_\x18z�\\#뼪ӄ0\x02��ܴԱ0�\x0f�\0\$�ː\x0fZ\n쳝󱟘Error showing value of type UTF8String:
ERROR: invalid UTF-8 character index
 in next at ./utf8.jl:68
 in print_escaped at string.jl:869
 in show at string.jl:69
 in anonymous at show.jl:1160
 in with_output_limit at ./show.jl:1137
 in showlimited at show.jl:1159
 in writemime at replutil.jl:2
 in display at REPL.jl:117
 in display at REPL.jl:120
 in display at multimedia.jl:149
 in print_response at REPL.jl:139
 in print_response at REPL.jl:124
 in anonymous at REPL.jl:586
 in run_interface at ./LineEdit.jl:1379
 in run_frontend at ./REPL.jl:818
 in run_repl at ./REPL.jl:169
 in _start at ./client.jl:400

julia> versioninfo()
Julia Version 0.3.7-pre+1
Commit d15f183 (2015-02-17 22:12 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin14.1.0)
  CPU: Intel(R) Core(TM) i5-4258U CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

julia> Pkg.installed("Requests")
v"0.0.8"

This issue is being filed by a script, but if you reply, I will see it.

PackageEvaluator.jl is a script that runs nightly. It attempts to load all Julia packages and run their test (if available) on both the stable version of Julia (0.2) and the nightly build of the unstable version (0.3).

The results of this script are used to generate a package listing enhanced with testing results.

The status of this package, Requests, on...

• Julia 0.2 is 'Tests fail, but package loads.'
• Julia 0.3 is 'Tests pass.'

'No tests, but package loads.' can be due to their being no tests (you should write some if you can!) but can also be due to PackageEvaluator not being able to find your tests. Consider adding a test/runtests.jl file.

'Package doesn't load.' is the worst-case scenario. Sometimes this arises because your package doesn't have BinDeps support, or needs something that can't be installed with BinDeps. If this is the case for your package, please file an issue and an exception can be made so your package will not be tested.

This automatically filed issue is a one-off message. Starting soon, issues will only be filed when the testing status of your package changes in a negative direction (gets worse). If you'd like to opt-out of these status-change messages, reply to this message.

I'm blatantly and unapologetically stealing this from the python-requests library. Something like

get("http://github.com"; timeout=0.001)

Problems with percent-encoded queries

Requests uses URIs under the hood since it relies on URIParser. Strictly speaking, URIs are not allowed to have Unicode characters in them, but instead require most characters to be percent-encoded. However, there is currently something wrong with how percent-encoded data is handled.

Consider the URI

http://nominatim.openstreetmap.org/search?format=json&q=%E6%B7%B1%E5%9C%B3

where the query value (after q=) is percent encoding for "深圳" (Shenzhen). If I paste this string into my Chrome browser's address bar, it automatically gets converted to

http://nominatim.openstreetmap.org/search?format=json&q=深圳

However, the equivalent request

get("http://nominatim.openstreetmap.org/search", query=Dict("format"=>"json","q"=>"%E3%82%A2"))

returns an empty response.

I can hack URIParser to allow URI.query to be of type String instead of ASCIIString, in which case

get("http://nominatim.openstreetmap.org/search", query=Dict("format"=>"json","q"=>"深圳"))

works as expected, but then the URI object being constructed and used under the hood is no longer standards compliant.

I've tried looking under the hood and it looks like there is the function format_query_str exists to handle non-ASCIIString queries, but this never seems to get called with q=深圳. With q=%E3%82%A2 it gets called and gets percent-encoded a second time into q=%25E6%25B7%25B1%25E5%259C%25B3, which makes me very sad.

In the longer term, we can consider switching over to IRIs, which allow native UTF-8 encoding.

cc @Keno

julia> z = get("https://github.com")
ERROR: KeyError: current_header not found
 in on_header_value at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:123
 in http_parser_execute at /Users/seth/.julia/v0.4/HttpParser/src/HttpParser.jl:106
 in process_response at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:231
 in do_request at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:565
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:579
 in get at /Users/seth/.julia/v0.4/Requests/src/Requests.jl:578

This is with GnuTLS and the ird/tempfix branch.