juliaweb / http.jl Goto Github PK

View Code? Open in Web Editor NEW

627.0 627.0 177.0 4.52 MB

HTTP for Julia

Home Page: https://juliaweb.github.io/HTTP.jl/stable/

License: Other

Julia 99.60% HTML 0.14% TypeScript 0.24% Makefile 0.03%

http.jl's People

Contributors

Stargazers

Watchers

Forkers

kleopatra999 bicycle1885 staticfloat tkelman zehcnas34 omus ararslan joshbode garborg samoconnor iblislin sarvghotra kristofferc ericforgy oxinabox invenia jsanter zgornel simondanisch amanica amellnik rened vtjnash pvazteixeira davidanthoff erikedin scls19fr vdayanand conning contextflow dtjohnson jakobetzel carpenecopinum introspection3 fredrikekre snthot xh4h stjordanis aybotalov chrisrackauckas houd1ni trickster tkoolen bcolloran kura-okubo timotaularson non-jedi guohf julia-works uganda-open-source-foundation tlienart acdupont nicoleepp wniels kcajf nosferican peteristhegreat srenatus quatrix mikolajhojda dietercastel w-zhiwei asinghvi17 3zn radhika3504 ciaranomara jademackay saurabh180 chris-b1 dut3062796s stevengj 5l1v3r1 aespar21 fonsp mauradriscoll astupidbear zhanglix oboukary quntion pfitzseb kernelmethod scrambldchannel mmiller-max pixel27 c-p-murphy fagan2888 roger-luo zeta1999 cmcaine jebej p1scescom sean-bennett112 jacquesdc racinmat ramboldio atsushisakai boostolive spirarel crghilardi strogo

http.jl's Issues

Migrating from Requests.jl to HTTP.jl, experience and suggestions...

Hi @quinnj,

I read (your announcement in Feb)[https://discourse.julialang.org/t/ann-new-http-jl-package/2048] with interest. My AWSCore.jl package relies on Requests.jl and I found the experience of getting it to work reliably was less than ideal, so the idea of a cleaner HTTP implementation appeals to me.

I've just finished an experimental port to HTTP.jl. On the whole, it looks like a big improvement. The new HTTP API pretty much just does what I want with no fuss. Compare my new much simpler http.jl to my old http.jl.

Note that the AWSCore.jl/src/http.jl layer does not disappear completely, it still adds the following features to HTTP.jl:

HTTPException type for non 20x result status.
Retry with exponential back-off for status < 200 or >= 500 and UVError.

Would you consider a PR to introduce a HTTPException type to HTTP.jl?

Would you consider a PR to add exponential back-off using Retry.jl ?

My other suggestion is to consider separating some of HTTP.jl's functionality into layers (seperate files and distinct APIs). HTTP.jl currently implements both stateless low-level HTTP wire-protocol stuff and high-level stateful stuff (e.g. Cookies, Authentication, Retries, Redirect processing). It's great to have all this functionality, but it would be cleaner to factor it out a little.

I suggest something like this:

Low level request/response layer:

Just send a request and read the response.
No default headers, just encode and send headers and body passed by the caller.

Middle retry layer:

Retry with exponential back-off for status < 200 or >= 500 and low level IO Exceptions.
Throw an exception for non 20x status.

High level stateful layer:

Useful default headers (User-Agent, Accept etc)
Authentication
Redirect handling (e.g. handle issues like authentication signatures changing on redirect).
Transfer-Encoding?
Cookies

A library like AWSCore.jl would use the middle layer directly (rather than using the high-level API with a bunch of options to disable default behaviour as it does now).

Too early query escaping?

I'm not perfectly sure, but query escaping looks too early to me. For example, the following effectively-equivalent URIs aren't equal in Julia:

julia> HTTP.URI("http://example.com?query=foo/bar")
HTTP.URI("http://example.com?query=foo/bar")

julia> HTTP.URI("http://example.com", query=Dict("query"=>"foo/bar"))
HTTP.URI("http://example.com?query=foo%2Fbar")

julia> HTTP.URI("http://example.com?query=foo/bar") == HTTP.URI("http://example.com", query=Dict("query"=>"foo/bar"))
false

file descriptor leak

I recommend setting a low ulimit first, so that this doesn't take a long time (ulimit -n 60)

julia> for i in 1:10000
         HTTP.get("google.com")
       end
ERROR (unhandled task failure): DNSError: google.com, unknown node or service (EAI_NONAME)
Stacktrace:
 [1] getaddrinfo(::String) at socket.jl:649
 [2] (::HTTP.##35#36{HTTP.http,SubString{String}})() at task.jl:335
ERROR: MethodError: Cannot `convert` an object of type Base.DNSError to an object of type HTTP.Connection
This may have arisen from a call to the constructor HTTP.Connection(...),
since type constructors fall back to convert methods.
Stacktrace:
 [1] HTTP.Connection(::Base.DNSError) at sysimg.jl:24
 [2] getconn(::Type{HTTP.http}, ::HTTP.Client{Base.TTY}, ::String, ::HTTP.RequestOptions, ::Bool) at /Users/jameson/.julia/v0.6/HTTP/src/client.jl:153
 [3] #request#32(::Array{HTTP.Response,1}, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at /Users/jameson/.julia/v0.6/HTTP/src/client.jl:109
 [4] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at <missing>:0
 [5] #request#30(::Dict{String,String}, ::HTTP.FIFOBuffer, ::Bool, ::Bool, ::Array{Any,1}, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at /Users/jameson/.julia/v0.6/HTTP/src/client.jl:93
 [6] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at <missing>:0
 [7] #get#43(::Bool, ::String, ::Array{Any,1}, ::Function, ::String) at /Users/jameson/.julia/v0.6/HTTP/src/client.jl:388
 [8] macro expansion; at REPL[5]:2 [inlined]
 [9] anonymous at <missing>:0
 [10] eval(::Module, ::Any) at ./boot.jl:235
 [11] eval_user_input(::Any, ::Base.REPL.REPLBackend) at REPL.jl:66
 [12] macro expansion; at REPL.jl:97 [inlined]
 [13] (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at event.jl:73

Even after gc(), the count of open descriptors does not decrease.

Not working with Julia 0.6

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0 (2017-06-19 13:05 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-apple-darwin13.4.0

julia> Pkg.checkout("HTTP")
INFO: Checking out HTTP master...
INFO: Pulling HTTP latest master...
INFO: No packages to install, update or remove

julia> using HTTP
INFO: Precompiling module HTTP.
ERROR: LoadError: LoadError: MethodError: Cannot `convert` an object of type Symbol to an object of type Val
This may have arisen from a call to the constructor Val(...),
since type constructors fall back to convert methods.
Stacktrace:
 [1] Val(::Symbol) at ./sysimg.jl:24
 [2] include_from_node1(::String) at ./loading.jl:569
 [3] include(::String) at ./sysimg.jl:14
 [4] include_from_node1(::String) at ./loading.jl:569
 [5] include(::String) at ./sysimg.jl:14
 [6] anonymous at ./<missing>:2
while loading /Users/sam/.julia/v0.6/HTTP/src/handlers.jl, in expression starting on line 55
while loading /Users/sam/.julia/v0.6/HTTP/src/HTTP.jl, in expression starting on line 37
ERROR: Failed to precompile HTTP to /Users/sam/.julia/lib/v0.6/HTTP.ji.
Stacktrace:
 [1] compilecache(::String) at ./loading.jl:703
 [2] _require(::Symbol) at ./loading.jl:490
 [3] require(::Symbol) at ./loading.jl:398

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @quinnj

Origin header breaks AWS EC2 API

See: 90061e7#commitcomment-23615586

HTTP.get() returns no content though Requests.get() does

This may not be an issue, maybe it's just a usage question. I'm not that experienced with HTTP requests, so apologies if I'm doing something obviously wrong here.

using HTTP

api_url = "https://www.bea.gov/api/data?"
api_key = "_my API key_"
api_method = "GetData"
api_dataset = "NIPA"

TableID = "5"
frequency = "Q"
years = "2014,2015"

querydict = Dict("UserID" => api_key,
                 "Method" => api_method,
                 "DatasetName" => api_dataset,
                 "TableID" => TableID,
                 "Frequency" => frequency,
                 "Year" => years,
                 "ResultFormat" => "JSON")

a = HTTP.get(api_url; query = querydict)

There appears to be no content returned:

HTTP.Response:
"""
HTTP/1.1 200 OK
Access-Control-Allow-Methods: GET, POST
Date: Sun, 05 Mar 2017 23:19:47 GMT
Age: 0
Access-Control-Allow-Origin: *
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Content-Length: 0
Access-Control-Max-Age: 60, must-revalidate
Strict-Transport-Security: max-age=31536000; includeSubDomains
Content-Type: text/html;charset=UTF-8
X-Frame-Options: SAMEORIGIN

"""

If I use Requests.jl for the identical query I get:

using Requests
b = Requests.get(api_url; query = querydict)
Requests.headers(b)

Dict{AbstractString,AbstractString} with 15 entries:
  "Access-Control-Allow-Methods" => "GET, POST"
  "Date"                         => "Sun, 05 Mar 2017 23:27:12 GMT"
  "Age"                          => "0"
  "Access-Control-Allow-Origin"  => "*"
  "http_minor"                   => "1"
  "Keep-Alive"                   => "1"
  "status_code"                  => "200"
  "Server"                       => "Microsoft-IIS/7.5"
  "X-Powered-By"                 => "ASP.NET"
  "Content-Length"               => "39247"
  "Access-Control-Max-Age"       => "60, must-revalidate"
  "http_major"                   => "1"
  "Strict-Transport-Security"    => "max-age=31536000; includeSubDomains"
  "Content-Type"                 => "application/json;charset=ISO-8859-1"
  "X-Frame-Options"              => "SAMEORIGIN"

This returns the data I anticipated which I can parse using Requests.json(). (Incidentally, this request throws an error - ERROR (unhandled task failure): HTTP Parser Exception: HPE_INVALID_CONSTANT(28):invalid constant string - though the data I'm looking for is still returned.)

I'd appreciate any help figuring out what's going wrong here.

Digest Authentication

Marking this as up for grabs if someone feels so inclined to implement. From what I can tell, it's not generally viewed as a great method of authentication anyway, and I've never personally had a use-case. My only worry w/ supporting it is where we'd get an MD5 hash function (w/o requiring some huge dependency or something).

Happy to hear other users experiences and such on this.

Early EOF with streaming

In certain situations using stream=true will always return EOF:

julia> using HTTP

julia> stream = HTTP.body(HTTP.get("https://julialang.s3.amazonaws.com/bin/linux/x64/0.5/julia-0.5.1-linux-x86_64.tar.gz.asc", stream=true))
HTTP.FIFOBuffer(801,1048576,801,1,802,UInt8[0x2d,0x2d,0x2d,0x2d,0x2d,0x42,0x45,0x47,0x49,0x4e  …  0x54,0x55,0x52,0x45,0x2d,0x2d,0x2d,0x2d,0x2d,0x0a],Condition(Any[]),Task (done) @0x0000000114cd61d0,true)

julia> eof(stream)
true

julia> nb_available(stream)
801

julia> read(stream, 1)
1-element Array{UInt8,1}:
 0x2d

julia> eof(stream)
true

Convert POST request body to Dict

Thanks for the great library!

Is it possible to convert the body of a Request to a dictionary when responding to a POST request?

using HTTP
function handle_post(req::Request, rep::Response)
   String(take!(req)) # convert to Dict here
   return rep::Response
end

server = HTTP.Server(handle_post)
tsk = @async HTTP.serve(server, ip"0.0.0.0", 5000, verbose=true)

sleep(2)
r = HTTP.post("http://0.0.0.0:5000", body=Dict("amount"=> 1000))

Connecting to local network: RetryException

I can access a device on my home network using Requests.jl, but not HTTP.jl. Possibly a configuration issue?

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.1 (2017-03-05 13:25 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

julia> using HTTP, Requests

julia> Requests.get("http://192.168.0.100")
Response(200 OK, 5 headers, 134 bytes in body)

julia> Requests.get("http://192.168.0.100:80")
Response(200 OK, 5 headers, 134 bytes in body)

julia> HTTP.get("http://192.168.0.100")
ERROR: RetryException: # of allowed retries (3) was exceeded when making request
 in request(::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions, ::HTTP.Connection{TCPSocket}, ::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:224
 in #request#34(::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:114
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at .\<missing>:0
 in request(::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions, ::HTTP.Connection{TCPSocket}, ::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:225
 in #request#34(::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:114
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at .\<missing>:0
 in request(::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions, ::HTTP.Connection{TCPSocket}, ::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:225
 in #request#34(::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:114
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at .\<missing>:0
 in #request#32(::Dict{String,String}, ::HTTP.FIFOBuffer, ::Bool, ::Bool, ::Array{Any,1}, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:98
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at .\<missing>:0
 in #get#51(::Bool, ::String, ::Array{Any,1}, ::Function, ::String) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:424
 in get(::String) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:424

julia> HTTP.get("http://192.168.0.100:80")
ERROR: RetryException: # of allowed retries (3) was exceeded when making request
 in request(::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions, ::HTTP.Connection{TCPSocket}, ::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:224
 in #request#34(::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:114
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at .\<missing>:0
 in request(::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions, ::HTTP.Connection{TCPSocket}, ::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:225
 in #request#34(::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:114
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at .\<missing>:0
 in request(::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions, ::HTTP.Connection{TCPSocket}, ::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:225
 in #request#34(::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:114
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at .\<missing>:0
 in #request#32(::Dict{String,String}, ::HTTP.FIFOBuffer, ::Bool, ::Bool, ::Array{Any,1}, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:98
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at .\<missing>:0
 in #get#51(::Bool, ::String, ::Array{Any,1}, ::Function, ::String) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:424
 in get(::String) at C:\Users\Evan\.julia\v0.5\HTTP\src\client.jl:424

ArgumentError("stream is closed or unusable")

See 527618c#commitcomment-23614516

missing body content?

It seems like all content gets lost if there's more than a few dozen seconds since the last request. (on Julia master and current HTTP.jl release 15f33bc)

julia> r = HTTP.get(uriroot, headers = headers)
HTTP.Response:
"""
HTTP/1.1 200 OK
Connection: keep-alive
Via: 1.1 vegur
...
"""

<wait for a minute>

julia> r = HTTP.get(uriroot, headers = headers)
HTTP.Response:
"""
HTTP/1.1 200 OK

"""

julia> r = HTTP.get(uriroot, headers = headers)
HTTP.Response:
"""
HTTP/1.1 200 OK
Connection: keep-alive
Via: 1.1 vegur
...

Websocket integration

It'd be nice to support websocket upgrade requests in an HTTP.Server, as well as have websocket client functionality as well.

cc: @sarvjeethghotra

HTTP.Client functionality

~~proxy stuff~~ moved to #218
multi-part encoded files
~~digest authentication~~: moved to #37
~~auto-gunzip response bodys w/ gzip transfer-encoding; auto-gzip chunked transfers~~ moved to #217

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @quinnj

FIFOBuffer does not follow `read(::Any, ::Type{UInt8})` standard

I'm trying to use readbytes! with FIFOBuffer but I'm encountering errors since read(::FIFOBuffer, ::Type{UInt8}) returns a Tuple{UInt8,Bool} rather than the typical UInt8.

julia> using HTTP

julia> f = HTTP.FIFOBuffer(5);

julia> b = Array{UInt8}(3);

julia> write(f, [0x01, 0x02, 0x03, 0x04, 0x05])
5

julia> readbytes!(f, b)
ERROR: MethodError: Cannot `convert` an object of type Tuple{UInt8,Bool} to an object of type UInt8
This may have arisen from a call to the constructor UInt8(...),
since type constructors fall back to convert methods.
 in readbytes!(::HTTP.FIFOBuffer, ::Array{UInt8,1}, ::Int64) at ./io.jl:351
 in readbytes!(::HTTP.FIFOBuffer, ::Array{UInt8,1}) at ./io.jl:342

julia> io = IOBuffer([0x01, 0x02, 0x03, 0x04, 0x05])
IOBuffer(data=UInt8[...], readable=true, writable=false, seekable=true, append=false, size=5, maxsize=Inf, ptr=1, mark=-1)

julia> readbytes!(io, b)
3

Could you use a different function name for this behaviour if it is required?

Failed to precompile HTTP

ERROR: LoadError: UndefVarError: eval_comparison not defined
Stacktrace:
 [1] _precompile_1() at /Users/300006808/.julia/v0.7/HTTP/src/precompile/precompile_Base.jl:140
 [2] include_from_node1(::Module, ::String) at ./loading.jl:556
 [3] include(::Module, ::String) at ./sysimg.jl:14
 [4] anonymous at ./<missing>:2
while loading /Users/300006808/.julia/v0.7/HTTP/src/HTTP.jl, in expression starting on line 35
ERROR: Failed to precompile HTTP to /Users/300006808/.julia/lib/v0.7/HTTP.ji.
Stacktrace:
 [1] compilecache(::String) at ./loading.jl:696
 [2] _require(::Symbol) at ./loading.jl:495
 [3] require(::Symbol) at ./loading.jl:404

Version 0.7.0-DEV.950 (2017-07-12 21:47 UTC)
Commit 7c31c41b5 (5 days old master)
x86_64-apple-darwin16.6.0

Can someone please help to resolve the issue ?

Starting basic server creates error in parser.jl

The following code:

using HTTP

import HTTP: http, serve

server = HTTP.Server{http}(
	ip"127.0.0.1",
	8080,
	(request, response) -> println("$request,\t$response"),
	STDOUT
	)

serve(server)

results in the following error:

Starting server to listen on: 127.0.0.1:8080
ERROR: LoadError: MethodError: no method matching HTTP.Parser{R}(::Type{HTTP.Request}, ::Type{HTTP.http})
Closest candidates are:
HTTP.Parser{R}{T}(::Type{T}) at /home/joel/.julia/v0.5/HTTP/src/parser.jl:33
HTTP.Parser{R}{T}(::Any) at sysimg.jl:53
in HTTP.ServerClient{T<:HTTP.Scheme,I<:IO}(::HTTP.Server{HTTP.http}) at /home/joel/.julia/v0.5/HTTP/src/server.jl:111
in serve(::HTTP.Server{HTTP.http}) at /home/joel/.julia/v0.5/HTTP/src/server.jl:188
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /home/joel/Development/Julia/Vocabulary/vocabulary_website.jl, in expression starting on line 9

I'm quite fresh to Julia and not sure whether I'm doing something wrong or this is a bug.

Thoughts of defining `take!(::Response)`?

I wonder if it would be a good idea to essentially implement the IOBuffer API from Julia .6 for Response objects.

Thorough review of client-side error-handling

The current error-handling has gradually iterated to the current state, but it's gotten pretty messy. I'd like to do a review, organization, and consolidation of the various types of errors we can encounter when performing requests, the various actions we can take for various error types, and also do a thorough review of resource cleanup to ensure we're being as tidy as possible. (i.e. I don't think we're closing connections on more serious errors yet).

Use a Nullable for client/server loggers and then use STDOUT by default at each request level

Server response streaming early close

I create a Server with a FIFOBuffer Response body.
I have an async task that writes data to the FIFOBuffer once per second forever.
Rather than the client receiving an infinite stream of data, it only gets the first line, after which the server closes the connection.

It looks like maybe Base.write(io::IO, r::Union{Request, Response} ...) should be doing while !eof(response.body) write(io, readavailable(response.body) end ?

Also, a content-length header is being returned even though the content is unknown (because the response FIFOBuffer has not been closed yet).

julia> server = HTTP.Server((req, rep) -> begin
           io = HTTP.FIFOBuffer()
           @async while true
               println(io, "data: $(now())\n")
               sleep(1)
           end
           r = Response(200, Dict(
           "Content-Type" => "text/event-stream",
           "Cache-Control" => "no-cache",
           "Connection" => "keep-alive"
       ), io)
       end, STDOUT)

julia> HTTP.serve(server)

$ curl 127.0.0.1:8081
data: 2017-08-22T22:00:04.988

$ curl 127.0.0.1:8081
data: 2017-08-22T22:00:05.635

Startup is much slower than e.g. Requests.jl

I feel like we must be exercising a part of the compiler that doesn't normally get much exercise in this package, because startup times are particularly bad (this is on Julia 0.5):

julia> @time using Requests # (for reference)
  0.358254 seconds (329.11 k allocations: 15.328 MB)

julia> @time using HTTP
  4.433749 seconds (13.34 M allocations: 480.260 MB, 3.84% gc time)

Could this be another case where there's just a bunch of code in Base that isn't precompiled, and thus we need to mess around with base/precompile.jl in order to get this to speed up at all?

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @quinnj

Content-Type charset ISO-8859-1

For example,

julia> take!(String, HTTP.get("http://google.com"))
[ truncated either error while showing string (0.5) or string of invalid UTF-8 data (0.6) ]

julia> using StringEncodings

julia> decode(take!(HTTP.get("http://google.com")), enc"ISO-8859-1")
[ truncated correct source code of Google.com ]

Because the HTTP header contains

Content-Type: text/html; charset=ISO-8859-1

it should be possible to detect the correct encoding and decode it automatically.

UnicodeError: invalid character index when sending image

I just tried to respond to a request for an image and it failed with this error.
The same seems to happen when I build a response with a FIFOBuffer, which is filled from an image file in the REPL.
Note: I used the default headers, but I guess the response writing code doesn't care?

Error showing value of type HTTP.Response:
ERROR: UnicodeError: invalid character index
in getindex(::String, ::UnitRange{Int64}) at ./strings/string.jl:130
in show(::IOContext{Base.Terminals.TTYTerminal}, ::HTTP.Response, ::HTTP.RequestOptions) at /home/joel/.julia/v0.5/HTTP/src/types.jl:400
in display(::Base.REPL.REPLDisplay{Base.REPL.LineEditREPL}, ::MIME{Symbol("text/plain")}, ::HTTP.Response) at ./REPL.jl:132
in display(::Base.REPL.REPLDisplay{Base.REPL.LineEditREPL}, ::HTTP.Response) at ./REPL.jl:135
in display(::HTTP.Response) at ./multimedia.jl:143
in print_response(::Base.Terminals.TTYTerminal, ::Any, ::Void, ::Bool, ::Bool, ::Void) at ./REPL.jl:154
in print_response(::Base.REPL.LineEditREPL, ::Any, ::Void, ::Bool, ::Bool) at ./REPL.jl:139
in (::Base.REPL.##22#23{Bool,Base.REPL.##33#42{Base.REPL.LineEditREPL,Base.REPL.REPLHistoryProvider},Base.REPL.LineEditREPL,Base.LineEdit.Prompt})(::Base.LineEdit.MIState, ::Base.AbstractIOBuffer{Array{UInt8,1}}, ::Bool) at ./REPL.jl:652
in run_interface(::Base.Terminals.TTYTerminal, ::Base.LineEdit.ModalInterface) at ./LineEdit.jl:1579
in run_frontend(::Base.REPL.LineEditREPL, ::Base.REPL.REPLBackendRef) at ./REPL.jl:903
in run_repl(::Base.REPL.LineEditREPL, ::Base.##930#931) at ./REPL.jl:188
in _start() at ./client.jl:360

Can't read google.com

I removed the redirect to avoid confusion; for me Google redirects to http://www.google.ca/?gfe_rd=cr&ei=jJL_WPD_Oo7LyAGC2I_YAw

julia> using HTTP

julia> resp = HTTP.get("http://www.google.ca/?gfe_rd=cr&ei=jJL_WPD_Oo7LyAGC2I_YAw")
HTTP.Response:
"""
HTTP/1.1 200 OK
Date: Tue, 25 Apr 2017 18:39:42 GMT
Accept-Ranges: none
Transfer-Encoding: chunked
Cache-Control: private, max-age=0
Server: gws
Expires: -1
X-XSS-Protection: 1; mode=block
Content-Length: 10957
Vary: Accept-Encoding
P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
Set-Cookie: NID=102=FDjBnl9G8u4Ke99dnUBGlsS1HoIAvr3XKdl0SWvoJFr-OUVrlPJF5nIwBhEdRn7AS5GkxHmSA4IDc_zPO4ygoEoWbo1nFP3AcoKDWDcXnf5-xhQuzxihQQtt3Zgf_Pqi; expires=Wed, 25-Oct-2017 18:39:42 GMT; path=/; domain=.google.ca; HttpOnly
Content-Type: text/html; charset=ISO-8859-1
X-Frame-Options: SAMEORIGIN

[HTTP.Response body of 10959 bytes]

<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="en-CA"><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><title>Google</title><script>(function(){window.google={kEI:'7pf_WOrYDefgjwTGzr3QDw',kEXPI:'201761,750722,1351176,1352864,1352994,1353097,3700308,3700347,3700405,4029815,4031109,4032678,4036527,4039268,4043492,4045841,4048347,4065787,4071842,4072364,4072777,4075963,4076096,4076999,4078430,4078760,4081039,4081165,4083458,4084180,4086696,4090550,4090553,4090806,4090877,4092182,4092934,4093499,4093550,4093808,4094251,4094542,4094837,4095910,4095999,4096323,4096438,4097153,4097470,4097653,4097922
⋮
"""

julia> HTTP.body(resp)
HTTP.FIFOBuffer(10957,4611686014132420609,10957,1,10958,UInt8[0x3c,0x21,0x64,0x6f,0x63,0x74,0x79,0x70,0x65,0x20  …  0x64,0x79,0x3e,0x3c,0x2f,0x68,0x74,0x6d,0x6c,0x3e],Condition(Any[]),Task (runnable) @0x0000000119194fd0,true)

julia> bytes = readbytes(ans)
WARNING: readbytes is deprecated, use read instead.
 in depwarn(::String, ::Symbol) at deprecated.jl:64
 in readbytes(::HTTP.FIFOBuffer, ::Vararg{HTTP.FIFOBuffer,N}) at deprecated.jl:30
 in eval(::Module, ::Any) at boot.jl:234
 in eval(::Module, ::Any) at sys.dylib:?
 in eval_user_input(::Any, ::Base.REPL.REPLBackend) at REPL.jl:64
 in macro expansion at REPL.jl:95 [inlined]
 in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at event.jl:68
while loading no file, in expression starting on line 0
10957-element Array{UInt8,1}:
 0x3c
 0x21
 0x64
 0x6f
 0x63
 0x74
 0x79
 0x70
 0x65
 0x20
 0x68
 0x74
 0x6d
 0x6c
 0x3e
 0x3c
 0x68
 0x74
 0x6d
 0x6c
 0x20
 0x69
 0x74
 0x65
 0x6d
 0x73
 0x63
 0x6f
 0x70
 0x65
 0x3d
 0x22
 0x22
 0x20
 0x69
 0x74
 0x65
 0x6d
 0x74
 0x79
 0x70
 0x65
 0x3d
    ⋮
 0x67
 0x6c
 0x65
 0x2e
 0x6a
 0x2e
 0x78
 0x69
 0x2c
 0x30
 0x29
 0x3b
 0x7d
 0x0a
 0x3c
 0x2f
 0x73
 0x63
 0x72
 0x69
 0x70
 0x74
 0x3e
 0x3c
 0x2f
 0x64
 0x69
 0x76
 0x3e
 0x3c
 0x2f
 0x62
 0x6f
 0x64
 0x79
 0x3e
 0x3c
 0x2f
 0x68
 0x74
 0x6d
 0x6c
 0x3e

julia> String(ans)
10957-byte String of invalid UTF-8 data:
 0x3c
 0x21
 0x64
 0x6f
 0x63
 0x74
 0x79
 0x70
 0x65
 0x20
 0x68
 0x74
 0x6d
 0x6c
 0x3e
 0x3c
 0x68
 0x74
 0x6d
 0x6c
 0x20
 0x69
 0x74
 0x65
 0x6d
 0x73
 0x63
 0x6f
 0x70
 0x65
 0x3d
 0x22
 0x22
 0x20
 0x69
 0x74
 0x65
 0x6d
 0x74
 0x79
 0x70
 0x65
 0x3d
    ⋮
 0x67
 0x6c
 0x65
 0x2e
 0x6a
 0x2e
 0x78
 0x69
 0x2c
 0x30
 0x29
 0x3b
 0x7d
 0x0a
 0x3c
 0x2f
 0x73
 0x63
 0x72
 0x69
 0x70
 0x74
 0x3e
 0x3c
 0x2f
 0x64
 0x69
 0x76
 0x3e
 0x3c
 0x2f
 0x62
 0x6f
 0x64
 0x79
 0x3e
 0x3c
 0x2f
 0x68
 0x74
 0x6d
 0x6c
 0x3e

`Origin` header contains path

The Origin header seems to contain the full path, which causes some CORS-checking systems to choke.

julia> HTTP.request(client, "GET", HTTP.URI("https://buildog.julialang.org/api/v2/forceschedulers"); verbose=true)
[HTTP - 2017-05-18T19:33:00.093]: using request options: :chunksize=>1048576, :gzip=>true, :connecttimeout=>15.0, :readtimeout=>15.0, :
tlsconfig=>MbedTLS.SSLConfig(), :maxredirects=>5, :allowredirects=>true, :forwardheaders=>false, :retries=>3
[HTTP - 2017-05-18T19:33:00.368]: checking if any existing connections to 'buildog.julialang.org' are re-usable
[HTTP - 2017-05-18T19:33:00.368]: found re-usable connection #1
[HTTP - 2017-05-18T19:33:00.406]: sending request over the wire
HTTP.Request:
"""
GET /api/v2/forceschedulers HTTP/1.1
Host: buildog.julialang.org
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8,application/json
Origin: https://buildog.julialang.org/api/v2/forceschedulers
User-Agent: HTTP.jl/0.0.0

"""

However, according to Mozilla's docs:

The Origin request header indicates where a fetch originates from. It doesn't include any path information, but only the server name. It is sent with CORS requests, as well as with POST requests. It is similar to the Referer header, but, unlike this header, it doesn't disclose the whole path.

Will commonly used headers get special support?

I'm talking about headers like Accept, where we get stuff like this:

"text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8"

And I was thinking about splitting it up and sorting by "quality" myself, but I wanted to ask whether it is planned as part of this package anyway.

TimeoutException when HTTP.jl is loaded from a file.

When HTTP.jl is loaded from a file using -L option, HTTP.get fails with timeout exception:

~/.j/v/HTTP (master|…) $ cat test.jl
import HTTP
~/.j/v/HTTP (master|…) $ julia -L test.jl
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |  x86_64-apple-darwin14.5.0

julia> HTTP.get("http://httpbin.org/")
ERROR: TimeoutException: server did not respond for more than 10.0 seconds.
 in process!(::HTTP.Client{Base.TTY}, ::HTTP.Connection{TCPSocket}, ::HTTP.RequestOptions, ::String, ::HTTP.Method, ::HTTP.Response, ::Base.RefValue{Float64}, ::Bool, ::Bool) at /Users/kenta/.julia/v0.5/HTTP/src/client.jl:280
 in request(::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions, ::HTTP.Connection{TCPSocket}, ::Array{HTTP.Response,1}, ::Bool, ::Bool) at /Users/kenta/.julia/v0.5/HTTP/src/client.jl:212
 in #request#32(::Array{HTTP.Response,1}, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at /Users/kenta/.julia/v0.5/HTTP/src/client.jl:109
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at ./<missing>:0
 in #request#30(::Dict{String,String}, ::HTTP.FIFOBuffer, ::Bool, ::Bool, ::Array{Any,1}, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at /Users/kenta/.julia/v0.5/HTTP/src/client.jl:93
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at ./<missing>:0
 in #get#49(::Bool, ::String, ::Array{Any,1}, ::Function, ::String) at /Users/kenta/.julia/v0.5/HTTP/src/client.jl:385
 in get(::String) at /Users/kenta/.julia/v0.5/HTTP/src/client.jl:385

julia> ERROR (unhandled task failure): readcb: connection reset by peer (ECONNRESET)
 in yieldto(::Task, ::ANY) at ./event.jl:136
 in yieldto(::Task, ::ANY) at /usr/local/julia/v0.5/lib/julia/sys.dylib:?
 in wait() at ./event.jl:169
 in wait(::Condition) at ./event.jl:27
 in wait_readnb(::TCPSocket, ::Int64) at ./stream.jl:303
 in readavailable(::TCPSocket) at ./stream.jl:791
 in macro expansion at /Users/kenta/.julia/v0.5/HTTP/src/client.jl:247 [inlined]
 in (::HTTP.##42#46{HTTP.Client{Base.TTY},HTTP.Connection{TCPSocket},HTTP.RequestOptions,String,HTTP.Method,HTTP.Response,Base.RefValue{Float64},Bool,Bool,HTTP.Parser})() at ./task.jl:360
julia>

This is not reproducible when HTTP.jl is loaded in REPL:

~/.j/v/HTTP (master|…) $ julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |  x86_64-apple-darwin14.5.0

julia> import HTTP

julia> HTTP.get("http://httpbin.org/")
HTTP.Response:
"""
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 12150
Access-Control-Allow-Credentials: true
Date: Mon, 13 Feb 2017 18:10:49 GMT
Content-Type: text/html; charset=utf-8
Access-Control-Allow-Origin: *
Server: nginx

Currently not playing nice w/ gzip request bodies

Because of the auto chunking involved, gzipped request bodies get broken up in a way the remote server can't put back together.

12 "Transfer-Encoding: chunked" Tests Fail Due to httpbin issue

It looks like there is an unresolved httpbin issue / feature regression with chunked POST requests which is causing 10 of the 12 current test failures. Can anyone confirm this?

splitpath loses last character

I just found out you're trying to revive Julia's webstack. Thank you for the effort. ;)

I guess I'll start helping out by finding bugs, since I seem to have a talent for that. It looks like splitpath fails to capture the last path character:

julia> uri = HTTP.URI("http://www.example.com/blah/blub?q=blub")
HTTP.URI("http://www.example.com/blah/blub?q=blub")

julia> HTTP.splitpath(uri)
2-element Array{String,1}:
"blah"
"blu"

FIFOBuffer eof() true when close() not called

I create a fifo buffer and start an async task to write to it every 1 second forever.
Then i try to read while !eof(), but eof() returns true after the first read (and i have not called close()).

julia> using HTTP

julia> io = HTTP.FIFOBuffers.FIFOBuffer()
HTTP.FIFOBuffers.FIFOBuffer(0, 4611686014132420609, 0, 1, 1, -1, 0, UInt8[], Condition(Any[]), Task (runnable) @0x00000001178c1450, false)

julia> @async while true
           println(io, now())
           sleep(1)
       end
Task (runnable) @0x00000001178c3850

julia> while !eof(io)
           println(String(readavailable(io)))
           @show eof(io)
       end
2017-08-22T21:53:39.083
2017-08-22T21:53:40.249
2017-08-22T21:53:41.256
2017-08-22T21:53:42.262

eof(io) = true

Playing nicely with Windows Auth?

Does anyone know roughly what it would take to enable this? It looks like there's similar python libraries that support this, looking into them now.

non-string entries in `query` dict don't escape properly

Observe:

julia> params = Dict("foo" => 73)
Dict{String,Int64} with 1 entry:
  "foo" => 73

julia> HTTP.URI("http://bar.com/baz", query=params)
HTTP.ParsingError: encountered invalid url character for parsing state = es_req_query_string:
http://bar.com/baz?foo=HTTP.URI(%22$(jlbuild.buildbot_base)%2Fapi%2Fv2%2Fbuilds%22%3B%20query%3Dparams)%00%BC%FB%9E%8C%00%00%01
----------------------------------------------------------------------------------------------------------------------^
ERROR:
 in http_parser_parse_url(::Array{UInt8,1}, ::Int64, ::Int64, ::Bool) at /home/sabae/.julia/v0.5/HTTP/src/uri.jl:366
 in (::Base.#kw##parse)(::Array{Any,1}, ::Base.#parse, ::Type{HTTP.URI}, ::String) at ./<missing>:0
 in #URI#12(::String, ::String, ::Dict{String,Int64}, ::String, ::Bool, ::Type{T}, ::String) at /home/sabae/.julia/v0.5/HTTP/src/uri.jl:74
 in (::Core.#kw#Type)(::Array{Any,1}, ::Type{HTTP.URI}, ::String) at ./<missing>:0

That jlbuild.buildbot_base stuff is it reading off the end of a buffer and pulling in previously allocated memory. Exciting! But not exactly what we want. This works if I turn the values of the Dict into a string:

julia> params = Dict("foo" => "73")
Dict{String,String} with 1 entry:
  "foo" => "73"

julia> HTTP.URI("http://bar.com/baz", query=params)
HTTP.URI("http://bar.com/baz?foo=73")

Error when testing with ab

Hi there,

I have a simple server that works when making a request from Julia, but fails when making a request from the ab tool. Minimal example below. Any ideas?

using HTTP

function app(req, res)
    res.body = FIFOBuffer("Hello world!")
    res
end

server = HTTP.Server(app)
HTTP.serve(server, IPv4(127, 0, 0, 1), 8000)

This works:

using HTTP
res = HTTP.get("http://127.0.0.1:8000/")

But this fails:

ab -n 1 "http://127.0.0.1:8000/"

By contrast the following server works in both cases:

using HttpServer

function app(req, res)
    res.data = "Hello world!"
    res
end

server = Server(app)
run(server, 8000)

Get fails with default HTTP.Client

Works with explicit HTTP.Client argument:

julia> HTTP.get(HTTP.Client(STDOUT), "www.google.com")
HTTP.Response:
"""
HTTP/1.1 200 OK

Not without:

julia> HTTP.get("www.google.com")
ERROR: MethodError: no method matching write(::Void, ::String)
Closest candidates are:
  write(::HTTP.FIFOBuffer, ::String) at /Users/sam/.julia/v0.6/HTTP/src/fifobuffer.jl:349
  write(::IO, ::String) at strings/string.jl:71
  write(::AbstractString, ::Any...) at io.jl:152
  ...
Stacktrace:
 [1] #request#34(::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool, ::Function, ::HTTP.Client, ::HTTP.Request, ::HTTP.RequestOptions) at /Users/sam/.julia/v0.6/HTTP/src/client.jl:118
 [2] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client, ::HTTP.Request, ::HTTP.RequestOptions) at ./<missing>:0
 [3] #request#32(::Dict{String,String}, ::HTTP.FIFOBuffer, ::Bool, ::Bool, ::Array{Any,1}, ::Function, ::HTTP.Client, ::HTTP.Method, ::HTTP.URI) at /Users/sam/.julia/v0.6/HTTP/src/client.jl:105
 [4] (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client, ::HTTP.Method, ::HTTP.URI) at ./<missing>:0
 [5] #get#49(::Bool, ::String, ::Array{Any,1}, ::Function, ::String) at /Users/sam/.julia/v0.6/HTTP/src/client.jl:445
 [6] get(::String) at /Users/sam/.julia/v0.6/HTTP/src/client.jl:445

Weird gzip-related errors

@staticfloat @bicycle1885

I'm seeing some weird gzip-related errors. If I do a request that returns a compressed body and save that out to disk, everything works fine and dandy on OSX, but on linux, I always get

[jacob.quinn]$ gunzip file.gz

gzip: file.gz: invalid compressed data--crc error

gzip: file.gz: invalid compressed data--length error

Would you two (and anyone else) be able to try a similar request and see if you run into any issues?

Cookies aren't overwriting like they should

I'm sorry I don't have an MWE for you, I've been unable to reproduce this with a small amount of code.

I have a buildbot instance that I'm authenticating to and that authentication modifies a cookie. This cookie is called TWISTED_SESSION and is a giant encoded mess, but the important part is that the cookie must be overwritten properly in order for the authentication to stick. I've noticed that my authentication seems to randomly come and go; sometimes I can make dozens of requests with no problem, and sometimes it takes me 5 or 6 authentication requests for the cookie to be included in further requests that I make against the buildbot. My bandaid solution until I could get a good MWE so far has been to just try authenticating again and again until it works, (which has the happy accident of making a lot of my code very resilient! yay!) but I think I just found something of interest.

Looking into my client object, I see that there are many TWISTED_SESSION cookies from my buildbot:

8-element Array{HTTP.Cookies.Cookie,1}:
 HTTP.Cookies.Cookie("TWISTED_SESSION","eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2luZm8iOnsidXNlcm5hbWUiOiJqbGJ1aWxkIiwiZW1haWwiOiJqbGJ1aWxkIiwiZ3JvdXBzIjpbIkp1bGlhTGFuZyJdLCJmdWxsX25hbWUiOiJKdWxpYSBCdWlsZGJvdCJ9LCJleHAiOjE0ODg0Mjc1NTZ9.Y536-llbe8ox3dgvwXeD5US5cbRdbSTbqLoUhzq4N_s","/","buildtest.e.ip.saba.us",0001-01-01T00:00:00,0,false,false,true,String[])
 HTTP.Cookies.Cookie("TWISTED_SESSION","eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2luZm8iOnsidXNlcm5hbWUiOiJqbGJ1aWxkIiwiZW1haWwiOiJqbGJ1aWxkIiwiZ3JvdXBzIjpbIkp1bGlhTGFuZyJdLCJmdWxsX25hbWUiOiJKdWxpYSBCdWlsZGJvdCJ9LCJleHAiOjE0ODg0Mjc1NzN9.OfTFv2QJHxtQkVQEKRZg3P2bkhAiGmsRoVQI775U3pE","/","buildtest.e.ip.saba.us",0001-01-01T00:00:00,0,false,false,true,String[])
 HTTP.Cookies.Cookie("TWISTED_SESSION","eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2luZm8iOnsidXNlcm5hbWUiOiJqbGJ1aWxkIiwiZW1haWwiOiJqbGJ1aWxkIiwiZ3JvdXBzIjpbIkp1bGlhTGFuZyJdLCJmdWxsX25hbWUiOiJKdWxpYSBCdWlsZGJvdCJ9LCJleHAiOjE0ODg0Mjc1NTd9.ClKMBSCO7V_GlTHtOUH8fz9EI0p5MmnNBrDDeq4Qj9Q","/","buildtest.e.ip.saba.us",0001-01-01T00:00:00,0,false,false,true,String[])
 HTTP.Cookies.Cookie("TWISTED_SESSION","eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2luZm8iOnsidXNlcm5hbWUiOiJqbGJ1aWxkIiwiZW1haWwiOiJqbGJ1aWxkIiwiZ3JvdXBzIjpbIkp1bGlhTGFuZyJdLCJmdWxsX25hbWUiOiJKdWxpYSBCdWlsZGJvdCJ9LCJleHAiOjE0ODg0Mjc1NTh9.Qc6t2ykLerNIeORlJYqfI67NrgfHv0q8-k70TaVRXvE","/","buildtest.e.ip.saba.us",0001-01-01T00:00:00,0,false,false,true,String[])
 HTTP.Cookies.Cookie("TWISTED_SESSION","eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2luZm8iOnsidXNlcm5hbWUiOiJqbGJ1aWxkIiwiZW1haWwiOiJqbGJ1aWxkIiwiZ3JvdXBzIjpbIkp1bGlhTGFuZyJdLCJmdWxsX25hbWUiOiJKdWxpYSBCdWlsZGJvdCJ9LCJleHAiOjE0ODg0Mjc1NzR9.wK71WMj1HwVvnjzR5ho9uiua6b60oQLB0khNwIjhRYc","/","buildtest.e.ip.saba.us",0001-01-01T00:00:00,0,false,false,true,String[])
 HTTP.Cookies.Cookie("TWISTED_SESSION","eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2luZm8iOnsidXNlcm5hbWUiOiJqbGJ1aWxkIiwiZW1haWwiOiJqbGJ1aWxkIiwiZ3JvdXBzIjpbIkp1bGlhTGFuZyJdLCJmdWxsX25hbWUiOiJKdWxpYSBCdWlsZGJvdCJ9LCJleHAiOjE0ODg0Mjc1NTl9.jXJ3c-JjwBoZxV4YHdCx8l9YnAqcPTtUvGnUDnEC5e4","/","buildtest.e.ip.saba.us",0001-01-01T00:00:00,0,false,false,true,String[])
 HTTP.Cookies.Cookie("TWISTED_SESSION","eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2luZm8iOnsidXNlcm5hbWUiOiJqbGJ1aWxkIiwiZW1haWwiOiJqbGJ1aWxkIiwiZ3JvdXBzIjpbIkp1bGlhTGFuZyJdLCJmdWxsX25hbWUiOiJKdWxpYSBCdWlsZGJvdCJ9LCJleHAiOjE0ODg0Mjc1NzJ9.DdN7E1_Gg33lzdgYh7jb-zm4S59JouSBg9hTiix-RWk","/","buildtest.e.ip.saba.us",0001-01-01T00:00:00,0,false,false,true,String[])
 HTTP.Cookies.Cookie("TWISTED_SESSION","eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyX2luZm8iOnsiYW5vbnltb3VzIjp0cnVlfSwiZXhwIjoxNDg4NDI3NTUxfQ.hfPJGiuK1iJ2eH7Ueyc2uVvIVRw5p26q1v8FiGzjhtA","/","buildtest.e.ip.saba.us",0001-01-01T00:00:00,0,false,false,true,String[])

If all of these cookies are being sent to the server, I would not be surprised at all if the intermittent nature of my failures are due to Julia's Set key traversal pseudo-randomizing the order in which the cookies are sent to the server, thereby causing the "correct" cookie to sometimes be last/first/whatever position would cause the buildbot to use that cookie instead of any of the others.

Chrome/Postman/etc... do not get multiple cookies, they get only one for the same cookie name/subpath/host tuple, so I'm certain enough that this is a bug in HTTP.jl that I'll open this issue, but not certain enough that I'm willing to stake my reputation as an internet cookie connoisseur on it.

Moving from Requests.jl to HTTP.jl

Just wondering whether it's easy (or advised) to move from using Requests.jl to HTTP.jl. I thought it might be possible to simply swap, but now I'm not so sure:

Here's Requests, followed by HTTP:

julia> response = Requests.get("http://192.168.1.3/api/lSsXQfrm7rC32SQ0/lights/")
Response(200 OK, 14 headers, 1488 bytes in body)

julia> response = HTTP.get("http://192.168.1.3/api/lSsXQfrm7rC32SQ0/lights/")
HTTP.Response:
"""
HTTP/1.1 200 OK
Connection: close
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE, HEAD
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-type: application/json
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 1 Aug 2011 09:00:00 GMT
Content-Length: 0
Pragma: no-cache
Access-Control-Max-Age: 3600
Access-Control-Allow-Headers: Content-Type
"""

Multi-core server

Might be tricky, might be an easier way:

need to wrap/export http://docs.libuv.org/en/v1.x/stream.html#c.uv_write2 in Base julia
use above send a TCPSocket handle between processes using a pipe stream
"recreate" the TCPSocket on the slave process using the received handle

URI

Is there any reason not to use URIParser for the representation of URI's instead of AbstractString?

Strange failure parsing response

I am attempting to migrate to HTTP.jl from Requests.jl.
One of my requests is failing, and switching on debug, I see this:

[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:195]: top of main for-loop
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:196]: Base.escape_string(string(ch)) = \\r
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:967]: ParsingStateCode(p_state) = es_header_value_lws
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:588]: ParsingStateCode(p_state) = es_header_field_start
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:195]: top of main for-loop
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:196]: Base.escape_string(string(ch)) = \\n
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1017]: ParsingStateCode(p_state) = es_headers_almost_done
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1032]: checking for upgrade...
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1035]: upgrade = false
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:98]: onheaderscomplete
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1054]: ParsingStateCode(p_state) = es_headers_done
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1103]: ParsingStateCode(p_state) = es_body_identity_eof
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:195]: top of main for-loop
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:196]: Base.escape_string(string(ch)) = l
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1143]: ParsingStateCode(p_state) = es_body_identity_eof
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1279]: onheadervalue 4
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1280]: len = 396
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1281]: p = 397
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1285]: this onbody 3
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:100]: onbody
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:101]: String(r.body) = 
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:102]: String(bytes[i:j]) = logged in user session:1493621710548
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:116]: String(r.body) = logged in user session:1493621710548
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1290]: exiting maybe unfinished...
[DEBUG - /home/tan/.julia/v0.5/HTTP/src/parser.jl:1291]: ParsingStateCode(p_state) = es_body_identity_eof
ERROR: LoadError: LoadError: RetryException: # of allowed retries (0) was exceeded when making request
 in request(::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions, ::HTTP.Connection{TCPSocket}, ::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool) at /home/tan/.julia/v0.5/HTTP/src/client.jl:225
 in #request#34(::Array{HTTP.Response,1}, ::Int64, ::Bool, ::Bool, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at /home/tan/.julia/v0.5/HTTP/src/client.jl:114
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Request, ::HTTP.RequestOptions) at ./<missing>:0
 in #request#32(::Dict{String,String}, ::HTTP.FIFOBuffer, ::Bool, ::Bool, ::Array{Any,1}, ::Function, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at /home/tan/.julia/v0.5/HTTP/src/client.jl:98
 in (::HTTP.#kw##request)(::Array{Any,1}, ::HTTP.#request, ::HTTP.Client{Base.TTY}, ::HTTP.Method, ::HTTP.URI) at ./<missing>:0
 in #get#51(::Bool, ::Dict{String,String}, ::Array{Any,1}, ::Function, ::String) at /home/tan/.julia/v0.5/HTTP/src/client.jl:425
 in (::HTTP.#kw##get)(::Array{Any,1}, ::HTTP.#get, ::String) at ./<missing>:0
...

If I print the response that causes this error, it seems complete:

HTTP.Response:
"""
HTTP/1.1 200 OK
Connection: close
Access-Control-Allow-Methods: GET, POST, DELETE, PUT
Content-Length: 36
X-Rate-Limit: 5000
X-Expires-After: Mon May 01 07:55:10 UTC 2017
Date: Mon, 01 May 2017 06:55:10 GMT
Content-Type: application/json
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Content-Type, api_key, Authorization
Server: Jetty(9.2.9.v20150224)

logged in user session:1493621710548
"""

Changing parser.jl here to m = b | (p_state >= s_body_identity_eof) makes it pass.

What is the difference between s_message_done and s_body_identity_eof states?
And what should be done to resolve this?

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @quinnj

Exception handling problems

If I configure a local server to disconnect before sending anything, or disconnect before sending the number of bytes promised in Content-Length, then I get RetryException even though i've disabled retries:

julia> HTTP.get(c, "http://127.0.0.1:8080"; retries=0)
ERROR: RetryException: # of allowed retries (0) was exceeded when making request

I expected to get an exception relating to the cause. e.g. curl produces:

$ curl 127.0.0.1:8080
curl: (52) Empty reply from server
$ curl 127.0.0.1:8080
curl: (18) transfer closed with 1 bytes remaining to read

RetryException does not seem to be something that should be exposed to the public HTTP API. If something was retried, but still didn't work the exception should describe what went wrong with the retry.

If I try to get from a non-existant port, I get MethodError (although a log of ECONNREFUSED is printed on the console):

julia> HTTP.get(c, "http://127.0.0.1:8081"; retries=0)
ERROR (unhandled task failure): connect: connection refused (ECONNREFUSED)
Stacktrace:
 [1] try_yieldto(::Base.##296#297{Task}, ::Task) at ./event.jl:189
 [2] wait() at ./event.jl:234
 [3] wait(::Condition) at ./event.jl:27
 [4] stream_wait(::TCPSocket, ::Condition, ::Vararg{Condition,N} where N) at ./stream.jl:42
 [5] wait_connected(::TCPSocket) at ./stream.jl:258
 [6] connect at ./stream.jl:983 [inlined]
 [7] connect(::IPv4, ::Int64) at ./socket.jl:738
 [8] (::HTTP.##37#38{HTTP.http,Nullable{Int64}})() at ./task.jl:335
ERROR: MethodError: no method matching HTTP.Connection(::Int64, ::Base.UVError)

I expected to get something like ECONNREFUSED as the exception.

UndefVarError: issetcookie not defined

I guess the following error is not intended:

julia> import HTTP
INFO: Recompiling stale cache file /Users/kenta/.julia/lib/v0.5/HTTP.ji for module HTTP.

julia> HTTP.get("https://github.com/JuliaWeb/HTTP.jl/blob/master/src/client.jl")
ERROR (unhandled task failure): UndefVarError: issetcookie not defined
 in #parse!#21(::Bool, ::String, ::HTTP.Method, ::Int64, ::Int64, ::Int64, ::Function, ::HTTP.Response, ::HTTP.Parser, ::Array{UInt8,1}, ::Int64) at /Users/kenta/.julia/v0.5/HTTP/src/parser.jl:797
 in (::HTTP.#kw##parse!)(::Array{Any,1}, ::HTTP.#parse!, ::HTTP.Response, ::HTTP.Parser, ::Array{UInt8,1}, ::Int64) at ./<missing>:0
 in macro expansion at /Users/kenta/.julia/v0.5/HTTP/src/client.jl:257 [inlined]
 in (::HTTP.##42#46{HTTP.Client{Base.TTY},HTTP.Connection{MbedTLS.SSLContext},HTTP.RequestOptions,String,HTTP.Method,HTTP.Response,Base.RefValue{Float64},Bool,Bool,HTTP.Parser})() at ./task.jl:360
HTTP.Response:
"""
HTTP/1.1 200 OK
X-UA-Compatible: IE=Edge,chrome=1
Date: Mon, 13 Feb 2017 17:02:37 GMT
Transfer-Encoding: chunked
Cache-Control: no-cache
Status: 200 OK
X-Request-Id: 0f36f8e4277a730f62f8356b5c060108
X-Runtime: 0.073020
Server: GitHub.com
Vary: X-PJAX
Set-Cookie: logged_in=no; domain=.github.com; path=/; expires=Fri, 13 Feb 2037 17:02:37 -0000; secure; HttpOnly, _gh_sess=eyJzZXNzaW9uX2lkIjoiYTQ5YTY3NGUzMTFkY2M5YjUyNjQzODQxY2MxNjU0YjAiLCJzcHlfcmVwbyI6Ikp1bGlhV2ViL0hUVFAuamwiLCJzcHlfcmVwb19hdCI6MTQ4NzAwNTM1NywiX2NzcmZfdG9rZW4iOiJLQlRSZXlaLzBnNUFhOG5DUHVCKzZrcWJsOFkvNTJGeHRCVEJlY2wwSTJZPSJ9--41ed448fe293d76a35cc90b87c9b589a86f6cc39; path=/; secure; HttpOnly
Content-Type: text/html; charset=utf-8

"""

julia>

Better top-level URI/query params handling

Currently:

User can provide a string on which we call HTTP.URI(str)
User needs to escape their own query parameters using HTTP.escape
We don't have a query keyword argument
We don't do any kind of validity checking on strings before calling HTTP.URI on them

I just think the url-as-a-string+query parameters interface needs a little more thought to make things more convenient. Also need good docs around this.

Basic Documentation

Some very basic pieces of information should be found on the frontpage:

Getting Started: How to start server, a typical Hello Example, or something like that.
The scope of this package. As I'm new to Julia's ecosystem, I would find it helpful to know which Web-components will be supported by this Package.
Some notes on compatibility with other Packages, like WebSocket.jl, the SSL packages, etc,

juliaweb / http.jl Goto Github PK

http.jl's People

Contributors

Stargazers

Watchers

Forkers

http.jl's Issues

Recommend Projects

Recommend Topics

Recommend Org