Code Monkey home page Code Monkey logo

http-proxy's Introduction

HTTP/S Proxy with extensions for Lantern

Go Actions Status

These are Lantern-specific middleware components for the HTTP Proxy in Go:

  • A filter for access tokens

  • A filter for devices, based on their IDs

  • A filter for Pro users

  • A connection preprocessor to intercept bad requests and send custom responses

  • Custom responses for mimicking Apache in certain cases

Deploying

All pushes to the main branch are automatically deployed to production via CI in GitHub Actions.

All pushes to the canary branch are automatically deployed to the canary binary distribution URL for any proxies running the canary version.

See .github/workflows/go.yml, which uses the make build Makefile target NOT THE LEGACY make dist-on-docker.

Rolling back a Deployment

The http-proxy binary is distributed through S3 as this object. This object is versioned in S3, so if you need to roll back to a prior deployed version, you can simply delete the currently deployed version from here.

Usage

Build it with go build or with make build.

To get list of the command line options, please run http-proxy-lantern -help.

config.ini.default also has the list of options, make a copy (say, config.ini) and tweak it as you wish, then run the proxy with

http-proxy-lantern -config config.ini

To regenerate config.ini.default just run http-proxy-lantern -dumpflags.

Testing with Lantern extensions and configuration

Run tests

go test

Use this for verbose output:

TRACE=1 go test

Manual testing

Keep in mind that cURL doesn't support tunneling through an HTTPS proxy, so if you use the -https option you have to use other tools for testing.

You can run a local server with a set configuration (just a default ReflectToSite proxy as of this writing) with

make local-proxy

Note that make local-proxy is really just an alias for make local-rts -- i.e. a ReflectToSite local proxy.

You can then copy the rts-proxies.yaml file to your Lantern config directory, as in:

cp ./rts/rts-proxies.yaml ~/Library/Application\ Support/Lantern/proxies.yaml

Run a Lantern client accordingly from lantern-desktop, as in:

./lantern -readableconfig -stickyconfig

If you're developing a new transport, you can also add new versions of those files for that transport as you're testing.

You have two options to test it: the Lantern client or checkfallbacks.

Keep in mind that they will need to send some headers in order to avoid receiving 404 messages (the chained server response if you aren't providing them).

Currently, the only header you need to add is X-Lantern-Device-Id.

If you are using checkfallbacks, make sure that both the certificate and the token are correct. A 404 will be the reply otherwise. Running the server with -debug may help you troubleshooting those scenarios.

Handle requests config server specially

To prevent spoofers from fetching Lantern config with fake client IP, we need to attach auth tokens to such requests. Both below options should be supplied. Once http-proxy-lantern receives GET request to one of the cfgsvrdomains, it sets X-Lantern-Config-Auth-Token header with supplied cfgsvrauthtoken, and X-Lantern-Config-Client-IP header with the IP address it sees.

  -cfgsvrauthtoken string
        Token attached to config-server requests, not attaching if empty
  -cfgsvrdomains string
        Config-server domains on which to attach auth token, separated by comma

When something bad happens

With option -pprofaddr=localhost:6060, you can always access lots of debug information from http://localhost:6060/debug/pprof. Ref https://golang.org/pkg/net/http/pprof/.

Be sure to only listen on localhost or private addresses for security reason.

Temporarily Deploying a Preview Binary to a Single Server

Sometimes it's useful to deploy a preview binary to a single server. This can be done using either deployTo.bash or onlyDeployTo.bash. They do the same thing but deployTo.bash first runs make dist whereas onlyDeployTo.bash copies the existing binary at dist/http-proxy.

Deploying a Custom Binary

Sometimes it's useful to deploy a custom binary to one or more tracks. This can be done by running make deploy-custom and setting the environment variable BINARY_NAME to the desired binary name, e.g. http-proxy-custom-hwh33-tlsmasq999.

To deploy a track running the custom binary, add the custom_proxy_binary key to the track's pillar data, mapped to the name specified above. At the time of writing, track pillar data is specified in the track_pillars structure in lantern-infrastructure/etc/current_production_track_config.py

For example:

    'etc-teleport': {
        'custom_proxy_binary': 'http-proxy-custom-teleport-1',
    },

ssh config

Most of our proxies have servermasq enabled on them. This means that you cannot ssh directly into them. Instead you have to use a cloudmaster as a bastion jump host. You can do this relatively straightforwardly by adding this to your ~/.ssh/config file:

Host bastion
  HostName CM_IP
  ProxyCommand none
Host *
  User lantern
  ProxyJump bastion

where you'd replace CM_IP with an actual cloudmaster ip (probably the one closest to you).

Deploy Preview

./onlyDeployTo.bash <ip address>

Revert to Production Binary

Once you're done checking out the preview, revert back to the production binary with:

./revertToProductionBinary.bash <ip addres>

Logs on Server

To view proxy logs on a given machine, run:

journalctl -e -u http-proxy

http-proxy's People

Stargazers

 avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

Forkers

madhav-dhungana

http-proxy's Issues

Craigslist is really slow when accessed through proxy

cc: @myleshorton @aranhoide

This has been bothering me for a while. I thought it was due to a problem with our code, but I wrote a little test Go program that works just fine, and in fact just proxying through my Lantern client without hitting http-proxy, it's fine too.

package main

import (
    "io"
    "io/ioutil"
    "log"
    "net/http"
    "time"
)

func main() {
    start := time.Now()
    resp, err := http.Get("http://austin.craigslist.org/")
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()
    n, err := io.Copy(ioutil.Discard, resp.Body)
    if err != nil {
        log.Fatal(err)
    }
    delta := time.Now().Sub(start)
    log.Print(delta)
    log.Print(n)
}
go run main.go
2016/10/21 15:04:00 184.968188ms
2016/10/21 15:04:00 56259

56K in 184 ms

I then ran my test program on my proxy server and found that it was dog slow:

lantern@fp-obfs4-donyc3-20160925-001:~$ ./craigslist
2016/10/21 20:08:09 40.679969133s
2016/10/21 20:08:09 56259

56K in 40 seconds!

Curl is even slower, so slow that I stopped it at 43K

lantern@fp-obfs4-donyc3-20160925-001:~$ curl http://austin.craigslist.org/ > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 42842    0 42842    0     0    254      0 --:--:--  0:02:48 --:--:--   202

Suspecting that Craigslist is throttling known proxy servers, I spooled up another donyc3 machine and tried my test from there. It was also really slow, so perhaps they're throttling the entire datacenter.

root@craigslist:~# ./craigslist
2016/10/21 20:16:56 41.130397822s
2016/10/21 20:16:56 56259

In any case, this is not a problem with our code, so I'm just logging this for the record (in case it comes up again).

Error forwarding

{"extra":{"logLevel":"ERROR","version":"version (releasedate)"},"fullMessage":"ERROR errorhandler: errorhandler.go:36 Responding with 502 due to dial tcp: lookup hcfq9zfs.vmgoxp64.netdna-cdn.com on 8.8.8.8:53: no such host: Error forwarding from 91.99.1.144:54579 to hcfq9zfs.vmgoxp64.netdna-cdn.com\n","hostname":"fp-doams3-20160330-002","instanceid":"instanceid","locationInfo":"ERROR errorhandler","message":"no such host: Error forwarding from 91.99.1.144:54579 to hcfq9zfs.vmgoxp64.netdna-cdn.com","timestamp":1463722704992}

Vulnerability: Several requests with Host: localhost:<proxy-port> will segfault the server

Discovered after working with tests.
The reason for the segfault seems to be resource exhaustion.
This probably comes as a result of an infinite loop of redirection. As a protection, we had a filter for local interfaces addresses, but since UDPgw needed that to be taken out, we are currently exposed.
The solution seems simple: reactivate loop-avoidance filter but with the exception of UDPgw port. I will test this further.

Pro purchases on mobile: reliability of browser redirects

For mobile users to sign up for Pro using Alipay, we have to redirect them to the browser to open Stripe Checkout. [1] Once payment information is verified by Stripe, the user is re-directed back to the app, along with the Stripe token and e-mail address. This is done using an intent filter with a custom scheme, defined in the AndroidManifest, which tells the browser to open our app directly. Only back in the app do we submit the Pro token, device ID, and Stripe credentials to the /purchase endpoint.

The biggest issue with this is approach is that newer versions of Chrome on Android are blocking re-directs to custom URIs completely (unless the app is registered in the Play store). I can get this to work reliably in the stock browser though. But we can't depend on this..

A possible solution would be to configure a webhook to detect whenever a user first submits their credit card or Alipay information to Stripe. We'd have to pass along the pro token and device id somehow. Stripe would respond to this event, instead of returning a token, by hitting our /purchase endpoint. Then whenever the user re-opens the app, we would know via the /user-data call a user is Pro and be able to display a welcome message/switch the UI to Pro mode.

[1] Issues loading Stripe Checkout in a WebView necessitate re-directing the user to the browser

Limit concurrent devices

The purpose of this is to prevent people from abusing Lantern infrastructure by distributing their pro token to other people.

Have lantern clients report blockages to Google Analytics via domain fronting

I'm not sure that this is the best place to log this, but I needed a secret repo ...

While chatting with @aranhoide, it occurred to me that we have a great network of blocking detectors, namely our Lantern clients. When Lantern clients detect that they're consistently unable to connect to a proxy at some IP address in a way that it looks blocked, we should have them report this. We could log this to Google Analytics and treat each server and/or IP as a web page URL, and geolocate the clients while we're at it. This could provide some valuable insights into blocking.

Seeing throttling messages on a pro server

I see messages in the log for fp-obfs4-ggtwnb-20160912-060 like this:

Oct 01 02:58:41.258 DEBUG devicefilter: devicefilter.go:90 Throttling connection [client_ip=119.248.246.109 device_id=QmidpI/c op=tokenfilter origin=config.getiantem.org root_op=proxy]

Note - I see this for other IPs and device IDs on this server too.

In theory, this is a pro-only server, so all connecting clients should be pro, so nothing should get throttled.

Make Lantern look less like a proxy so that we don't get flagged by origin servers

Right now, sites like Google will misidentify traffic coming through our proxies as a bot, and will show captchas and such annoying things to force people to prove they're not bots. Similar logic is also used by content providers to prohibit the use of proxies.

We could address both issues by:

  1. Using a of IPv6 addresses for outbound connections (so the origin server sees less traffic by IP)
  2. Dropping X-Forwarded-For and any other proxy-added headers in case that's necessary

Enabling bandwidth tracking and throttling significantly increases load

In a production proxy, it makes the difference between about 0.4-0.5 vs 1.0+ load.

Note - the throttling limits are set high enough that no one should be getting throttled, so this load is purely from having the latent capability.

Here are some 2 minute profiles

With bandwidth tracking and throttling

profile

With only bandwidth tracking

profile_nothrottle

With neither throttling nor bandwidth tracking

profile_noreport

IP-Packet forwarding

IP-Packet forwarding to tunnel the IP packets themselves to
Lantern’s proxies, which then recreate the TCP streams on the server side

http-proxy-test blocks on acquiring mutex with Go 1.8rc3

It looks like the locking on the http server implementation changed and we're getting blocked on hijacking:

goroutine 56 [semacquire]:
sync.runtime_notifyListWait(0xc42024e410, 0x0)
	/usr/local/go/src/runtime/sema.go:297 +0x10b
sync.(*Cond).Wait(0xc42024e400)
	/usr/local/go/src/sync/cond.go:57 +0x89
net/http.(*connReader).abortPendingRead(0xc42024e380)
	/usr/local/go/src/net/http/server.go:686 +0xbf
net/http.(*conn).hijackLocked(0xc4201d57c0, 0x46441a8, 0xc4201d5850, 0x4580140, 0x45f88a0, 0xc42028e500)
	/usr/local/go/src/net/http/server.go:292 +0x53
net/http.(*response).Hijack(0xc42020c620, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:1892 +0xfc
github.com/getlantern/http-proxy-lantern/mimic.MimicApache(0x48dd080, 0xc42020c620, 0xc420185500)
	/Users/ox.to.a.cart/gocode/src/github.com/getlantern/http-proxy-lantern/mimic/apache.go:56 +0x6d
github.com/getlantern/http-proxy-lantern/tokenfilter.(*tokenFilter).Apply(0xc420222870, 0x48dd080, 0xc42020c620, 0xc420185500, 0xc42028a880, 0x0, 0x0)
	/Users/ox.to.a.cart/gocode/src/github.com/getlantern/http-proxy-lantern/tokenfilter/tokenfilter.go:47 +0x5b0
github.com/getlantern/http-proxy/filters.Chain.ServeHTTP(0xc420210d80, 0x3, 0x3, 0x48dd080, 0xc42020c620, 0xc420185500)
	/Users/ox.to.a.cart/gocode/src/github.com/getlantern/http-proxy/filters/filters.go:60 +0x167
github.com/getlantern/http-proxy/filters.(*Chain).ServeHTTP(0xc4201cfda0, 0x48dd080, 0xc42020c620, 0xc420185500)
	<autogenerated>:4 +0x7d
github.com/getlantern/http-proxy/server.NewServer.func1(0x48dd080, 0xc42020c620, 0xc420185500)
	/Users/ox.to.a.cart/gocode/src/github.com/getlantern/http-proxy/server/server.go:40 +0x111
net/http.HandlerFunc.ServeHTTP(0xc4201cfdc0, 0x48dd080, 0xc42020c620, 0xc420185500)
	/usr/local/go/src/net/http/server.go:1942 +0x44
net/http.serverHandler.ServeHTTP(0xc42018f5f8, 0x48dd080, 0xc42020c620, 0xc420185500)
	/usr/local/go/src/net/http/server.go:2568 +0x92
net/http.(*conn).serve(0xc4201d57c0, 0x48ddc40, 0xc42024e340)
	/usr/local/go/src/net/http/server.go:1825 +0x612
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2668 +0x2ce

GA related errors logged in loggly

Example:

{"extra":{"logLevel":"ERROR","version":"version (releasedate)"},"fullMessage":"ERROR http-proxy-lantern.analytics: analytics.go:123 Could not send HTTP request to GA: Post https://ssl.google-analytics.com/collect: dial tcp 173.194.126.190:443: i/o timeout\n","hostname":"fp-jp-20151019-519","instanceid":"instanceid","locationInfo":"ERROR http-proxy-lantern.analytics","message":"dial tcp 173.194.126.190:443: i/o timeout","timestamp":1454050540011}

Maybe silence these unless we want to make sure that all requests get to GA.

Makefile does not add all relevant changes to the changelog

This probably boils down to an issue with the git-chglog tool we're using. I ran make dist with git-chglog v0.9.1 and got the following:

2021-03-04
Pull Requests
Merge pull request #455 from getlantern/ox/throttle_logging

This misses a few commits and PRs which show up in git log:

commit c99474b0258c0ca8729d016e433d2c11c2534eee (HEAD -> master, origin/master, origin/HEAD)
Author: Harry Harpham <[email protected]>
Date:   Thu Mar 4 13:17:48 2021 -0700
    Updated changelog for v2.6.32
 CHANGELOG.md | 150 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------------------------------------------
 1 file changed, 86 insertions(+), 64 deletions(-)
commit 256dbb7701cf026cb3dd7073c26de598ef290a86 (tag: v2.6.32)
Author: hwh33 <[email protected]>
Date:   Thu Mar 4 13:11:58 2021 -0700
    Incorporate tlsmasq bug fix: (#456)
    https://github.com/getlantern/tlsmasq/issues/9
 go.mod | 2 +-
 go.sum | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)
commit 6557a8e09bb141a9ba379c5e4e31717f79d27db3
Merge: 51cd502 4578921
Author: Myles Horton <[email protected]>
Date:   Wed Mar 3 08:26:17 2021 -0500
    Merge pull request #455 from getlantern/ox/throttle_logging
    Added trace logging to help watch throttle cohort assignment
commit 45789218213aa4b57d8f601eb54344863a261abd
Author: Ox Cart <[email protected]>
Date:   Tue Mar 2 23:19:12 2021 -0600
    Added trace logging to help watch throttle cohort assignment
 redis/measured_reporter.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
commit 51cd50205001f4b80115e9045c50482c5ba193b4
Author: Myles Horton <[email protected]>
Date:   Mon Feb 15 13:21:26 2021 -0600
    Log details
 README.md | 7 +++++++
 1 file changed, 7 insertions(+)
commit bbb13b862e06553ed4c73bd559d9567a4bbf5c06
Author: hwh33 <[email protected]>
Date:   Fri Feb 12 09:02:15 2021 -0700
    Update tlsmasq (#454)
 go.mod |  2 +-
 go.sum | 16 ++++++++++------
 2 files changed, 11 insertions(+), 7 deletions(-)
commit e459a3a09d13e2cbcac7427f62e9af18c59ed2ea
Author: Ox Cart <[email protected]>
Date:   Mon Feb 8 13:33:40 2021 -0600
    Updated changelog for v2.6.31
 CHANGELOG.md | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)
commit 1307ddde14f049911974d4e49f3400ac0d34045f (tag: v2.6.31)
Merge: 460a349 23a7bd6
Author: Percy Wegmann <[email protected]>
Date:   Mon Feb 8 13:31:11 2021 -0600
    Merge pull request #453 from getlantern/ox/issue452
    Ignore empty array headers for measured

There is currently one later version, git-chglog v0.1.0, but it failed to compile for me.

XBQ header not appearing

This is currently happening with what looks like every request onhttps://cloud.digitalocean.com/droplets/12291187/graphs

OBFS4 proxies getting stuck in accept loop

Jun 09 14:19:41.942 ERROR server: server.go:2169 http: Accept error: read tcp 128.199.223.25:443-
Jun 09 14:20:46.956 ERROR server: server.go:2169 http: Accept error: read tcp 128.199.223.25:443->61.131.11.252:56611: i/o timeout; retrying in 10ms
Jun 09 14:21:51.973 ERROR server: server.go:2169 http: Accept error: read tcp 128.199.223.25:443-
Jun 09 14:22:56.996 ERROR server: server.go:2169 http: Accept error: read tcp 128.199.223.25:443->122.189.210.208:16086: i/o timeout; retrying in 40ms
Jun 09 14:24:02.038 ERROR server: server.go:2169 http: Accept error: read tcp 128.199.223.25:443->61.131.11.252:57558: i/o timeout; retrying in 80ms
Jun 09 14:25:07.120 ERROR server: server.go:2169 http: Accept error: read tcp 128.199.223.25:443->61.131.11.252:57560: i/o timeout; retrying in 160ms

Because Accept() is blocking, this prevents us from processing other connections.

Proxy might be throwing too many Bad Gateway errors

Related to getlantern/lantern#2645

This started with @aranhoide noticing that the proxy checkers were getting 502 errors. I added logging to see what causes the 502s, and there are various root causes. Below is a sampling of the errors from one server. Not sure if it's relevant, but interestingly all of the sites in question seem to be Chinese sites.

May 13 02:45:56.015 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:43564->123.58.182.253:80: use of closed network connection: Error forwarding from 14.23.61.75:42217 to notify3.ynote.youdao.com
May 13 02:45:56.084 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:45:58.789 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:45:59.470 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:43623->123.58.182.253:80: use of closed network connection: Error forwarding from 14.23.114.74:37633 to notify3.note.youdao.com
May 13 02:46:01.052 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:02.309 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:43669->123.58.182.253:80: use of closed network connection: Error forwarding from 14.18.205.103:65380 to notify3.ynote.youdao.com
May 13 02:46:05.085 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:07.029 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:07.073 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:43780->123.58.182.253:80: use of closed network connection: Error forwarding from 1.202.253.18:59705 to notify3.note.youdao.com
May 13 02:46:07.508 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:50499->123.125.65.210:80: use of closed network connection: Error forwarding from 14.23.80.167:23383 to message.tieba.baidu.com
May 13 02:46:08.021 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to dial tcp 1.1.1.1:80: getsockopt: network is unreachable: Error forwarding from 39.191.141.238:50420 to 1.1.1.1
May 13 02:46:08.744 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:43801->123.58.182.253:80: use of closed network connection: Error forwarding from 14.23.61.75:44500 to notify3.note.youdao.com
May 13 02:46:10.065 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:13.098 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:16.085 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:16.369 ERROR errorhandler: errorhandler.go:36 Responding with 504 due to dial tcp 198.11.189.98:80: i/o timeout: Error forwarding from 42.49.164.195:53687 to alimei-sub.alibaba.com
May 13 02:46:16.759 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:16.804 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to dial tcp 218.30.114.205:80: getsockopt: connection refused: Error forwarding from 1.202.222.147:30673 to online.sso.sina.com.cn
May 13 02:46:18.552 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:43913->123.58.182.253:80: use of closed network connection: Error forwarding from 1.202.169.226:51567 to notify3.ynote.youdao.com
May 13 02:46:19.116 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:26018 to hb.crm2.qq.com
May 13 02:46:20.157 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:44411->203.205.144.215:80: read: connection reset by peer: Error forwarding from 14.29.125.102:54274 to masterconn.qq.com
May 13 02:46:21.776 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:21.914 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:55707->180.149.138.108:80: use of closed network connection: Error forwarding from 1.202.15.102:59406 to 11.108.web1.im.weibo.com
May 13 02:46:22.072 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:22.822 ERROR errorhandler: errorhandler.go:36 Responding with 504 due to dial tcp 106.120.160.106:80: i/o timeout: Error forwarding from 1.202.193.65:51318 to 106.120.160.106
May 13 02:46:24.136 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:26018 to hb.crm2.qq.com
May 13 02:46:28.080 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:28.081 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:26018 to hb.crm2.qq.com
May 13 02:46:29.780 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:32.016 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:44023->123.58.182.253:80: use of closed network connection: Error forwarding from 14.18.207.75:52267 to notify3.note.youdao.com
May 13 02:46:32.273 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:44094->123.58.182.253:80: use of closed network connection: Error forwarding from 14.23.114.74:7233 to notify3.note.youdao.com
May 13 02:46:32.475 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:44098->123.58.182.253:80: use of closed network connection: Error forwarding from 14.18.236.70:11659 to notify3.note.youdao.com
May 13 02:46:33.049 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:33.074 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:26018 to hb.crm2.qq.com
May 13 02:46:33.401 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:44116->123.58.182.253:80: use of closed network connection: Error forwarding from 1.202.237.242:54824 to notify3.note.youdao.com
May 13 02:46:35.068 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:38.045 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:38.070 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:26018 to hb.crm2.qq.com
May 13 02:46:39.252 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:50912->123.125.65.210:80: use of closed network connection: Error forwarding from 14.23.80.167:24398 to message.tieba.baidu.com
May 13 02:46:39.455 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:44204->123.58.182.253:80: use of closed network connection: Error forwarding from 14.23.61.75:50814 to notify3.note.youdao.com
May 13 02:46:40.028 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:43.773 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:44.075 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:46.103 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:26018 to hb.crm2.qq.com
May 13 02:46:46.813 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:44292->123.58.182.253:80: use of closed network connection: Error forwarding from 14.18.207.75:52290 to notify3.note.youdao.com
May 13 02:46:49.079 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 14.18.205.103:58148 to hb.crm2.qq.com
May 13 02:46:49.097 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:23719 to hb.crm2.qq.com
May 13 02:46:49.369 ERROR errorhandler: errorhandler.go:36 Responding with 504 due to dial tcp 198.11.189.98:80: i/o timeout: Error forwarding from 42.49.164.195:53687 to alimei-sub.alibaba.com
May 13 02:46:50.958 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to read tcp 45.32.60.169:44383->123.58.182.253:80: use of closed network connection: Error forwarding from 14.18.207.75:61524 to notify3.note.youdao.com
May 13 02:46:51.092 ERROR errorhandler: errorhandler.go:36 Responding with 502 due to EOF: Error forwarding from 1.202.252.174:26018 to hb.crm2.qq.com

Redis Restart Wreaks Havoc on http-proxy

@oxtoacart commented on Fri Oct 21 2016

I noticed yesterday that several of my pro proxies were unresponsive. It turns out that they were stuck in a restart loop waiting for Redis to finish "loading dataset in memory". It turns out that Redis had been restarted due to a system update, and some of my proxies restarted at a similar time. This made Lantern basically unusable for me until Redis finished loading its dataset in memory (which takes a while).

We could temporarily disable system updates for the redis server, but I don't like the fact that this means we'll miss security patches. In reality, I think we will need to take patches at some point and restart, so we need to figure out a way to make our system resilient to redis restarts.

One way to address this would be to have proxies locally store the data that they need (i.e. authorized pro users) so that even if redis is temporarily unavailable, they can continue to function.

Another option, which would also help us secure our http-proxies for use on services like Aliyun is to not even connect them with Redis at all. If we had an authentication server for pro users that issued time-limited, signed authentication tokens, our pro http-proxies could simply check the signature on the tokens to verify that the user is in fact pro. This would also have the side benefit of reducing load on redis.

We'd also have to figure out what to do about throttling, but I bet that's solvable too. One easy solution would be to just stop pretending that we're throttling since we're not actually throttling anyway :)

A completely different solution would be to switch to a disk-based database that's api-compatible with Redis and can start up in a reasonable amount of time (e.g. LedisDB). I don't love this option because it introduces a fair amount of risk, especially from a performance perspective.

cc: @uaalto @aranhoide @myleshorton

Users getting assigned to wrong throttling cohort because of missing supported data caps header

The problem is on persistent HTTP connections, were HTTP requests after the first are missing values for the header. We have logic to prevent this, but it didn't support arrays of values.

So for example, you see something like this with extra trace logging:

Feb  7 16:56:12 fp-mskr1c001sea-20210202-068 http-proxy[1675]: TRACE redis: measured_reporter.go:175 throttle cohort for device blahblah country code CN platform windows version 6.3.5 and supported data caps [monthly weekly daily] is china monthly
Feb  7 16:57:35 fp-mskr1c001sea-20210202-068 http-proxy[1675]: TRACE redis: measured_reporter.go:175 throttle cohort for device blahblah country code CN platform windows version 6.3.5 and supported data caps [] is china legacy

Make sure everything works when hit via an IPv6 address

(https://github.com/getlantern/lantern_aws/pull/123 depends on this.)

We would like to publish IPv6 addresses for proxies it the cloud.yaml served by the config server, because that might help with infiltration. Before we do that, we need to make sure that http-proxy(-lantern) works without problems when reached through a IPv6 address.

Feel free to close this and move it to getlantern/http-proxy if you think it belongs there instead. Or just close it if it seems super obvious that http-proxy(-lantern) is not depending on IPv4 in any way.

Proxy 104.238.188.202 reports mismatched token on requests from proxy checker

This proxy is consistently failing checks from multiple proxy checkers, and the log shows that the proxy checkers are sending a different token than the proxy expects.

May 25 20:02:14.861 DEBUG blacklist: blacklist.go:125 104.238.188.202 connected but failed to successfully send an HTTP request within 30s
May 25 20:11:48.517 DEBUG tokenfilter: tokenfilter.go:72 Mismatched token(s) iQb4HrXHloyl38W6nRnSARnkzxo98hulWCNVMLIJROWLreNs3ruKcreztH9odjqM from 104.238.188.202:35879 for request to www.google.com, mimicking apache

Eventually this results in the checker becoming blacklisted.

cc: @aranhoide

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.