lancachenet / generic Goto Github PK
View Code? Open in Web Editor NEWGeneric LAN Download Content Cache
Home Page: https://hub.docker.com/r/lancachenet/generic/
License: MIT License
Generic LAN Download Content Cache
Home Page: https://hub.docker.com/r/lancachenet/generic/
License: MIT License
Can we expect Epic Games support in the future? (Fortnite, Unreal Tournament, UE4, etc.)
Thanks!
The Blizzard client is having issues with the bnet cache docker.
Other cache dockers (Steam and Origin) download at full network speeds (3.2mb/s). and caches the files downloaded and retrieved correctly. (Origin is also reporting false hits too when downloading fresh cache files, but not a major issue right now)
Using Ubuntu 18.10 with latest docker-ce and latest docker pulls.
Below is a Grafana parsing of the logs, the client download speed and disk usage.
Blizzard / Battle.net
docker run \
--restart unless-stopped -d \
--name blizzard-cache \
-v /home/data/steamcache/blizzard/data:/data/cache \
-v /home/data/steamcache/blizzard/logs:/data/logs \
-p 192.168.1.212:80:80 \
steamcache/generic:latest
DNS
docker run \
--restart unless-stopped -d \
--name steamcache-dns \
-p 192.168.1.210:53:53/udp \
-e UPSTREAM_DNS=1.1.1.1 \
-e STEAMCACHE_IP=192.168.1.210 \
-e ORIGINCACHE_IP=192.168.1.211 \
-e BLIZZARDCACHE_IP=192.168.1.212 \
-e RIOTCACHE_IP=192.168.1.213 \
-e FRONTIERCACHE_IP=192.168.1.214 \
-e WSUSCACHE_IP=192.168.1.215 \
steamcache/steamcache-dns:latest
HTTPS
docker run \
--restart unless-stopped -d \
--name sniproxy \
-p 443:443 \
steamcache/sniproxy:latest
error.log tail
2018/11/07 02:54:06 [error] 20#20: *14622 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", subrequest: "/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", upstream: "http://137.221.64.1:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
2018/11/07 02:54:08 [error] 16#16: *14646 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", subrequest: "/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", upstream: "http://137.221.64.4:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
2018/11/07 02:54:10 [error] 14#14: *14473 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", upstream: "http://137.221.64.4:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
2018/11/07 02:54:11 [error] 15#15: *14552 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", subrequest: "/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", upstream: "http://137.221.64.5:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
2018/11/07 02:54:17 [error] 24#24: *14602 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.1.120, server: , request: "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1", upstream: "http://137.221.64.3:80/tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8", host: "us.cdn.blizzard.com"
Access.log tail
192.168.1.120 / - - - [07/Nov/2018:02:55:14 +0000] "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1" 206 266240 "-" "-" "HIT" "us.cdn.blizzard.com" "bytes=147858222-148124461"
192.168.1.120 / - - - [07/Nov/2018:02:55:15 +0000] "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1" 206 266240 "-" "-" "MISS" "level3.blizzard.com" "bytes=158507822-158774061"
192.168.1.120 / - - - [07/Nov/2018:02:55:15 +0000] "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=158774062-159040301"
192.168.1.120 / - - - [07/Nov/2018:02:55:15 +0000] "GET /tpr/ovw/data/44/3f/443f051f88353e3e42254de35a8ef8a8 HTTP/1.1" 206 266240 "-" "-" "HIT" "level3.blizzard.com" "bytes=159040302-159306541"
A nice feature additional to the toolchain from the generic / steamcache itself would be to prefill the Cache with actual content beforehand a bigger event.
The way i solve it until now is to predownload a "most common" set of games through the gameservices. depending on the Client this uses much time for micromanaging to delete the local installs.
How about a tool that uses functions from the steamcmd tool, to download a set of specific game IDs, and delete them afterwards and/or if possible for other plattforms aswell.
This project has a bunch of other redirects that would be great if integrated into steamcache as well.
Using that project directly is surely an option but steamcache is super easy to get up and running with its docker support and IMHO seems much simpler to me.
So I have searched online and I can't find the cause of my issue at all, I have all my containers up and running but I cannot figure out why when I run the :
docker exec -it steam-cache /scripts/watchlog.sh
commannd it just returns this every time:
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: "/scripts/watchlog.sh": stat /scripts/watchlog.sh: no such file or directory": unknown
Following are the containers I have running:
sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cd9394c087f5 steamcache/steamcache-dns:latest "bash /scripts/boots…" 8 hours ago Up 3 seconds 192.168.1.142:53->53/udp steamcache-dns
24192243e997 steamcache/sniproxy:latest "sniproxy -f" 9 hours ago Up 8 hours 0.0.0.0:443->443/tcp sniproxy
f20a8c4a43c8 steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.146:80->80/tcp uplay-cache
23744b50368b steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.145:80->80/tcp windows-cache
1c3a35a7db33 steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.144:80->80/tcp blizzard-cache
de3c152902be steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.143:80->80/tcp origin-cache
2ad98fb607f9 steamcache/generic:latest "/scripts/bootstrap.…" 9 hours ago Up 8 hours 192.168.1.142:80->80/tcp steam-cache
Any help would be much appreciated.
My caching server suddenly started getting upstream time outs. I didn't make any changes to my configuration so I'm not sure what's causing it
I have Steamcache's sniproxy/generic/steamdns installed in containers on my Synology NAS.
Error I'm getting:
2018/08/08 09:49:44 [error] 27#27: *1036 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 192.168.0.3, server: , request: "GET /depot/420291/manifest/3127109883701325934/5?l=14&e=1534280479&sid=14638906&h=6bac8f454f2ab671ed054b10ce47ca4dd5410947 HTTP/1.1", upstream: "http://162.254.197.12:80/depot/420291/manifest/3127109883701325934/5?l=14&e=1534280479&sid=14638906&h=6bac8f454f2ab671ed054b10ce47ca4dd5410947", host: "valve502.steamcontent.com"
If I try testing it manually by going to the upstream link: http://162.254.197.12:80/depot/420291/manifest/3127109883701325934/5?l=14&e=1534280479&sid=14638906&h=6bac8f454f2ab671ed054b10ce47ca4dd5410947
I get unauthorized.
But if I include the hostname it works:
Hi,
could you add RSI Downloader (Star Citizen?) Its always huge GB to download.
Thanks.
Speaking to @astrolox about how to go about testing the cache images.
Initial thoughts:
This only covers a simple case though. It would be good for advanced options for testing range requests, etag headers, etc.
Other people's thoughts and comments?
Hello,
I am having trouble deploying the Docker container the way it is documented.
When I run this command:
docker run --restart unless-stopped --name cache-steam -v /cache/steam/data:/data/cache -v /cache/steam/logs:/data/logs -p 10.0.2.15:80:80 steamcache/generic:latest
As its documented under: https://hub.docker.com/r/steamcache/generic/
I only recieve this error Message:
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
2017/12/12 12:36:38 [emerg] 10#10: mkdir() "/data/cache/cache" failed (13: Permission denied)
nginx: [emerg] mkdir() "/data/cache/cache" failed (13: Permission denied)
nginx: configuration file /etc/nginx/nginx.conf test failed
When I try and run the Docker passing only "-v /cache/steam:/data" like this:
docker run --restart unless-stopped --name cache-steam -v /cache/steam:/data -p 10.0.2.15:80:80 steamcache/generic:latest
the container actually starts successfully, created the "/cache/steam/data" and "/cache/steam/logs" directorys and even caches Steam downloads, but the internal Docker folder /data is not linked to /cache/steam on the Server itself.
The internal directorys look like this:
docker exec cache-steam /bin/sh -c "ls -al /data/cache"
total 12
drwxr-xr-x 3 nginx nginx 4096 Dec 12 12:56 .
drwxr-xr-x 4 root root 4096 Dec 12 12:27 ..
drwx------ 2 nginx root 4096 Dec 12 12:56 cache
e.g: /data/cache/cache
Did I miss a step in the configuration or is the Documentation simply outdated?
Cheers,
Finn
Steamcache can still get stuck in a loop when requesting it's own IP
Within container
wget http://127.0.0.1/looop -S -O -
Nginx should detect the X-SteamCache-Processed-By header and drop the request
DNS issue please ignore.
Checking the heartbeat in wget is fine with no loops, checking it in chrome generates a request loop
wget http://cacheip/steamcache-heartbeat -S -O -
I am redirecting the data created by the lancache docker (see my command line below).
sudo docker run --name lancache --restart=always -d -v /srv/lancache/data:/data/cache -v /srv/lancache/logs:/data/logs -p 10.0.0.12:80:80 steamcache/generic:latest
Are the /data/cache and the /data/logs the only directories I have to redirect?
Can I just redirect /data to /srv/lancache/data (or another directory on the computer running the cache) and get the same result?
Any plans of adding PlayStation Network compatibility too?
(edited by admin)
Its not an issue direct, its more a question on how you handle the log file watching.
I like to analyze the log files in production use to see how much files are served from the cache and how much not.
Are there any tool you have for this purpose?
The docker-compose file from the docker-compose Repository.
One container of the generic cache for every service.
Logfiles under /data/{service}/logs/access.log
No problem, everything works fine
Is it possible to run this generic lancache on the same machine which run steamcache?
I get a bind x.x.x.x:80 failed:port is already allocated error message.
hello, caching on blizzard is having some issues, several gaming lounges reported the issue.
As soon as i stopped the container that was running blizzard cache and stopped the dns re-route downloads worked fine
issues : blizzard app is stuck on 0% and does not update , game updates also stuck or run for a while and then hang with a download speed reaching 0-25b/s
im running those commands :
sudo docker run --name cache-steam --restart=always -d -v /cache/steam/data:/data/cache -v /cache/steam/logs:/data/logs -p 192.168.88.68:80:80 steamcache/generic:latest
sudo docker run --name cache-blizzard --restart=always -d -v /cache/blizzard/data:/data/cache -v /cache/blizzard/logs:/data/logs -p 192.168.88.69:80:80 steamcache/generic:latest
sudo docker run --name steamcache-dns --restart=always -p 192.168.88.68:53:53/udp -e CACHE_MAX_AGE=95d -e UPSTREAM_DNS=1.1.1.1 -e STEAMCACHE_IP=192.168.88.68 -e BLIZZARDCACHE_IP=192.168.88.69 -e DISABLE_RIOT=true -e DISABLE_FRONTIER=true -e DISABLE_ORIGIN=true -e DISABLE_WINDOWS=true steamcache/steamcache-dns:latest
note that the caching was working fine for the past couple of months, suddenly the blizzard issue came up in several gaming lounges
Hi,
I'm trying to run the steamcache generic with the following comand;
docker run \
--restart unless-stopped \
--name steam-cache \
-v /home/sniper/Lan/steamcache/steam/data:/data/cache \
-v /home/sniper/Lan/steamcache/steam/logs:/data/logs \
-p 80:80 \
steamcache/generic:latest
But I get the error:
Error response from daemon: driver failed programming external connectivity on endpoint steam-cache (189994c288542e25e5360c614a0035be230385b6fec688dd96f7d77c2a764387): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use. ERRO[0000] error waiting for container: context canceled
�
How can I fix it?
thanks
Tracked via uklans/cache-domains#33
Hello,
before using steamcache I had download speeds from 45-50MB/s .
Now with steamcache (run in docker) the initial download speed is at max 8-9 MB/s per second.
What could the issue be here?
I would like to know if this repo works flawless as I see so many docker steamcache and dont know which to pick. I would like something to cache all games from any editors. Thank you
I am running a combined Steamcache/Lancache setup which works like a charm.
My Origin games are cached when I install them so a second installation will be taken care of by the local cache, like designed.
However, if the Origin client on my game PC needs to be updated, it keeps telling me to work offline and is unable to connect to the Origin cloud and download the client update files.
If I stop the Origin cache I can perform the Origin client update without any problem.
Not a big issue for me but less experienced users on my network will be confused by these extra steps...
As the title says, windows updates are no longer caching. In fact they ajust hange when routed through generic cache.
The generic image appears to suffer from the same issue described here, in the steamcache image.
Would it be possible to have this implemented for the generic image as well in order to improve caching?
Currently, a game has to be downloaded 6+ times in order to get enough saturation of chunks to guarantee the local cache is used.
Hello,
I want to use this with my QNAP NAS.
QNAP has a Container Station for handling docker images:
https://www.qnap.com/solution/container_station/en/
How do I need to configure it so it will work ?
Tracked via uklans/cache-domains#32
Hi, @mintopia @mikeweyandt @GotenXiao @astrolox @JasonRivers,
I was trying to use generic image with steamcache and steamcache-dns.
I installed the generic image on another server, it seemed to work fine, but now blizzard is not updating
Any idea? Is it working fien for you guys?
Best Regards,
For steam, /server-status shouldn't be cached.
Contains load information about that CDN server, and may result in lower performance if it remains cached.
example content:
"status" { "csid" "3" "load" "22" "cell" "31" }
I'm having some difficulty getting GTAV to download from the cache. The game was already downloaded to my caching server, but now the launcher keeps stating that it lost connection to the download servers. This can be reproduced on any computer I try to download it from. Avoiding the cache allows the game to download normally. Interestingly, I did check the error log for the Rockstar cache as it tried to connect to these servers and I would get errors that mention "invalid argument" and "unexpected status code 500." I'm attaching the error log as well as the access log.
Hi @mintopia
I was having an issue with generic image (blizzard client not updating), and then i noticed i was running sniproxy with generic image which obviously caused a conflict since they both use 443 TCP port.
Can i run lancache without listening to 443 port? AFAIK the sniproxy should be used to redirect https traffic to avoid the MITM scenario.
Please advise,
Hi,
Im running ubunto 18.04 desktop edition on vmWARE(using windows os)with a 100 GB given space.
when i run this code:
sudo docker run
--restart unless-stopped -d
--name steam-cache
-v /home/user/cache/steam/data:/data/cache
-v /home/user/cache/steam/logs:/data/logs
-p 192.168.1.50:80:80
steamcache/generic:latest
4fe2ade4980c: Pull complete
e11b023b07a0: Pull complete
aaf7d5908de5: Pull complete
8ae3a5ca453f: Pull complete
b494cbfdaa4c: Pull complete
489e1e313017: Pull complete
Digest: sha256:04d20e341efbe344dc8cda6dea56e90a8801cc1194277d0afb289879943ea4ef
Status: Downloaded newer image for steamcache/generic:latest
4660032cc496b4dd3cfac2753ebf7c38c068bb4262324b7e6a667c641f34b35f
docker: Error response from daemon: driver failed programming external connectivity on endpoint steam-cachee (921438246a6a4ba7fc0233335883bde17af1c1b30c186f3478f7108ab4166d1f): Error starting userland proxy: listen tcp 192.168.1.50:80: bind: cannot assign requested address.
user is my 'username'
Please help ! 👍
trying to set up a multi-purpose cache system with each having its own IP address so that later down the track I can just assign said IP addresses to more NIC and not have to play with docker anymore.
I have steamcache-dns working fins and so is sliproxy, but steamcache and the generic version give me permissions issues when they start up and in portainer they lost their internal IP addresses after about 30 seconds
I have tried setting all the directories to both nginx and dockerroot and it was root by default...
the only way I have managed to get around the issue myself has been to manually recreate all the containers myself in portainer with the same settings that were given by your page.
tbh: I'm not sure if it works yet, but I was just wondering what I was doing wrong for the cli lines to not work.
thanks
While downloading League of legends it downloads until a certain point and than gives a error 004.
When i delete the DNS it downloads again.
Hi, I'm preparing my cache downloading all games but, after few hours of download, Steam stops the download. When analysing the cache-dns log I find many timeout
messages but I can't figure it out why.
here is a part of the log
2018/06/27 16:25:46 [error] 13#13: *315735 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:25:51 [error] 13#13: *315742 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:25:56 [error] 13#13: *315746 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:01 [error] 13#13: *315753 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:06 [error] 13#13: *315758 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:11 [error] 13#13: *315765 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:16 [error] 13#13: *315770 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:21 [error] 13#13: *315777 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:26 [error] 13#13: *315781 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:31 [error] 13#13: *315788 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:36 [error] 13#13: *315793 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:41 [error] 13#13: *315800 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:46 [error] 13#13: *315808 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:51 [error] 13#13: *315815 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:26:56 [error] 13#13: *315822 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:01 [error] 13#13: *315827 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:06 [error] 13#13: *315835 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:11 [error] 13#13: *315841 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:16 [error] 13#13: *315849 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:21 [error] 13#13: *315854 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:26 [error] 13#13: *315860 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:31 [error] 13#13: *315865 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:36 [error] 13#13: *315871 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:41 [error] 13#13: *315874 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:46 [error] 13#13: *315882 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:51 [error] 13#13: *315885 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:27:56 [error] 13#13: *315892 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:28:01 [error] 13#13: *315897 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 16:28:06 [error] 13#13: *315903 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.0", upstream: "http://10.0.0.101:80/", host: "10.0.0.101", referrer: "http://10.0.0.101/"
2018/06/27 17:23:15 [error] 13#13: *319046 unexpected status code 200 in slice response while reading response header from upstream, client: 10.0.0.5, server: , request: "GET /depot/578081/chunk/808d535edacfe106ed80e0f5bbf75569f682ea7b?l=14&e=1530633594&sid=1095275&h=1add064222793b11f97738341bfae1530ab22bb1 HTTP/1.1", subrequest: "/depot/578081/chunk/808d535edacfe106ed80e0f5bbf75569f682ea7b", upstream: "http://146.66.155.12:80/depot/578081/chunk/808d535edacfe106ed80e0f5bbf75569f682ea7b?l=14&e=1530633594&sid=1095275&h=1add064222793b11f97738341bfae1530ab22bb1", host: "valve1202.steamcontent.com"
2018/06/28 07:38:01 [error] 14#14: *319486 upstream timed out (110: Operation timed out) while reading upstream, client: 10.0.0.5, server: , request: "GET /depot/578081/chunk/9f0bb3ed6db976c99e2ca5517eea9a91352f1b0f?l=14&e=1530633594&sid=1095275&h=1add064222793b11f97738341bfae1530ab22bb1 HTTP/1.1", upstream: "http://146.66.155.6:80/depot/578081/chunk/9f0bb3ed6db976c99e2ca5517eea9a91352f1b0f?l=14&e=1530633594&sid=1095275&h=1add064222793b11f97738341bfae1530ab22bb1", host: "valve1206.steamcontent.com"
Can you help me please.
thanks
Hi,
It seems the the generic steamcache don't work. It doesn't cache the Hearthstone nor the Windows Update.
I use these settings:
version: '3'
services:
steamcache:
image: steamcache/generic
container_name: steamcache
restart: always
volumes:
- /cache/data:/data/cache
network_mode: bridge
ports:
- "81:80"
environment:
- CACHE_MEM_SIZE=500m
- CACHE_DISK_SIZE=100g
- CACHE_MAX_AGE=14d
steamcache-dns:
image: steamcache/steamcache-dns
container_name: steamcache-dns
depends_on:
- steamcache
restart: always
network_mode: bridge
ports:
- "54:53/udp"
environment:
- USE_GENERIC_CACHE=true
- LANCACHE_IP=192.168.0.100
- UPSTREAM_DNS=1.1.1.1
sniproxy:
image: steamcache/sniproxy
container_name: sniproxy
depends_on:
- steamcache
restart: always
network_mode: bridge
ports:
- "444:443"
The log shows:
Checking permissions (This may take a long time if the permissions are incorrect on large caches)...
Done. Starting caching server.
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Hi,
IMO this docker would be much more user friendly if it were support the HTTPS by default and wasn't required another docker container for that.
I am mainly referring to these: https://github.com/steamcache/generic#dns-entries
Sorry, not a network expert and had a bit of trouble with the docs.
I am looking to cache Steam + Battle.NET + Origin using the generic image.
I am going to attent a lanparty (12 persons) where we have to share a slow ADSL connection (6Mbit down at best).
Steamcache is running as a Docker service on an Ubuntu 16.04 installation on a I5-2500, 8GB, 2GB WD SATA harddrive.
I have steamcache and steamcache-dns running for my Xbox One downloads. The caching process is working, except I'm seeing slow download speeds on the initial download of un-cached data. I have gigabit internet from my ISP. Without caching, the download goes at 250mbps. With caching, it seems to be about 1/3 of that. Once a file is in cache, my Xbox will download it at about 250mbps once again. This result has been very consistent over the past 3 days. I understand that once the cache is primed, then the benefits become apparent. My thread here is all about the initial download though.
Read Dead Redemption 2 (80GB), for example, would take 3.5 to download through steamcache. That download would normally take under 1 hour without caching. I expected some overhead, but this seems unusual. Is this typical?
Also, I used Wireshark to inspect traffic and I don't see any obvious issues there. I saw that one of the links my Xbox was fetching was this link. I can open that in Firefox and begin downloading at 240mbps to 400mbps. That means my OS can fetch files from Xbox's servers at good speeds.
So... fetching is fast. Serving cached data is fast. But downloading fresh data is very slow.
steamcache was setup using this command:
sudo docker run --restart unless-stopped --name gamecache -v /mnt/gamecache:/data/cache -v /mnt/gamecache:/data/logs -p 192.168.1.223:80:80 steamcache/generic:latest
I am running this from a virtual machine. The VM is running Ubuntu 18 x64. Host is Windows 10. CPU i7 7700HQ - 4 cores at 2.8ghz. 16GB RAM. The VM has use of 2 cores and 4GB of RAM (will be more in production later). The VM and the cache/data folder are stored on an external SSD. I experimented with running Ubuntu natively but I didn't see a difference in speeds.
Here's how my system resources look while downloading:
Disk Z is where the VM and game cache are stored. At this same time, the Xbox itself reports about 55mbps.
sudo docker run --name steamcache-dns -p 192.168.1.223:53:53/udp -e STEAMCACHE_IP=192.168.1.223 -e USE_GENERIC_CACHE=true -e LANCACHE_IP=192.168.1.223 -e UPSTREAM_DNS=8.8.8.8 steamcache/steamcache-dns:latest
Not sure how to get this. When I run the docker commands at first, I see output from the DNS setup command, but I'm not sure how to get back to it after a reboot.
In the Steamcache guide on ARS Technica (https://arstechnica.com/gaming/2017/01/building-a-local-steam-caching-server-to-ease-the-bandwidth-blues/) there is a script on how to check if the cache functions properly.
sudo docker exec steamcache /scripts/watchlog.sh
Is a similar script available to check on the cache function in the Generic cache?
I previously had mine working great on a raspberrypi using a NAS as storage, pushing full 100Mbit
I moved to a VM setup now, and i cant get the generic cache nor the sniproxy to act right.
My Blizzard docker steamcache/generic gets me the below in error.log
2018/07/18 03:56:59 [error] 19#19: *172 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 172.17.0.1, server: , request: "GET /tpr/bnt001/config/b5/49/b549cde7f3a27e4aed8b09f3661dfa41 HTTP/1.0", upstream: "http://10.0.0.83:80/tpr/bnt001/config/b5/49/b549cde7f3a27e4aed8b09f3661dfa41", host: "level3.blizzard.com"
I've got a steamcache-dns, steamcache and steamcache-generic docker setup going, and it's working really well for everything except Battle.net.
I have a 100/20 connection, but with caching enabled, I only get 2-3mbps from Blizzard's servers, and I'm not sure why. I also get intermittent disconnections while downloading from Battle.net.
Just wondering if anyone else is also seeing this?
Using generic now, defined al zones for steam and other platform direct at my sophos firewall to route to the steamcache:generic.
Works fine at start. Now 2 Probem:
Any suggestions on this probs?
Hello,
I dont have any experience with docker and steamcache.
After reading all differents threats, I install docker and start to pull steamcache/generic + steamcache-dns on my ubuntu cache server.
I type this command and hit enter:
docker run --restart=always --name cache-steam -v /cache/steam/data:/data/cache -v /cache/steam/logs:/data/logs -p 192.168.1.104:80:80 steamcache/generic:latest
I got the prompt:
nginx: the configuration file /etc/nginx.nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is sucessful
after the last line, I'm stuck inside the prompt. I tried to hit Ctrl + P, Ctrl + Q, Ctrl PQ, Ctrl + C, i'm still stuck inside. any idea how to deal with it ?
If someone had a white page how to proceed in every step pls.
THank you
It seems the majority of the requests to steam miss because steam responds in HTTP 302 most of the time (which, in code, isn't cached).
Is there anything I can do to fix this on my end to make this more useful? Caching for blizzard works wonderfully, but that is mostly HTTP 200.
hey there caching is working awesome but it will cache all the things on 80 http not just steam can someone help me to get rid of
We're using the the Generic Steamcache in our environment, and we've run into an issue this morning where the steam client is failing to download. Took a peek in the error log, and this is what I am seeing:
2018/03/06 16:10:33 [error] 10#10: *95612 unexpected status code 302 in slice response while reading response header from upstream, client: 10.16.10.158, server: , request: "GET /client/bins_webhelpers_win32.zip.vz.c05f07ba6a52429edb32cd7b40e04d5f638f4697_1095795 HTTP/1.1", subrequest: "/client/bins_webhelpers_win32.zip.vz.c05f07ba6a52429edb32cd7b40e04d5f638f4697_1095795", upstream: "http://162.254.192.11:80/client/bins_webhelpers_win32.zip.vz.c05f07ba6a52429edb32cd7b40e04d5f638f4697_1095795", host: "client-download.steampowered.com"
Any ideas on how I could mitigate this? My research on this showed that I need to increase the proxy buffer size in nginx, which I can totally do, but it means modifying the container.
As an added note, the error the user gets is as follows:
"Steam needs to be online to update. Please confirm your network connection and try again"
Hey guys,
at first -> cool project!
But i have a problem with loggin in to steam. It hangs at "Connection with Steam-Account ***"
In the log of the Docker-Container 'steam-cache' there are many logs of the ngnix with MISS:
10.10.10.27 / - - - [23/Aug/2018:20:10:07 +0000] "GET /depot/252494/chunk/129705e028733989abfdef19c458c9e83261cdbd?l=14&e=1535659709&h=535e57032c1551bafbfc220656bc0796 HTTP/1.1" 200 316464 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cdn.highwinds.cs.steampowered.com" "-" 10.10.10.27 / - - - [23/Aug/2018:20:10:07 +0000] "GET /depot/252494/chunk/bd1a68c7a0a939fcebe3c03c05240669320a611a?l=14&e=1535659709&h=535e57032c1551bafbfc220656bc0796 HTTP/1.1" 200 465264 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cdn.highwinds.cs.steampowered.com" "-" 10.10.10.27 / - - - [23/Aug/2018:20:10:07 +0000] "GET /depot/252494/chunk/a1fa4bf66bb639230f1799f0fe966ba1fa52dc3d?l=14&e=1535659709&h=535e57032c1551bafbfc220656bc0796 HTTP/1.1" 200 27593 "-" "Valve/Steam HTTP Client 1.0" "MISS" "cdn.highwinds.cs.steampowered.com" "-"
Maybe i am not alone with this problem?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.