Comments (11)
lucaslorentz
commented on August 24, 2024
The restarting container is probably causing a lot of docker events, and that triggers Caddyfile updates.
We do have a 100ms throttling to avoid redundant updates based on events, but 100ms is probably too short to prevent high CPU. I will make the 100ms configurable, so you can increase it to 30sec, or maybe even a minute.
from caddy-docker-proxy.
lucaslorentz
commented on August 24, 2024
Merged a fix, can you please try setting CADDY_DOCKER_EVENT_THROTTLE_INTERVAL or --event-throttle-interval duration to something like 30s?
This should prevent the overhead by containers in start/stop loop.
The caveat is that Caddyfile updates from events will be delayed by 30s as well.
from caddy-docker-proxy.
teodorescuserban
commented on August 24, 2024
Hello, thank you very much for your quick addition!
I am now running caddy 2.7.4 with caddy-docker-proxy 2.8.8.
I am not able to set it to 30 seconds, it seems a bit much, for now I am on 15 seconds.
The cpu is still going quite high. It does jump quite high even without a container restarting, but with a container restarting in a loop the cpu remains steady high.
I grabbed 180 seconds profiles with:
- no restart loop, 5 second throttle
- restart loop, 5 second throttle
- restart loop, 15 second throttle
Attaching them below, I hope it provides some pointers. cc @francislavoie in case this is not actually a caddy-docker-proxy issue or if he has an idea.
from caddy-docker-proxy.
lucaslorentz
commented on August 24, 2024
I didn't find anything CDP-specific in the recorded profiles.
Looks like the overhead is coming from Caddy itself, and GC is a big portion of it.
I wonder if we can optimize a bit the parsing and regex bits in Caddy.
from caddy-docker-proxy.
francislavoie
commented on August 24, 2024
Without seeing a config I don't know what to suggest.
from caddy-docker-proxy.
teodorescuserban
commented on August 24, 2024
@francislavoie would you rather close this one and have me post to caddy community instead?
from caddy-docker-proxy.
teodorescuserban
commented on August 24, 2024
@francislavoie, here are the full configs
/etc/caddy/Caddyfile:
{
# debug
log {
output file /logs/main.log
level WARN
format json {
time_format iso8601
}
}
}
import conf.d/*
:80 {
header Server "{$MY_HOSTNAME}"
import logs access.log
redir https://{$MY_HOSTNAME}/ 301
}
{$MY_HOSTNAME} {
header Content-Type "text/html; charset=utf-8"
respond "<html><head><title>Oops</title></head><body><p>This is not the website you are looking for.</body></html>"
import headers headers_overwrite
import logs access.log
}
import vhost.d/*
/etc/caddy/conf.d/snippets:
(auth) {
@check_auth expression `"{args[0]}" == "auth_enabled"`
@check_auth_except `"{args[0]}" == "auth_except" && !remote_ip({args[2]})`
basicauth @check_auth {
# if a file is missing, the whole vhost wont work.
# base64 encoded bcrypt: doable from ansible yes, same resulting hash, no :(
#bob JDJhJDEwJEVCNmdaNEg2Ti5iejRMYkF3MFZhZ3VtV3E1SzBWZEZ5Q3VWc0tzOEJwZE9TaFlZdEVkZDhX
# prevent the whole vhost block to fail by adding a wildcard to the filename import
# this will only generate a warning and because the basicauth block is empty,
# if the auth is enabled no user will get in but all will work otherwise :)
import /etc/caddy/htpasswd/{args[1]}*
}
basicauth @check_auth_except {
import /etc/caddy/htpasswd/{args[1]}*
}
}
(headers) {
@check_add expression `"{args[0]}" == "headers_add" || "{args[0]}" == "headers_overwrite"`
header Server "{http.request.host}"
# not so fast
header @check_add {
# X-Frame-Options "DENY"
Strict-Transport-Security `max-age=31536000; includeSubDomains; preload; always;`
X-Content-Type-Options "nosniff"
Referrer-Policy "strict-origin-when-cross-origin"
X-Robots-Tag "noindex, nofollow, nosnippet, noarchive"
X-XSS-Protection `1; mode=block`
# X-Headers-Check "Overwritten or Added"
}
}
(logs) {
log {
output file /logs/{args[0]}
format filter {
wrap json {
time_format iso8601
}
fields {
common_log delete
request>headers>Authorization delete
}
}
}
}
(rev_proxy) {
@check_remove expression `"{args[1]}" == "headers_overwrite"`
@check_keep expression `"{args[1]}" != "headers_overwrite"`
reverse_proxy @check_remove {
to {args[0]}
header_down -Server
header_down -X-Hudson
header_down -X-Jenkins
# header_down -X-Frame-Options
header_down -Strict-Transport-Security
header_down -X-Content-Type-Options
header_down -Referrer-Policy
header_down -X-Robots-Tag
header_down -X-XSS-Protection
# header_down X-Headers-Check "Upstream Removed"
}
reverse_proxy @check_keep {
to {args[0]}
header_down -Server
header_down -X-Hudson
header_down -X-Jenkins
# header_down X-Headers-Check "Upstream Kept"
}
}
/etc/caddy/conf.d/templates:
#caddy.0_import: "tmpl {{upstreams 80}} {{.Labels.caddy}} headers_overwrite auth_enabled"
#caddy.0_import: "tmpl {{upstreams 80}} {{.Labels.caddy}} headers_add auth_enabled"
#caddy.0_import: "tmpl {{upstreams 80}} {{.Labels.caddy}} headers_no auth_noop"
(tmpl) {
import rev_proxy {args[0]} {args[2]}
import headers {args[2]}
import auth {args[3]} {args[1]} `"10.1.1.1/32"`
import logs access.log
}
# caddy.import: tmpl_except {{index .Names 0}} generic.small headers_overwrite
(tmpl_except) {
import rev_proxy {args[0]} {args[2]}
import headers {args[2]}
import auth auth_except {args[1]} `"192.168.1.1/32", "10.1.1.1/32"`
import logs access.log
}
# caddy.import: tmpl_except_these {{index .Names 0}} generic.small headers_overwrite `"192.168.1.1/32", "10.1.1.1/32"`
(tmpl_except_these) {
import rev_proxy {args[0]} {args[2]}
import headers {args[2]}
import auth auth_except {args[1]} {args[3]}
import logs access.log
}
(tmpl_no_auth) {
import rev_proxy {args[0]} {args[2]}
import headers {args[2]}
import logs access.log
}
(tmpl_drupal) {
import rev_proxy {args[0]} {args[2]}
import headers {args[2]}
import auth auth_enabled generic.drupal `"10.1.1.1/32"`
import logs access.log
}
# caddy.import: tmpl_drupal_except {{index .Names 0}} `{{.Labels.caddy}}` headers_overwrite
(tmpl_drupal_except) {
import rev_proxy {args[0]} {args[2]}
import headers {args[2]}
import auth auth_except generic.drupal `"192.168.1.1/32", "10.1.1.1/32"`
import logs access.log
}
# caddy.import: tmpl_drupal_except {{index .Names 0}} `{{.Labels.caddy}}` headers_overwrite `"192.168.1.1/32", "10.1.1.1/32"`
(tmpl_drupal_except_these) {
import rev_proxy {args[0]} {args[2]}
import headers {args[2]}
import auth auth_except generic.drupal {args[3]}
import logs access.log
}
/etc/caddy/vhost.d/example:
host1.example.local {
import rev_proxy host1:80 headers_overwrite
import headers headers_overwrite
import logs access.log
}
The caddy-docker-proxy labels:
caddy: host2.example.local
caddy.import: tmpl {{index .Names 0}} generic.small headers_noop auth_enabled
or
caddy: host3.example.local
caddy.import: tmpl_drupal {{index .Names 0}} `{{.Labels.caddy}}` headers_overwrite
from caddy-docker-proxy.
teodorescuserban
commented on August 24, 2024
The upgrade (both caddy, from 2.6.1 to 2.7.4 and caddy-docker-proxy from 2.7.1 to 2.8.8) did seem to improve things.
See cpu graphs below. The cpu limit is at 2 (Y axis). Where the cpu average goes down (starting 13th around 00:00 UTC).
The average is still around 40% of a CPU core which seems pretty high.
I would appreciate any ideas here, @francislavoie
from caddy-docker-proxy.
francislavoie
commented on August 24, 2024
Ah, running an older version would explain the CPU usage on reload. basicauth did perform hashing on config reload which would be quite expensive, especially if the directive is used multiple times (one hash per directive usage). We've since fixed that a while back to avoid doing that by hard-coding a dummy hash.
You don't need common_log delete nor request>headers>Authorization delete anymore, the former is never logged anymore and the latter is hidden by default.
If you still have a lot of HTTP requests hitting sites with basicauth enabled, then that would explain the CPU usage; basicauth hashing is very CPU expensive, especially with the default bcrypt cost of 14 (which IMO is way too high, but it is secure). You could try replacing the hashes with lower cost ones to see if it helps. Unfortunately the Caddy command doesn't support specifying the cost by you can make a hash with any other bcrypt tool with a lower cost like 10-12 and see if it makes an obvious difference.
Other than that, your config looks fine I guess? But with the amount of expansion it has going on it's hard to really get a solid sense for how expensive it is overall. More profiles would be helpful to get a sense of where it's spending time, but that's not an area of Go debugging I have much experience in.
from caddy-docker-proxy.
teodorescuserban
commented on August 24, 2024
Alright, I'll try making the bcrypt less expensive and get rid of common_log and Auth delete!
Will report back and then close the ticket.
Thank you very much, @francislavoie and @lucaslorentz for your time and the awesome work you're doing!
from caddy-docker-proxy.
teodorescuserban
commented on August 24, 2024
It looks that it works "good enough" to shift the priorities 😞 . Closing this ticket, but I'll make sure to provide feedback to the last suggestion regarding the config optimisation.
from caddy-docker-proxy.
Related Issues (20)
- Dynamically expose additional ports without restarting Caddy or target services HOT 21
- Container tag vs build version mismatch? 2.8-alpine reports v2.7.5 HOT 2
- Removing invalid block in Caddyfile HOT 5
- Trying to use the LinuxServer Nextcloud Docker returns public IPs. HOT 5
- Same domain in different swarm services of a stack HOT 5
- Upstreams macro not inserting container IP into caddy config HOT 3
- Containerizing a "Host" network docker image HOT 2
- Is there support for HAProxy/Proxy Protocol in CDP HOT 2
- Support IPv6-only containers
- risc-v releases? HOT 1
- Caddy Docker proxy + Authelia + Cloudflared (Argo Tunnel)
- Caddy Docker Proxy + Authelia + (Cloudflare Argo Tunnel) HOT 5
- Empty label caddy.reverse_proxy.transport.tls_insecure_skip_verify not expanding correctly HOT 1
- Caddyfile defaults HOT 1
- Cannot compile HOT 2
- [Q] Moving from plain Caddy HOT 3
- Use the same wildcard certificate for multiple sub domains HOT 4
- Reverse proxy doesn't work when using internal network HOT 1
- stalling uploads for proxy over LAN, cannot set proxy_protocol timeout HOT 4
- Creating reverse proxy only works via Caddyfile, not labels HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from caddy-docker-proxy.