Comments (17)
Hey @wmashal , I also work with istio and I think it's not related. It looks like slowness while reaching the Kubernetes API/Metrics Server from the controller itself. Could you maybe reduce the number of concurrentWorkers to 1? That will make less API calls to the kubelet and might solve the issue.
from kube-reqsizer.
Also make sure your Prometheus instance scrapes Kube-Reqsizer pod at the correct service/port
from kube-reqsizer.
I would also note it's best to disable istio sidecar for the controller itself
from kube-reqsizer.
from kube-reqsizer.
@wmashal did you enable annotations on relevant pods/namespaces? I seems like you did it on deny-all mode where you annotate what you want to resize one by one
from kube-reqsizer.
@wmashal reqsizer.jatalocks.github.io/optimize=true
on namespace or deployment you want.
the rest you can leave default
from kube-reqsizer.
@wmashal the offset means how much CPU or Memory request it removed/increased over time. For example if you see the offset is minus (going down) you can say that Reqsizer noticed your pods request more than they need and reqsizer made it less
from kube-reqsizer.
Hey @wmashal it works like this:
log.Info(fmt.Sprint(c.Name, " Comparing CPU: ", fmt.Sprintf("%dm", AverageUsageCPU), " <> ", fmt.Sprintf("%dm", currentC.CPU)))
log.Info(fmt.Sprint(c.Name, " Comparing Memory: ", fmt.Sprintf("%dMi", AverageUsageMemory), " <> ", fmt.Sprintf("%dMi", currentC.Memory)))
On the left side it prints average usage, and on the right side the current requested resources it has in its manifest. It shows you both numbers in order to see what numbers the reqsizer found when it compares
from kube-reqsizer.
@wmashal hey,
That depends on your logging solution. I haven't gone too far with this project to add good enough logging, I'm afraid. If you want, contribution is always welcome from you or your teammates 🙏
from kube-reqsizer.
@jatalocks I did the following
- disable the istio inject in the reqsizer controller so there is no sidecar
- change the conncurrent workers to 1
now I see the controllers logs like this
But still the metrics count is 0 , should I wait ?
from kube-reqsizer.
@jatalocks Also I note this in the redis logs
from kube-reqsizer.
@jatalocks I should add the following on each namespace or pod deployment ?
reqsizer.jatalocks.github.io/optimize=false # Ignore Pod/Namespace when optimizing entire cluster
reqsizer.jatalocks.github.io/mode=average # Default Mode. Optimizes based on average. If ommited, mode is average
reqsizer.jatalocks.github.io/mode=max # Sets the request to the MAXIMUM of all sample points
reqsizer.jatalocks.github.io/mode=min # Sets the request to the MINIMUM of all sample points
from kube-reqsizer.
@jatalocks I can see now the cash metric count increased but what the best way to simulate kube_reqsizer_memory_offset and kube_reqsizer_cpu_offset for my demo purpose in my job, in other words I wana show how you tool make the difference and increase or decrease the resources
from kube-reqsizer.
@jatalocks Great I can see changes in the logs , can you discuss what we see here in few words , if I understand I saved some memory (the minus) and cpu also in logs if I see
resource-automation-operator Comparing CPU: 10m <> 8m that mean its increased to 10m or decreased to 8m ?
from kube-reqsizer.
@amitai-devops Thank you I see now
Is there any graphs I can show all these logs catches prints or maybe in the redis ?
from kube-reqsizer.
@jatalocks yes sure let me think about it and see how we can see all this in elegant way ;-)
from kube-reqsizer.
@wmashal thanks! Now that you tested it, its best to increase sampleSize so the results are more accurate. I'm closing this issue.
from kube-reqsizer.
Related Issues (20)
- Add a factor parameter to multiply requests
- Export prometheus metrics about applied request changes
- Add support for Custom CRDs like Argo Rollouts
- Hot reload configuration from configmap without needing pod restart HOT 2
- BUG: min-seconds is flaky and not working properly
- Patch ArgoCD applications to exclude requests HOT 1
- Re-Calculate HPAs after reconciliation to keep min-max pod ratio of HPA
- "Learning period" for reqsized pods HOT 1
- Parameters for upsizing/downsizing minimum percentage difference before scale
- Optimization is not working - Azure AKS - v1.25.6 HOT 19
- Reconcile Errors HOT 3
- Latency Metric HOT 2
- Error when trying to get stats of pods that are not in a running state
- Not updating deployment resources requests HOT 1
- Persistent Cache for Metrics history
- Ability to fetch Metrics from External Provider
- Controller Pod Theoretically becomes OOM after a long time
- Reconciliation loop too slow
- Add Go Tests HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kube-reqsizer.