Comments (11)
@ggnaegi and @RaynaldM
Any ideas on how to monitor such cases and memory management?
Seems we need definitely to design some lite memory controller/monitor...
Metrics?
from ocelot.
As you can see, memory consumption varies greatly from pod to pod, due to the diversity of operations to be carried out.
It varies from 150 MB to 1.2 GB.
from ocelot.
Our experience with metrics is currently limited to measuring response times, the number of active requests (requests but not yet responses), the number of requests per second and error counts.
As far as OOMs are concerned, we see them in the logs, but with a slight delay, and it's quite difficult to trace the thread of events that generated them. In many cases, an OOM is triggered on a method, but it is merely the victim of a leak elsewhere.
And to top it all off, OOMs happen quite randomly.
I think it's better to concentrate on improving Ocelot gently and punctually, whenever we recognize a problematic piece of code (and there really are lots of places where we can improve it).
from ocelot.
For more lucky hunting I would say you could add more logger points...
I agree OOM problems cannot be detected easily now because Ocelot has no memory consuming monitor, indicators and memory events...
from ocelot.
@ggnaegi and @RaynaldM
Any ideas on how to monitor such cases and memory management?
Seems we need definitely to design some lite memory controller/monitor...
from ocelot.
Does our lovely Bla-bla gateway have some metrics/indicators or monitoring features?
We could start from general metrics for memory consumption, but I'm afraid that will be not easy... Seems it requires a refactoring of all Ocelot core... 🤔
I believe for the first time we can read consumed memory on app domain level... That should be enough...
More precise metrics/indicators require big refactoring road/milestone...
from ocelot.
@RaynaldM Intresting, how you are measuring these graphs? Is it something on Production with azure/aws, which tool it is?
What about just finding through visual studio profilers, is it effective enough (Cause it will not be able to replicate high traffic cases)?
Just gone through statement and trying to understand, what approach you are following.
https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2022
from ocelot.
Here we can see from 50s to 1:40min we got 2 times allocations, one before GC and after GC. This is done with 100 Virtual Users load for 5 mins.
We can through VS revisit our codes where it makes more allocations?
from ocelot.
.Net 8 have new feature for realtime GC monitoring, .Net Aspire
.
https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview
https://www.youtube.com/watch?v=DORZA_S7f9w
https://github.com/dotnet/eShop
Can we give user option to integrate?
from ocelot.
But the hunt is not finish... 😃
@RaynaldM @ggnaegi Do we need to hunt more?
from ocelot.
@raman-m @RaynaldM We should try to improve the overall application performance, but the last changes had a big impact on OOMs. I think this issue isn't specific enough, maybe we should convert it to a discussion?
from ocelot.
Related Issues (20)
- More open customization of Polly use HOT 2
- How to work with multiple configuration files per microservice? HOT 1
- sockect seems like not reuse on cenos or docker but reuse in window(iis) HOT 2
- Default timeout of HttpClient via route configuration HOT 14
- File-scoped namespace HOT 2
- Use Polly v8 syntax HOT 4
- Authentication docs could do with a note around .NET 8 changes HOT 1
- The Getting Started document could be updated for .NET 8 HOT 5
- Use Ocelot / Identity Server4 JwtBearerEvents.OnAuthenticationFailed addHeaders not work HOT 9
- Apply rate limiting globally HOT 5
- Download file API causes Gateway memory to rise by fileSize HOT 15
- UpstreamHost not working HOT 8
- Routes containing attribute "UpstreamHost " cannot be found HOT 1
- Issue with Aggregator in Ocelot: Requires At Least Two Routes to Execute HOT 11
- 在使用异步流返回时,并不是实时刷新逐条返回结果,而是一次返回结果 HOT 10
- How to Authorize with nested JWT claim generated from Keycloak
- Access to the path '/app/ocelot.json' is denied HOT 1
- Ocelot is missing NuGet package README file HOT 3
- Kubernetes provider should identify different ports HOT 14
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ocelot.