Comments (11)
@ggnaegi and @RaynaldM
Any ideas on how to monitor such cases and memory management?
Seems we need definitely to design some lite memory controller/monitor...
Metrics?
from ocelot.
As you can see, memory consumption varies greatly from pod to pod, due to the diversity of operations to be carried out.
It varies from 150 MB to 1.2 GB.
from ocelot.
Our experience with metrics is currently limited to measuring response times, the number of active requests (requests but not yet responses), the number of requests per second and error counts.
As far as OOMs are concerned, we see them in the logs, but with a slight delay, and it's quite difficult to trace the thread of events that generated them. In many cases, an OOM is triggered on a method, but it is merely the victim of a leak elsewhere.
And to top it all off, OOMs happen quite randomly.
I think it's better to concentrate on improving Ocelot gently and punctually, whenever we recognize a problematic piece of code (and there really are lots of places where we can improve it).
from ocelot.
For more lucky hunting I would say you could add more logger points...
I agree OOM problems cannot be detected easily now because Ocelot has no memory consuming monitor, indicators and memory events...
from ocelot.
@ggnaegi and @RaynaldM
Any ideas on how to monitor such cases and memory management?
Seems we need definitely to design some lite memory controller/monitor...
from ocelot.
Does our lovely Bla-bla gateway have some metrics/indicators or monitoring features?
We could start from general metrics for memory consumption, but I'm afraid that will be not easy... Seems it requires a refactoring of all Ocelot core... 🤔
I believe for the first time we can read consumed memory on app domain level... That should be enough...
More precise metrics/indicators require big refactoring road/milestone...
from ocelot.
@RaynaldM Intresting, how you are measuring these graphs? Is it something on Production with azure/aws, which tool it is?
What about just finding through visual studio profilers, is it effective enough (Cause it will not be able to replicate high traffic cases)?
Just gone through statement and trying to understand, what approach you are following.
https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2022
from ocelot.
![image](https://private-user-images.githubusercontent.com/29522704/290968490-0e2d8781-9852-4050-879a-e9086e78b479.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTc5MzE3MjEsIm5iZiI6MTcxNzkzMTQyMSwicGF0aCI6Ii8yOTUyMjcwNC8yOTA5Njg0OTAtMGUyZDg3ODEtOTg1Mi00MDUwLTg3OWEtZTkwODZlNzhiNDc5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjA5VDExMTAyMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJjODVmZWQzZmY3ZWQ5NjcxZjk1MzgwNTIzM2VlMjFhNzFjZWU1YTU2NWM4MWQwMWRhNWFjZDRkNWUyOTNmZTAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.9nzcA_2j-r8y3IT_hsihSOB8kpHQM6oG_dIL52_XvhI)
Here we can see from 50s to 1:40min we got 2 times allocations, one before GC and after GC. This is done with 100 Virtual Users load for 5 mins.
We can through VS revisit our codes where it makes more allocations?
from ocelot.
.Net 8 have new feature for realtime GC monitoring, .Net Aspire
.
https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview
https://www.youtube.com/watch?v=DORZA_S7f9w
https://github.com/dotnet/eShop
Can we give user option to integrate?
from ocelot.
But the hunt is not finish... 😃
@RaynaldM @ggnaegi Do we need to hunt more?
from ocelot.
@raman-m @RaynaldM We should try to improve the overall application performance, but the last changes had a big impact on OOMs. I think this issue isn't specific enough, maybe we should convert it to a discussion?
from ocelot.
Related Issues (20)
- Long duration of CircleCI builds HOT 1
- Map response of rate limit quota into exception
- On the fly `ocelot.json` configuration merging HOT 1
- Body cannot be forwarded twice on Aggregator HOT 17
- 当下游服务返回"text/plain"类型时导致"response.Body"中变得异常得长,这正常吗?
- Resolving 'IsAuthenticated' False Issue with Ocelot API Gateway and OKTA Authentication
- `FileCacheOptions` not working after the header was introduced in FileCache settings in version 23.0.0 HOT 4
- Receiving 401 depending on the order of my API Route, when calling API's through Ocelot API Gateway
- Getting load balancer error with latest version HOT 6
- Incorrect routing when the query parameter is in the configuration
- Unusual spike in response with 499 status code HOT 24
- Downstream route is not allowed to end on a forward slash HOT 3
- 404 exception caused by QoS configuration item conflict HOT 10
- In C#, when using Ocelot, the ContentLength is 0 when forwarding data to downstream services. HOT 1
- Polly V7 syntax is no longer supported
- `FailureRatio` and `SamplingDuration` parameters of Polly V8 circuit-breaker
- After upgrade to Ocelot 23.2, `ocelot.global.json` file no longer gets merged correctly HOT 7
- Circuit Breaker behavior in Polly v7 vs v8 HOT 1
- Request Body Not Considered in Ocelot Cache Key Calculation HOT 7
- NuGet issues aka `PublishToNuget` release target task HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ocelot.