Code Monkey home page Code Monkey logo

Comments (11)

ggnaegi avatar ggnaegi commented on June 9, 2024 2

@ggnaegi and @RaynaldM
Any ideas on how to monitor such cases and memory management?
Seems we need definitely to design some lite memory controller/monitor...

Metrics?

from ocelot.

RaynaldM avatar RaynaldM commented on June 9, 2024 2

As you can see, memory consumption varies greatly from pod to pod, due to the diversity of operations to be carried out.

image

It varies from 150 MB to 1.2 GB.

from ocelot.

RaynaldM avatar RaynaldM commented on June 9, 2024 1

Our experience with metrics is currently limited to measuring response times, the number of active requests (requests but not yet responses), the number of requests per second and error counts.
As far as OOMs are concerned, we see them in the logs, but with a slight delay, and it's quite difficult to trace the thread of events that generated them. In many cases, an OOM is triggered on a method, but it is merely the victim of a leak elsewhere.
And to top it all off, OOMs happen quite randomly.
I think it's better to concentrate on improving Ocelot gently and punctually, whenever we recognize a problematic piece of code (and there really are lots of places where we can improve it).

from ocelot.

raman-m avatar raman-m commented on June 9, 2024

For more lucky hunting I would say you could add more logger points...
I agree OOM problems cannot be detected easily now because Ocelot has no memory consuming monitor, indicators and memory events...

from ocelot.

raman-m avatar raman-m commented on June 9, 2024

@ggnaegi and @RaynaldM
Any ideas on how to monitor such cases and memory management?
Seems we need definitely to design some lite memory controller/monitor...

from ocelot.

raman-m avatar raman-m commented on June 9, 2024

Does our lovely Bla-bla gateway have some metrics/indicators or monitoring features?
We could start from general metrics for memory consumption, but I'm afraid that will be not easy... Seems it requires a refactoring of all Ocelot core... 🤔
I believe for the first time we can read consumed memory on app domain level... That should be enough...
More precise metrics/indicators require big refactoring road/milestone...

from ocelot.

ks1990cn avatar ks1990cn commented on June 9, 2024

@RaynaldM Intresting, how you are measuring these graphs? Is it something on Production with azure/aws, which tool it is?

What about just finding through visual studio profilers, is it effective enough (Cause it will not be able to replicate high traffic cases)?

Just gone through statement and trying to understand, what approach you are following.

https://learn.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2022

from ocelot.

ks1990cn avatar ks1990cn commented on June 9, 2024
image

Here we can see from 50s to 1:40min we got 2 times allocations, one before GC and after GC. This is done with 100 Virtual Users load for 5 mins.
We can through VS revisit our codes where it makes more allocations?

from ocelot.

ks1990cn avatar ks1990cn commented on June 9, 2024

.Net 8 have new feature for realtime GC monitoring, .Net Aspire.

https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview
https://www.youtube.com/watch?v=DORZA_S7f9w
https://github.com/dotnet/eShop

Can we give user option to integrate?

from ocelot.

raman-m avatar raman-m commented on June 9, 2024

But the hunt is not finish... 😃

@RaynaldM @ggnaegi Do we need to hunt more?

from ocelot.

ggnaegi avatar ggnaegi commented on June 9, 2024

@raman-m @RaynaldM We should try to improve the overall application performance, but the last changes had a big impact on OOMs. I think this issue isn't specific enough, maybe we should convert it to a discussion?

from ocelot.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.