Comments (8)
In the profiles I've looked at, the major bottleneck was not gRPC, but rather the (de)serialization of protobuf messages. Arrow uses flatbuffers with some additional optimizations for zero-copy data transfer. We do have to check whether these optimizations are available in the Go library though, but zero-copy is one of the key selling points of the format so there's a good chance it's already done. I have a testing branch which adds an arrow API for the Series call, but I haven't had a chance to move it forward: main...fpetkovski:thanos:series-arrow.
I think we can also experiment with various approaches for the internal replication calls, since those are self contained and can be tested quickly.
from thanos.
vtprotobuf only provides the ability to use pooling on one side. I have noted that on the serialization side, it is completely unsafe to use pooling due to gRPC: planetscale/vtprotobuf#20. :)
from thanos.
There's some recent movement though apparently grpc/grpc-go#6619 so maybe this year this will be fixed in the Go gRPC implementation 😄
This service demonstrated the benefit of this approach with a very significant reduction in allocation volume (>90% fewer bytes allocated per second and a 7x reduction in allocations) and much lower CPU usage (>30% less). We offer an implementation of this proposal for context and as a conversation starter: grpc/grpc-go#6613
Nice potential gains (:
from thanos.
So my first google search for "drpc" returned the homepage of the "danville rifle and pistol club". It does not seem to be super popular at least. Thats just an observation not a technical argument against it of course. If we are considering alternatives (which we probably should) I would like to see something more mainstream though I think.
from thanos.
😄 it's https://github.com/storj/drpc. Fair point, I'm not arguing against that but I wonder what other libraries would fit the bill?
from thanos.
I have not tried yet but if we are going to look into replacements I think we should also check https://arrow.apache.org/docs/format/Flight.html; arrow seems to be pretty popular lately.
from thanos.
It is based on gRPC so I think that immediately disqualifies it because it has the same problems. Twirp also seems like it might be a valid option https://github.com/twitchtv/twirp. But it looks like it also allocates a new []byte
for each request:
https://github.com/twitchtv/twirp/blob/main/example/service.twirp.go#L441-L450
So, we would be fighting with the RPC framework from the start again :/
from thanos.
Interesting points! Definitely it's healthy to revisit gRPC vs the alternatives from time to time!
The Go implementation at least has a bunch of deprecated code, it has a HTTP/2 implementation inside, and a bunch of functionality that we don't really use. I think Thanos can be much leaner in that regard. This post outlines the problems well https://www.storj.io/blog/introducing-drpc-our-replacement-for-grpc. Even though, of course, the presentation about dRPC will talk negatively about gRPC, my experience reflects what's written there.
dRPC looks pretty nice, especially for Go (the most supported language). It's essentially what we do in Prometheus, HTTP 1.1 + protobuf, but supporting rpc elements in proto file. However, it is a niche, plus does not support other languages well -- maybe that's fine.
Plus, I wonder if we really don't have any improvements from multiplexing on single connections with HTTP/2, we do stream a lot. Benchmarks would tell for our use case (I know dRPC has some benchmarks for their use case).
The Go implementation at least has a bunch of deprecated code
What exactly deprecated code? Do you mean using old protobuf API? This seems to be changing soon: grpc/grpc-go#6919 .. and the reason for this AND bunch of deprecated methods is for us to not have a breaking changes all the time, so it is a pretty weak point to me. They do deprecate stuff and sometimes use deprecated repos in some places to keep our (downstream users) work smooth. Why this is bad?
It will be a tabula rasa for starting from scratch and not using gogoproto #4557 because gogoproto is now deeply entrenched in our codebase due to copy/paste of generated code. Thus it is tough to remove it.
It's not though to remove it. It's just a lot of places to change, but it's generally easier than people think. And yes, you lose non-pointer fields, but as I shown in my experiments with new go generators the results on v2 + e.g. vtproto are still better vs gogo proto, despite this.
It's hard/impossible to use pooling on both sender/receiver sides in gRPC because gRPC hides this stuff and doesn't allow controlling allocations. gRPC effectively "takes ownership" of whatever byte slice we pass to it.
Are you really sure it's impossible? gRPC is doing a good job on identifying and pooling for us, plus my impression was that we can totally inject our own custom pooling to any place using custom codec have you checked that? e.g https://github.com/planetscale/vtprotobuf?tab=readme-ov-file#vtprotobuf-with-grpc (see pool
option before that section).
It doesn't allow implementing string deduplication because the unmarshaling function always receives a fresh byte slice (#7067)
Same doubt as above, plus this particular problem is fixed with Remote Write 2.0 that includes interning..
from thanos.
Related Issues (20)
- alert.query-template added too much escaping to the expression HOT 2
- [receiver] high resource utilization during the time of uploading TSDB blocks HOT 8
- Ruler: Implement flag for max-source-resolution in the rule query HOT 7
- Enable Receiver to extract Tenant from a label present in incoming timeseries HOT 9
- Thanos queries taking too long HOT 2
- Subject: Thanos Compactor Fails to Delete Downsampling Data, Resulting in Disk Space Overfill HOT 12
- Can't add Thanos stores from other EKS clusters
- sidecar, stores, and query container memory spikes HOT 4
- Query frontend,store: redis config sends "default" instead of username HOT 2
- Website does not have the correct trademark disclaimer
- Store Gateway query minio for index file with illegal characters (backslash) HOT 4
- Compactor:ThanosCompactHasNotRun HOT 1
- Compactor & Store pod has Permission denied for block HOT 1
- Can the Compactor deduplicate data for Prometheus TSDB blocks stored on disk?
- REST API For update Rules HOT 4
- Thanos Compactor Fails to Delete Downsampling Data & Thanos Store failed to block
- Support for /api/v1/targets/metadata HOT 1
- offset table size exceeds 4 bytes & symbol table size exceeds 4 bytes || Thanos Compactor error HOT 1
- Broken dependency on github.com/grpc-ecosystem/go-grpc-middleware HOT 1
- compactor: Irregular compaction and downsampling HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from thanos.