datadog / pg_tracing Goto Github PK
View Code? Open in Web Editor NEWDistributed Tracing for PostgreSQL
License: MIT License
Distributed Tracing for PostgreSQL
License: MIT License
postgres=# /*traceparent='00-00000000000000000000000000000128-0000000000000128-01'*/begin; insert into foo values(1); commit;
BEGIN
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
TRAP: failed Assert("!traceid_zero(span->trace_id)"), File: "src/pg_tracing_span.c", Line: 127, PID: 556201
postgres: heikki postgres ::1(42026) INSERT(ExceptionalCondition+0xaf)[0x55900b67551f]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0xc978)[0x7f045eeb0978]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x7527)[0x7f045eeab527]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x45f2)[0x7f045eea85f2]
postgres: heikki postgres ::1(42026) INSERT(planner+0x3d)[0x55900b2d09ad]
postgres: heikki postgres ::1(42026) INSERT(pg_plan_query+0x94)[0x55900b44d424]
postgres: heikki postgres ::1(42026) INSERT(pg_plan_queries+0x139)[0x55900b44d5d9]
postgres: heikki postgres ::1(42026) INSERT(+0x71b575)[0x55900b450575]
postgres: heikki postgres ::1(42026) INSERT(PostgresMain+0x922)[0x55900b44f992]
postgres: heikki postgres ::1(42026) INSERT(+0x713063)[0x55900b448063]
postgres: heikki postgres ::1(42026) INSERT(postmaster_child_launch+0xe7)[0x55900b348867]
postgres: heikki postgres ::1(42026) INSERT(+0x61a6fa)[0x55900b34f6fa]
postgres: heikki postgres ::1(42026) INSERT(+0x617b99)[0x55900b34cb99]
postgres: heikki postgres ::1(42026) INSERT(PostmasterMain+0x18e7)[0x55900b34bd87]
postgres: heikki postgres ::1(42026) INSERT(+0x4bd7dd)[0x55900b1f27dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f045e24824a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f045e248305]
postgres: heikki postgres ::1(42026) INSERT(_start+0x21)[0x55900ae0fb21]
2024-05-07 14:14:26.263 EEST [556193] LOG: server process (PID 556201) was terminated by signal 6: Aborted
2024-05-07 14:14:26.263 EEST [556193] DETAIL: Failed process was running: insert into foo values(1);
2024-05-07 14:14:26.263 EEST [556193] LOG: terminating any other active server processes
2024-05-07 14:14:26.264 EEST [556193] LOG: all server processes terminated; reinitializing
postgres=# create table foo (i int);
CREATE TABLE
postgres=# insert into foo select g from generate_series(1, 100000) g;
INSERT 0 100000
postgres=# set pg_tracing.sample_rate = 1.0;
SET
postgres=# begin;
BEGIN
postgres=*# declare foocur CURSOR FOR SELECT * from foo;
DECLARE CURSOR
postgres=*# fetch forward 10 from foocur;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
TRAP: failed Assert("nested_level <= max_nested_level"), File: "src/pg_tracing.c", Line: 620, PID: 510625
postgres: heikki postgres [local] FETCH(ExceptionalCondition+0xaf)[0x55fb74f9651f]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x7242)[0x7f96b5563242]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x4940)[0x7f96b5560940]
postgres: heikki postgres [local] FETCH(ExecutorRun+0x45)[0x55fb74a77db5]
postgres: heikki postgres [local] FETCH(+0x720924)[0x55fb74d76924]
postgres: heikki postgres [local] FETCH(+0x7217ec)[0x55fb74d777ec]
postgres: heikki postgres [local] FETCH(PortalRunFetch+0x179)[0x55fb74d77149]
postgres: heikki postgres [local] FETCH(PerformPortalFetch+0x191)[0x55fb749d7f31]
postgres: heikki postgres [local] FETCH(standard_ProcessUtility+0x515)[0x55fb74d78b25]
/home/heikki/pgsql.master/lib/pg_tracing.so(+0x5366)[0x7f96b5561366]
postgres: heikki postgres [local] FETCH(ProcessUtility+0x132)[0x55fb74d785d2]
postgres: heikki postgres [local] FETCH(+0x721c7a)[0x55fb74d77c7a]
postgres: heikki postgres [local] FETCH(+0x720734)[0x55fb74d76734]
postgres: heikki postgres [local] FETCH(PortalRun+0x29f)[0x55fb74d7637f]
postgres: heikki postgres [local] FETCH(+0x71b727)[0x55fb74d71727]
postgres: heikki postgres [local] FETCH(PostgresMain+0x922)[0x55fb74d70992]
postgres: heikki postgres [local] FETCH(+0x713063)[0x55fb74d69063]
postgres: heikki postgres [local] FETCH(postmaster_child_launch+0xe7)[0x55fb74c69867]
postgres: heikki postgres [local] FETCH(+0x61a6fa)[0x55fb74c706fa]
postgres: heikki postgres [local] FETCH(+0x617b99)[0x55fb74c6db99]
postgres: heikki postgres [local] FETCH(PostmasterMain+0x18e7)[0x55fb74c6cd87]
postgres: heikki postgres [local] FETCH(+0x4bd7dd)[0x55fb74b137dd]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f96b484824a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f96b4848305]
postgres: heikki postgres [local] FETCH(_start+0x21)[0x55fb74730b21]
2024-05-06 20:44:22.611 EEST [510617] LOG: server process (PID 510625) was terminated by signal 6: Aborted
2024-05-06 20:44:22.611 EEST [510617] DETAIL: Failed process was running: fetch forward 10 from foocur;
2024-05-06 20:44:22.611 EEST [510617] LOG: terminating any other active server processes
$ psql postgres -c "select count(*) from pg_am; select count(*) from pg_class"
count
-------
7
(1 row)
count
-------
420
(1 row)
$ psql postgres -c "select trace_id, span_id, span_type, span_operation from pg_tracing_consume_spans"
trace_id | span_id | span_type | span_operation
----------------------------------+------------------+--------------+-----------------------------
266759934ee85fe1100bf7a07bd43421 | 1b40b22bc5da4fa8 | Planner | Planner
266759934ee85fe1100bf7a07bd43421 | 88167138ea5945e5 | Executor | ExecutorRun
266759934ee85fe1100bf7a07bd43421 | d10fe19e7702ecd0 | SeqScan | SeqScan on pg_am
266759934ee85fe1100bf7a07bd43421 | 85e37a824bf3962b | Aggregate | Aggregate
266759934ee85fe1100bf7a07bd43421 | 7ffdd4d4338ba069 | Select query | select count(*) from pg_am;
(5 rows)
There are no spans for the second query, select count(*) from pg_class
.
Currently, you need another program to poll the pg_tracing_consume_spans
view and push the spans to an OpenTelemetry Collector. See https://github.com/bonnefoa/pg-tracing-otel-forwarder. I wish the extension could connect directly to the OpenTelemetry Collector, using the OTLP protocol. That would simplify the deployment.
This was discussed on pgsql-hackers: https://www.postgresql.org/message-id/CAO6_Xqr5cNC-fK7kX4Pt9LkYSNjuZQjDfxbm_ckypM9LD8PT1Q%40mail.gmail.com. A pull model made sense for a patch for core PostgreSQL, to avoid having a dependency on HTTP or gRPC or other heavy libraries. But as an extension, that's less of an issue.
One approach is to implement the simplest OTLP/HTTP in JSON protocol using libcurl. Libcurl is widely available and doesn't require a lot of extra baggage.
Another approach would be to add a dependency to the OpenTelemetry C++ client library (https://github.com/open-telemetry/opentelemetry-cpp). That would come with more full support for different transports, but is a much bigger dependency.
Yet another approach would be to bundle a standalone binary like pg-tracing-otel-forwarder with this extensions. The extension could launch it on startup as a separate process, for example. Or just improve the binary to make it nicer to deploy, even though it needs to be deployed separately.
I'm leaning towards the libcurl approach myself. What do you think?
If we go down this route, we could simplify the rest of the code by removing the pull model. It's nice to have something to play with, though, even when you don't have OpenTelemetry colletor or something like Jaeger running. Maybe add an option to dump all the spans to a file instead of pushing them with OTLP?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.