Hello,
I am trying to run your demo, i think it's great!
I am using main git branch
As a user,
[vagrant@fedora boot-opentelemetry-tempo]$ docker-compose up
checking dockers ..
[vagrant@fedora boot-opentelemetry-tempo]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
387377145b68 grafana/grafana:7.4.0-ubuntu "/run.sh" 56 minutes ago Up 43 seconds 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp boot-opentelemetry-tempo_grafana_1
4af1c2b2156d grafana/promtail:2.2.0 "/usr/bin/promtail -…" 56 minutes ago Up 43 seconds boot-opentelemetry-tempo_promtail_1
9df372e9b436 prom/prometheus:latest "/bin/prometheus --c…" 56 minutes ago Up 43 seconds 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp boot-opentelemetry-tempo_prometheus_1
699cf488dc37 mnadeem/boot-otel-tempo-api:0.0.1-SNAPSHOT "sh /app/bin/start-a…" 56 minutes ago Up 44 seconds 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp boot-opentelemetry-tempo_boot-otel-tempo-api_1
c7a113f40b01 mnadeem/boot-otel-tempo-provider1:0.0.1-SNAPSHOT "sh /app/bin/start-a…" 56 minutes ago Up 44 seconds 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp boot-opentelemetry-tempo_boot-otel-tempo-provider1_1
7bb1454e015f dpage/pgadmin4 "/entrypoint.sh" 56 minutes ago Up 44 seconds 443/tcp, 0.0.0.0:7070->80/tcp, :::7070->80/tcp boot-opentelemetry-tempo_pgadmin_1
40e01d4ef144 grafana/tempo-query:0.7.0 "/go/bin/query-linux…" 56 minutes ago Up 44 seconds 0.0.0.0:16686->16686/tcp, :::16686->16686/tcp boot-opentelemetry-tempo_tempo-query_1
91464ff22136 postgres "docker-entrypoint.s…" 56 minutes ago Up 44 seconds 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp boot-opentelemetry-tempo_provider1-db_1
6c518fb52f80 grafana/tempo:0.7.0 "/tempo -config.file…" 56 minutes ago Up 44 seconds 0.0.0.0:14250->14250/tcp, :::14250->14250/tcp, 0.0.0.0:14268->14268/tcp, :::14268->14268/tcp, 0.0.0.0:55680->55680/tcp, :::55680->55680/tcp, 0.0.0.0:3102->3100/tcp, :::3102->3100/tcp boot-opentelemetry-tempo_tempo_1
c416a8970576 mnadeem/volume_exporter "/bin/volume_exporte…" 56 minutes ago Up 45 seconds 0.0.0.0:9889->9888/tcp, :::9889->9888/tcp boot-opentelemetry-tempo_volume_exporter_1
2208cc1c597e grafana/loki:2.2.0 "/usr/bin/loki -conf…" 56 minutes ago Up 44 seconds 0.0.0.0:3101->3100/tcp, :::3101->3100/tcp boot-opentelemetry-tempo_loki_1
[vagrant@fedora boot-opentelemetry-tempo]$
I can do sql "insert" into postgres, ok ..
I can get a web resquest-response
[
{
"origin": "SEA",
"destination": "LAS",
"airline": "Deltoid",
"departureTime": "2020-12-01T18:10:00.000+00:00"
},
{
"origin": "SEA",
"destination": "LAS",
"airline": "Unitely",
"departureTime": "2020-12-01T19:25:00.000+00:00"
},
{
"origin": "SEA",
"destination": "LAS",
"airline": "Pan-Um",
"departureTime": "2020-12-01T15:15:00.000+00:00"
}
]
Console messages show that there is somethings running ok
loki_1 | level=info ts=2023-01-25T09:50:08.153612696Z caller=table_manager.go:476 msg="creating table" table=index_18981
loki_1 | level=info ts=2023-01-25T09:50:08.153619912Z caller=table_manager.go:476 msg="creating table" table=index_19075
loki_1 | level=info ts=2023-01-25T09:50:08.153629425Z caller=table_manager.go:476 msg="creating table" table=index_19167
loki_1 | level=info ts=2023-01-25T09:50:08.153634125Z caller=table_manager.go:476 msg="creating table" table=index_19314
loki_1 | level=info ts=2023-01-25T09:50:08.153638346Z caller=table_manager.go:476 msg="creating table" table=index_19361
loki_1 | level=info ts=2023-01-25T09:50:08.153642468Z caller=table_manager.go:476 msg="creating table" table=index_18714
loki_1 | level=info ts=2023-01-25T09:50:08.153646605Z caller=table_manager.go:476 msg="creating table" table=index_18894
loki_1 | level=info ts=2023-01-25T09:50:08.153650688Z caller=table_manager.go:476 msg="creating table" table=index_18933
loki_1 | level=info ts=2023-01-25T09:50:08.15365477Z caller=table_manager.go:476 msg="creating table" table=index_19216
loki_1 | level=info ts=2023-01-25T09:50:08.153663394Z caller=table_manager.go:476 msg="creating table" table=index_18588
loki_1 | level=info ts=2023-01-25T09:50:08.153670899Z caller=table_manager.go:476 msg="creating table" table=index_18939
loki_1 | level=info ts=2023-01-25T09:50:08.153675518Z caller=table_manager.go:476 msg="creating table" table=index_19014
tempo_1 | level=info ts=2023-01-25T09:50:08.183434929Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1 | level=info ts=2023-01-25T09:50:08.183468568Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
tempo_1 | level=info ts=2023-01-25T09:50:38.183476029Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1 | level=info ts=2023-01-25T09:50:38.183556963Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
boot-otel-tempo-api_1 | 2023-01-25 03:50:45.028 INFO [boot-otel-tempo-api,f0ba2bd160dca1f79ed6ccf0d0b8135d,aacf0ccfaaccffaf] 7 --- [nio-8080-exec-1] i.o.example.flight.FlightController : Before Service Method Call
boot-otel-tempo-api_1 | 2023-01-25 03:50:45.028 INFO [boot-otel-tempo-api,f0ba2bd160dca1f79ed6ccf0d0b8135d,aacf0ccfaaccffaf] 7 --- [nio-8080-exec-1] i.o.example.flight.FlightService : Getting flights for "SEA"
boot-otel-tempo-api_1 | 2023-01-25 03:50:45.029 INFO [boot-otel-tempo-api,f0ba2bd160dca1f79ed6ccf0d0b8135d,ae7ab8e6bccd017b] 7 --- [nio-8080-exec-1] i.o.example.flight.FlightClient : Getting Flights from http://boot-otel-tempo-provider1:8090/flights
boot-otel-tempo-provider1_1 | 2023-01-25 03:50:45.033 INFO [boot-otel-tempo-provider1,f0ba2bd160dca1f79ed6ccf0d0b8135d,a764d501bf0e2483] 7 --- [io-8090-exec-10] i.o.example.flight.FlighController : processing Request
boot-otel-tempo-provider1_1 | 2023-01-25 03:50:45.033 INFO [boot-otel-tempo-provider1,f0ba2bd160dca1f79ed6ccf0d0b8135d,519e77d05f580ee9] 7 --- [io-8090-exec-10] i.o.example.flight.FlightService : Doing some work In New span
boot-otel-tempo-api_1 | 2023-01-25 03:50:45.040 INFO [boot-otel-tempo-api,f0ba2bd160dca1f79ed6ccf0d0b8135d,4b36a7c39b9ca986] 7 --- [nio-8080-exec-1] i.o.example.flight.FlightService : Doing some work In New span
loki_1 | level=info ts=2023-01-25T09:51:08.142503097Z caller=table_manager.go:208 msg="syncing tables"
loki_1 | level=info ts=2023-01-25T09:51:08.147230987Z caller=table_manager.go:171 msg="uploading tables"
loki_1 | level=info ts=2023-01-25T09:51:08.147296101Z caller=table.go:336 msg="uploading table index_19382"
loki_1 | level=info ts=2023-01-25T09:51:08.147306867Z caller=table.go:363 msg="finished uploading table index_19382"
loki_1 | level=info ts=2023-01-25T09:51:08.147311227Z caller=table.go:421 msg="cleaning up unwanted dbs from table index_19382"
tempo_1 | level=info ts=2023-01-25T09:51:08.183746382Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1 | level=info ts=2023-01-25T09:51:08.183771985Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
provider1-db_1 | 2023-01-25 09:51:08.192 UTC [27] LOG: checkpoint starting: time
tempo_1 | level=info ts=2023-01-25T09:51:08.209778748Z caller=flush.go:256 msg="flushing block" userid=single-tenant block=2c376cae-b17a-4cf4-a41c-1593cbe51812
provider1-db_1 | 2023-01-25 09:51:08.221 UTC [27] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.023 s, total=0.029 s; sync files=2, longest=0.021 s, average=0.012 s; distance=0 kB, estimate=0 kB
and again
boot-otel-tempo-api_1 | 2023-01-25 03:52:30.820 INFO [boot-otel-tempo-api,e00a373d52614d324a56f23cd91dd564,eb88cd7496af3413] 7 --- [nio-8080-exec-5] i.o.example.flight.FlightController : Before Service Method Call
boot-otel-tempo-api_1 | 2023-01-25 03:52:30.820 INFO [boot-otel-tempo-api,e00a373d52614d324a56f23cd91dd564,eb88cd7496af3413] 7 --- [nio-8080-exec-5] i.o.example.flight.FlightService : Getting flights for "SEA"
boot-otel-tempo-api_1 | 2023-01-25 03:52:30.821 INFO [boot-otel-tempo-api,e00a373d52614d324a56f23cd91dd564,c18b60005f6f3a67] 7 --- [nio-8080-exec-5] i.o.example.flight.FlightClient : Getting Flights from http://boot-otel-tempo-provider1:8090/flights
boot-otel-tempo-provider1_1 | 2023-01-25 03:52:30.827 INFO [boot-otel-tempo-provider1,e00a373d52614d324a56f23cd91dd564,41b8c42ba1bc195b] 7 --- [io-8090-exec-10] i.o.example.flight.FlighController : processing Request
boot-otel-tempo-provider1_1 | 2023-01-25 03:52:30.827 INFO [boot-otel-tempo-provider1,e00a373d52614d324a56f23cd91dd564,80270a556d6a59c4] 7 --- [io-8090-exec-10] i.o.example.flight.FlightService : Doing some work In New span
boot-otel-tempo-api_1 | 2023-01-25 03:52:30.833 INFO [boot-otel-tempo-api,e00a373d52614d324a56f23cd91dd564,b7a2906447b9ad7b] 7 --- [nio-8080-exec-5] i.o.example.flight.FlightService : Doing some work In New span
tempo_1 | level=info ts=2023-01-25T09:52:38.183271501Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1 | level=info ts=2023-01-25T09:52:38.183332092Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
Running tempo, i cannot get anything, it only shows:
tempo_1 | level=info ts=2023-01-25T09:49:08.184133771Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1 | level=info ts=2023-01-25T09:49:08.184275657Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
tempo_1 | level=error ts=2023-01-25T09:49:08.311343846Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1 | level=error ts=2023-01-25T09:49:08.311410872Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1 | level=error ts=2023-01-25T09:49:08.311407186Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1 | level=error ts=2023-01-25T09:49:08.311434239Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1 | level=error ts=2023-01-25T09:49:08.311450076Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
tempo_1 | level=info ts=2023-01-25T09:49:38.183278219Z caller=compactor.go:91 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo_1 | level=info ts=2023-01-25T09:49:38.183351782Z caller=compactor.go:95 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
It shows:
Then, if i use jaeger, put a traceid, response is empty and lokie shows:
tempo-query_1 | {"level":"error","ts":1674639544.7755914,"caller":"zap/logger.go:33","msg":"failed to flush Jaeger spans to server: write udp 127.0.0.1:46646->127.0.0.1:6831: write: connection refused","stacktrace":"github.com/uber/jaeger-client-go/log/zap.(*Logger).Error\n\tgithub.com/uber/[email protected]+incompatible/log/zap/logger.go:33\ngithub.com/uber/jaeger-client-go.(*remoteReporter).processQueue.func1\n\tgithub.com/uber/[email protected]+incompatible/reporter.go:288\ngithub.com/uber/jaeger-client-go.(*remoteReporter).processQueue\n\tgithub.com/uber/[email protected]+incompatible/reporter.go:298"}
And, jaeger info is empty ...
what am i doing wrong?
Tx
PD
loki is ok ( it seems that loki doesn't like touching 'job string':
![image](https://user-images.githubusercontent.com/1777753/214538322-71dce6e1-cf5e-4638-96bf-1f130ad1bd20.png)
Temp seems ok:
![image](https://user-images.githubusercontent.com/1777753/214536013-81f5f331-ed56-42b3-a9cb-f1ebbdf1a38f.png)
Prometheus ok:
![image](https://user-images.githubusercontent.com/1777753/214536476-a2417efb-e0df-470a-9dac-4628736eb09b.png)
Jaeger ... [no service]
![image](https://user-images.githubusercontent.com/1777753/214540323-f8692eba-93ef-4d2e-9b73-ddc274d9fb32.png)
Requesting using url browser, runs ok ... http://localhost:6650/trace/5ad80347b409a4b5d9aa7317c4e7ca3c
![image](https://user-images.githubusercontent.com/1777753/214540879-f643ede2-5e56-47da-986a-4dca886b0328.png)