Comments (7)
bourbonkk
commented on August 29, 2024
I need to flatten the data in order to use the druid
The clymene is wrapped in a timeseries array and loaded in kafka.
{
"timeseries": [
{
"labels": [
{
"name": "__name__",
"value": "go_gc_duration_seconds"
},
{
"name": "cluster",
"value": "target-cluster"
},
{
"name": "instance",
"value": "localhost:9100"
},
{
"name": "job",
"value": "node-exporter"
},
{
"name": "quantile",
"value": "0"
}
],
"samples": [
{
"value": 0.000013105,
"timestamp": "1671108031021"
}
]
},
{
"labels": [
{
"name": "__name__",
"value": "go_gc_duration_seconds"
},
{
"name": "cluster",
"value": "target-cluster"
},
{
"name": "instance",
"value": "localhost:9100"
},
{
"name": "job",
"value": "node-exporter"
},
{
"name": "quantile",
"value": "0.25"
}
],
"samples": [
{
"value": 0.000024546,
"timestamp": "1671108031021"
}
]
},
{
"labels": [
{
"name": "__name__",
"value": "go_gc_duration_seconds"
},
{
"name": "cluster",
"value": "target-cluster"
},
{
"name": "instance",
"value": "localhost:9100"
},
{
"name": "job",
"value": "node-exporter"
},
{
"name": "quantile",
"value": "0.5"
}
],
"samples": [
{
"value": 0.000026219,
"timestamp": "1671108031021"
}
]
},
{
"labels": [
{
"name": "__name__",
"value": "go_gc_duration_seconds"
},
{
"name": "cluster",
"value": "target-cluster"
},
{
"name": "instance",
"value": "localhost:9100"
},
{
"name": "job",
"value": "node-exporter"
},
{
"name": "quantile",
"value": "0.75"
}
],
"samples": [
{
"value": 0.000028494,
"timestamp": "1671108031021"
}
]
},
{
"labels": [
{
"name": "__name__",
"value": "go_gc_duration_seconds"
},
{
"name": "cluster",
"value": "target-cluster"
},
{
"name": "instance",
"value": "localhost:9100"
},
{
"name": "job",
"value": "node-exporter"
},
{
"name": "quantile",
"value": "1"
}
],
"samples": [
{
"value": 0.000079431,
"timestamp": "1671108031021"
}
]
}
]
}from clymene.
bourbonkk
commented on August 29, 2024
Can't you make it flat in the flattenSpec setting without modifying the code?
from clymene.
litkhai
commented on August 29, 2024
From Druid 24.0, you may use nested json functions without flattening.
Please refer to below and feel free to reach out for more details.
https://druid.apache.org/docs/latest/querying/nested-columns.html
https://druid.apache.org/docs/latest/querying/sql-json-functions.html
from clymene.
litkhai
commented on August 29, 2024
In ingestion spec, the below part should be set:
"transformSpec": {
"transforms": [
{
"type": "expression",
"name": "data",
"expression": "parse_json(\"data\")"
}
]
},
"dimensionsSpec": {
"dimensions": [
{
"name": "data",
"type": "json"
}
]
},
from clymene.
bourbonkk
commented on August 29, 2024
@litkhai 안녕하세요. 답변이 가능하실지는 모르겠지만, 다시한번 문의드립니다.
timeseries라는 Object에 쌓여있는 포맷을 json array로 데이터를 평탄화할 수 있도록 수정했는데요
이런 케이스는 druid에서 사용할 수 있을까요?
테스트 해본바로는 json format으로 jq flattening 설정을 추가해서 .[] 넣었더니 parse 에러가 발생합니다.
데이터가 카프카에 한번에 쌓이는 양이 많다보니 건건히 produce하기에는 무리가 있다는 생각이 들어서 아래와 같은 포맷으로 변경해봤습니다.
[{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0","timestamp":"2023-01-23T02:14:16.019Z","value":0.000006993},{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.25","timestamp":"2023-01-23T02:14:16.019Z","value":0.00002612},{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.5","timestamp":"2023-01-23T02:14:16.019Z","value":0.000026601},{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.75","timestamp":"2023-01-23T02:14:16.019Z","value":0.000027072},{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"1","timestamp":"2023-01-23T02:14:16.019Z","value":0.0000423},{"__name__":"go_gc_duration_seconds_sum","cluster...
from clymene.
bourbonkk
commented on August 29, 2024
Thank you for your support.
I checked that it is applied in the format below.
Except for , it was connected in byte array form.
{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0","timestamp":"2023-01-24T06:17:31.02Z","value":0.000008857}{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.25","timestamp":"2023-01-24T06:17:31.02Z","value":0.00002599}{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.5","timestamp":"2023-01-24T06:17:31.02Z","value":0.00002668}{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.75","timestamp":"2023-01-24T06:17:31.02Z","value":0.000027523}{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"1","timestamp":"2023-01-24T06:17:31.02Z","value":0.000050135}
from clymene.
litkhai
commented on August 29, 2024
안녕하세요.
먼저 새해 복 많이 받으세요.
제가 아직 내용을 상세히 못봤지만, 배열처리는 아직은 기능 개선이 상당히 필요한 부분이긴 합니다.
내용 상세히 보고 차주 전에 회신 드리겠습니다. 감사합니다!
… On Jan 23, 2023, at 1:39 PM, Allen Kim ***@***.***> wrote:
@litkhai <https://github.com/litkhai> 안녕하세요. 답변이 가능하실지는 모르겠지만, 다시한번 문의드립니다.
timeseries라는 Object에 쌓여있는 포맷을 json array로 데이터를 평탄화할 수 있도록 수정했는데요
이런 케이스는 druid에서 사용할 수 있을까요?
테스트 해본바로는 json format으로 jq flattening 설정을 추가해서 .[] 넣었더니 parse 에러가 발생합니다.
데이터가 카프카에 한번에 쌓이는 양이 많다보니 건건히 produce하기에는 무리가 있다는 생각이 들어서 아래와 같은 포맷으로 변경해봤습니다.
[{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0","timestamp":"2023-01-23T02:14:16.019Z","value":0.000006993},{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.25","timestamp":"2023-01-23T02:14:16.019Z","value":0.00002612},{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.5","timestamp":"2023-01-23T02:14:16.019Z","value":0.000026601},{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"0.75","timestamp":"2023-01-23T02:14:16.019Z","value":0.000027072},{"__name__":"go_gc_duration_seconds","cluster":"target-cluster","instance":"localhost:9100","job":"node-exporter","quantile":"1","timestamp":"2023-01-23T02:14:16.019Z","value":0.0000423},{"__name__":"go_gc_duration_seconds_sum","cluster...
—
Reply to this email directly, view it on GitHub <#76 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGCNIQPTY25VPU4YWZYPKU3WTYDOVANCNFSM6AAAAAAS7WXGFM>.
You are receiving this because you were mentioned.
from clymene.
Related Issues (20)
- Release v2.2.7
- Developing an agent that pushes node-exporter metrics to clymene backend
- Developing an agent that pushes cAdvisor metrics to the clymene backend
- Gateway LOG_STORAGE factory function separation HOT 1
- helm chart development HOT 1
- clymene, promtail, jaeger backend configuration yaml development
- After saving data from gateway to kafka, iningester cannot be consumed HOT 1
- Added split transmission function when sending timeseries
- elasticsearch timestamp convert HOT 1
- eBPF trace agent architecture design
- CrashLoopBackOff Issue HOT 3
- Support for Druid Storage
- Setting the influxdb results in an error
- release v2.2.0
- Panic occurs when kafka produce fails
- Add test code and apply CI
- Performance Comparison of Clymene-agent and Prometheus agent Modes
- Separation of grpc, http server/client options of gateway component
- Vulnerability Patch Release v2.2.3 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clymene.