Code Monkey home page Code Monkey logo

xk6-output-influxdb's Introduction

xk6-output-influxdb

k6 extension for InfluxDB v2, it adds the support for the latest v2 version and the compatibility API for v1.8+.

Why is this output not directly part of k6 core?

The k6 core already supports the InfluxDB v1 so the natural feeling would be to do the same for the v2. Unfortunately, the v2 has introduced some breaking changes in the core parts of the API. This would make it difficult to support both versions without taking a bunch of compromises for maintaining the retro-compatibility or introducing breaking changes in the current user experience of the k6's InfluxDB output, with a high probability to create more confusion for k6's users. For this main reason, the k6 development team has decided to create a new and independent extension for InfluxDB v2.

Install

To build a k6 binary with this extension, first ensure you have the prerequisites:

  1. Build with xk6:
xk6 build --with github.com/grafana/xk6-output-influxdb

This will result in a k6 binary in the current directory.

  1. Run with the just built k6 binary:
K6_INFLUXDB_ORGANIZATION=<insert-here-org-name> \
K6_INFLUXDB_BUCKET=<insert-here-bucket-name> \
K6_INFLUXDB_TOKEN=<insert-here-valid-token> \
./k6 run -o xk6-influxdb=http://localhost:8086 <script.js>

Using Docker

This Dockerfile builds a docker image with the k6 binary.

Configuration

Options for fine-grained control for flushing and connections.

ENV Default Description
K6_INFLUXDB_ORGANIZATION The Organization.
K6_INFLUXDB_BUCKET The Bucket.
K6_INFLUXDB_TOKEN The Token.
K6_INFLUXDB_ADDR http://localhost:8086 The address of the instance.
K6_INFLUXDB_PUSH_INTERVAL 1s The flush's frequency of the k6 metrics.
K6_INFLUXDB_CONCURRENT_WRITES 4 Number of concurrent requests for flushing data. It is useful when a request takes more than the expected time (more than flush interval).
K6_INFLUXDB_TAGS_AS_FIELDS vu:int,iter:int,url A comma-separated string to set k6 metrics as non-indexable fields (instead of tags). An optional type can be specified using :type as in vu:int will make the field integer. The possible field types are int, bool, float and string, which is the default. Example: vu:int,iter:int,url:string,event_time:int.
K6_INFLUXDB_INSECURE false When true, it will skip https certificate verification.
K6_INFLUXDB_PRECISION 1ns The timestamp Precision.

Docker Compose

This repo includes a docker-compose.yml file that starts InfluxDB, Grafana and k6. This is just a quick setup to show the usage; for real use case you might want to deploy outside of docker, use volumes and probably update versions.

Clone the repo to get started and follow these steps:

  1. Put your k6 scripts in the samples directory or use the http_2.js example.

  2. Start the docker compose environment.

    docker compose up -d
    # Output
    Creating xk6-output-influxdb_influxdb_1 ... done
    Creating xk6-output-influxdb_k6_1       ... done
    Creating xk6-output-influxdb_grafana_1  ... done
  3. Use the k6 Docker image to run the k6 script and send metrics to the InfluxDB container started on the previous step. You must set the testid tag with a unique identifier to segment the metrics into discrete test runs for the Grafana dashboards.

    docker compose run --rm -T k6 run -<samples/http_2.js --tag testid=<SOME-ID>

    For convenience, the docker-run.sh can be used to simply:

    ./docker-run.sh samples/http_2.js
  4. Visit http://localhost:3000/ to view results in Grafana.

    This repository includes a basic dashboard. If you want to build a custom Dashboard, contact the k6 team in Slack.

Compatibility API

The v2 includes a InfluxDB v1.8+ compatibility API that adds endpoints for communicating with an InfluxDB v1.

Client API usage differences summary:

  1. Use the form username:password for an authentication token. Example: my-user:my-password. Use an empty string ("") if the server doesn't require authentication.
  2. The organization parameter is not used. Use an empty string ("") where necessary.
  3. Use the form database/retention-policy where a bucket is required. Skip retention policy if the default retention policy should be used. Examples: telegraf/autogen, telegraf.

xk6-output-influxdb's People

Contributors

codebien avatar javaducky avatar jwcastillo avatar mstoykov avatar ppcano avatar rodzac avatar toscott avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

xk6-output-influxdb's Issues

Dockerfile error

Used the sample Dockerfile will get those error:

=> ERROR [builder 6/6] RUN xk6 build --with github.com/grafana/xk6-output-influxdb=. --output /tmp/k6 0.5s

[builder 6/6] RUN xk6 build --with github.com/grafana/xk6-output-influxdb=. --output /tmp/k6:
0.285 2024/07/14 10:42:04 [INFO] Temporary folder: /tmp/buildenv_2024-07-14-1042.3784705506
0.285 2024/07/14 10:42:04 [INFO] Initializing Go module
0.285 2024/07/14 10:42:04 [INFO] exec (timeout=10s): /usr/local/go/bin/go mod init k6
0.294 go: creating new go.mod: module k6
0.296 2024/07/14 10:42:04 [INFO] Replace github.com/grafana/xk6-output-influxdb => /go/src/go.k6.io/k6
0.296 2024/07/14 10:42:04 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod edit -replace github.com/grafana/xk6-output-influxdb=/go/src/go.k6.io/k6
0.302 2024/07/14 10:42:04 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
0.311 go: warning: "all" matched no packages
0.314 2024/07/14 10:42:04 [INFO] Pinning versions
0.315 2024/07/14 10:42:04 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
0.331 go: finding module for package github.com/grafana/xk6-output-influxdb
0.426 k6 imports
0.426 github.com/grafana/xk6-output-influxdb: module github.com/grafana/xk6-output-influxdb@latest found (v0.4.3, replaced by /go/src/go.k6.io/k6), but does not contain package github.com/grafana/xk6-output-influxdb
0.428 2024/07/14 10:42:04 [INFO] Cleaning up temporary folder: /tmp/buildenv_2024-07-14-1042.3784705506
0.429 2024/07/14 10:42:04 [FATAL] exit status 1


1 warning found (use --debug to expand):

  • FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 1)
    Dockerfile:6

4 | RUN apk --no-cache add git
5 | RUN go install go.k6.io/xk6/cmd/xk6@latest
6 | >>> RUN xk6 build --with github.com/grafana/xk6-output-influxdb=. --output /tmp/k6
7 |
8 | FROM alpine:3.17

ERROR: failed to solve: process "/bin/sh -c xk6 build --with github.com/grafana/xk6-output-influxdb=. --output /tmp/k6" did not complete successfully: exit code: 1

xk6 build with influxdb fails after k6 version v0.38.0

OS: Windows

Most likely caused by:
grafana/k6#2433

ERROR:

C:\repo>k6 version
k6 v0.38.0 ((devel), go1.18.1, windows/amd64)

C:\repo>xk6 build --with github.com/grafana/xk6-output-influxdb
2022/05/10 17:10:51 [INFO] Temporary folder: C:\Users\###\AppData\Local\Temp\2\buildenv_2022-05-10-1710.586738848
2022/05/10 17:10:51 [INFO] Writing main module: C:\Users\###\AppData\Local\Temp\2\buildenv_2022-05-10-1710.586738848\main.go
2022/05/10 17:10:51 [INFO] Initializing Go module
2022/05/10 17:10:51 [INFO] exec (timeout=10s): C:\Program Files\Go\bin\go.exe mod init k6
go: creating new go.mod: module k6
go: to add module requirements and sums:
        go mod tidy
2022/05/10 17:10:51 [INFO] Pinning versions
2022/05/10 17:10:51 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod edit -require go.k6.io/k6@latest
2022/05/10 17:10:51 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod tidy -compat=1.17
go: finding module for package github.com/nxadm/tail
go: found github.com/nxadm/tail in github.com/nxadm/tail v1.4.8
2022/05/10 17:10:52 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod edit -require github.com/grafana/xk6-output-influxdb@latest
2022/05/10 17:10:52 [INFO] exec (timeout=0s): C:\Program Files\Go\bin\go.exe mod tidy -compat=1.17
go: finding module for package go.k6.io/k6/stats
k6 imports
        github.com/grafana/xk6-output-influxdb imports
        github.com/grafana/xk6-output-influxdb/pkg/influxdb imports
        go.k6.io/k6/stats: module go.k6.io/k6@latest found (v0.38.1), but does not contain package go.k6.io/k6/stats
2022/05/10 17:10:54 [INFO] Cleaning up temporary folder: C:\Users\###\AppData\Local\Temp\2\buildenv_2022-05-10-1710.586738848
2022/05/10 17:10:54 [FATAL] exit status 1

error while building docker image

Yesterday everything was fine, today when I launched the project in Jenkins I got the following error:

go.buf.build/grpc/go/prometheus/prometheus: unrecognized import path "go.buf.build/grpc/go/prometheus/prometheus": https fetch: Get "https://go.buf.build/grpc/go/prometheus/prometheus?go-get=1": dial tcp: lookup go.buf.build on 10.1.5.5:53: **no such host**
09:26:53  #12 11.31 2023/08/04 06:26:53 [INFO] Cleaning up temporary folder: /tmp/buildenv_2023-08-04-0626.2242724175
09:26:53  #12 11.31 2023/08/04 06:26:53 [FATAL] exit status 1
09:26:54  #12 ERROR: process "/bin/sh -c xk6 build --with github.com/grafana/xk6-output-influxdb=. --output /tmp/k6" **did not complete successfully**: exit code: 1

I see we have an incorrect link to: buf.build

Discuss whether to not move some fields to be int

This was previously discussed when field types were added, but we didn't change them as it would break if used on an influxdb with the same fields but different types.

Even then I was of the opinion that this might not be so bad given the benefit of all future deployments having more accurate field types, so I guess now is a good time to probably do that?

We might also want to revisit the tag to fields list and see if we should not add some newer tag

Build command in command prompt seems to fail.

When I run the command

K6_INFLUXDB_ORGANIZATION=myorg
K6_INFLUXDB_BUCKET=mybucket
K6_INFLUXDB_TOKEN=mytoken
./k6 run -o xk6-influxdb=http://localhost:8086 myscript.js

I get this error message:

K6_INFLUXDB_ORGANIZATION=myorg : The term 'K6_INFLUXDB_ORGANIZATION=myorg' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was
included, verify that the path is correct and try again.
At line:1 char:1

  • K6_INFLUXDB_ORGANIZATION=myorg \
  •   + CategoryInfo          : ObjectNotFound: (K6_INFLUXDB_ORGANIZATION=myorg:String) [], CommandNotFoundException
      + FullyQualifiedErrorId : CommandNotFoundException
    
    

K6_INFLUXDB_BUCKET=mybucket : The term 'K6_INFLUXDB_BUCKET=mybucket' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:2 char:1

  • K6_INFLUXDB_BUCKET=mybucket \
  •   + CategoryInfo          : ObjectNotFound: (K6_INFLUXDB_BUCKET=mybucket:String) [], CommandNotFoundException
      + FullyQualifiedErrorId : CommandNotFoundException
    
    

K6_INFLUXDB_TOKEN=mytoken : The term 'K6_INFLUXDB_TOKEN=mytoken' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and t
ry again.
At line:3 char:1

  • K6_INFLUXDB_TOKEN=mytoken
  •   + CategoryInfo          : ObjectNotFound: (K6_INFLUXDB_TOK...mytoken:String) [], CommandNotFoundException
      + FullyQualifiedErrorId : CommandNotFoundException
      
      
    

Execute in command line

ERRO[0000] The moduleSpecifier "samples/http_2.js" couldn't be found on local disk. Make sure that you've specified the right path to the file. If you're running k6 using the Docker image make sure you have mounted the local directory (-v /local/path/:/inside/docker/path) containing your script and modules so that they're accessible by k6 from inside of the container, see https://k6.io/docs/using-k6/modules#using-local-modules-with-docker. Additionally it was tried to be loaded as remote module by prepending "https://" to it, which also didn't work. Remote resolution error: "Get "https://samples/http_2.js": dial tcp: lookup samples on 127.0.0.11:53: no such host"

Number of measurements in InfluxDB is less then in result.json

Hi,
I'am wondering why the number of record during the testrun differ from the report summary of k6 itself and also differ from json result output. There are always less records in InfluxDB.
My Last run:
K6 excerpt
http_reqs......................: 29500 132.648204/s

JQ Query:
jq '. | select(.type=="Point" and .metric == "http_reqs" and .data.tags.testrun == "20240422145315" ) | .data.value' script_results.json | jq -s "length"
29500

Influx Flux Query:
from(bucket: "k6")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "http_reqs")
|> filter(fn: (r) => r["_field"] == "value")
|> filter(fn: (r) => r["testrun"] == "20240422145315")
|> group()
|> count()
|> yield(name: "mean")

29268 ---> missing 232 records

Any idea what is going wrong here?

Kind Regards
Alexey

Improve Grafana dashboard examples

Improve upon the currently provided dashboards (grafana/dashboards/).

Another goal for this change would be to align the look-and-feel for the Influx-backed dashboard (this extension), the xk6-output-prometheus-remote, and xk6-output-timescaledb possibly with the output provided by the k6 Cloud App. This would provide a consistent user experience for the varied data sources.

Ideally, this dashboard would be promoted to the Grafana site as the "official" dashboard for displaying k6 metrics backed by InfluxDB datasources.

google.golang.org/genproto/googleapis/rpc/status: ambiguous import issue

Hi,

Am trying to build locally k6 with xk6-output-influxdb

Command I am running:

docker run --rm -e GOOS=darwin -e CGO_ENABLED=1 -u "$(id -u):$(id -g)" -v "$PWD:/xk6" \
            xk6-andrei build master \
            --with github.com/grafana/xk6-output-influxdb@latest

But seems I get this error:

2023/11/20 09:19:29 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod edit -require github.com/grafana/xk6-output-influxdb@latest
2023/11/20 09:19:29 [INFO] exec (timeout=0s): /usr/local/go/bin/go mod tidy -compat=1.17
go: downloading github.com/grafana/xk6-output-influxdb v0.4.1
go: downloading github.com/influxdata/influxdb-client-go/v2 v2.12.2
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/deepmap/oapi-codegen v1.12.4
go: downloading github.com/influxdata/line-protocol v0.0.0-20210922203350-b1ad95c89adf
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/apapsch/go-jsonmerge/v2 v2.0.0
go: downloading gopkg.in/guregu/null.v3 v3.5.0
go: downloading github.com/spf13/afero v1.9.5
go: downloading google.golang.org/genproto v0.0.0-20210226172003-ab064af71705
k6 imports
	github.com/mostafa/xk6-kafka imports
	go.k6.io/k6/js/modules imports
	go.k6.io/k6/loader tested by
	go.k6.io/k6/loader.test imports
	go.k6.io/k6/lib/testutils/httpmultibin imports
	google.golang.org/grpc/status imports
	google.golang.org/genproto/googleapis/rpc/status: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/status in multiple modules:
	google.golang.org/genproto v0.0.0-20210226172003-ab064af71705 (/go/pkg/mod/google.golang.org/[email protected]/googleapis/rpc/status)
	google.golang.org/genproto/googleapis/rpc v0.0.0-20230525234030-28d5490b6b19 (/go/pkg/mod/google.golang.org/genproto/googleapis/[email protected]/status)

The image xk6-andrei is just a docker image that I've built for myself to verify m1 compatibility (native xk6 docker image is not).

I can confirm that v0.3.0 is not giving me this error,v0.4.0 is giving the same error as above

I don't think it's related to the custom docker image I've created, as it highlighted here, where the image is built against ubuntu: https://github.com/sfat/xk6/actions/runs/6919193432/job/18842123451

(I am trying to modify existing xk6 to also build for arm64 (M1) in order for the docker container to run faster)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.