Code Monkey home page Code Monkey logo

snyk_exporter's Introduction

Snyk exporter [Archived]

Note: This project is no longer actively maintained and is therefore archived.

Build Status Go Report Card Docker Repository on Quay

Prometheus exporter for Snyk written in Go. Allows for exporting scanning data into Prometheus by scraping the Snyk HTTP API.

Installation

Several pre-compiled binaries are available from the releases page.

A docker image is also available on our Quay.io registry.

docker run quay.io/lunarway/snyk_exporter --snyk.api-token <api-token>

Usage

You need a Snyk API token to access to API. Get your through the Snyk account settings.

It exposes prometheus metrics on /metrics on port 9532 (can be configured).

snyk_exporter --snyk.api-token <api-token>

See all configuration options with the --help flag

$ snyk_exporter --help
usage: snyk_exporter --snyk.api-token=SNYK.API-TOKEN [<flags>]

Snyk exporter for Prometheus. Provide your Snyk API token and the organization(s) to scrape to expose Prometheus metrics.

Flags:
  -h, --help               Show context-sensitive help (also try --help-long and --help-man).
      --snyk.api-url="https://snyk.io/api/v1"
                           Snyk API URL
      --snyk.api-token=SNYK.API-TOKEN
                           Snyk API token
  -i, --snyk.interval=600  Polling interval for requesting data from Snyk API in seconds
      --snyk.organization=SNYK.ORGANIZATION ...
                           Snyk organization ID to scrape projects from (can be repeated for multiple organizations)
      --snyk.timeout=10    Timeout for requests against Snyk API
      --web.listen-address=":9532"
                           Address on which to expose metrics.
      --log.level="info"   Only log messages with the given severity or above. Valid levels: [debug, info, warn, error, fatal]
      --log.format="logger:stderr"
                           Set the log target and format. Example: "logger:syslog?appname=bob&local=7" or "logger:stdout?json=true"
      --version            Show application version.

It is possible to use a file to pass arguments to the exporter. For example:

 echo --snyk.api-token=<>\n > args

And run the exporter using:

./snyk-exporter @args

Design

The exporter starts a long-running go routine on startup that scrapes the Snyk API with a fixed interval (default every 10 minutes). The interval can be configured as needed.

The API results are aggregated and recorded on the snyk_vulnerabiilities_total metric with the following labels:

  • organization - The organization where the vulnerable project exists
  • project - The project with a vulnerability
  • severity - The severity of the vulnerability, can be critical, high, medium and low
  • issue_type - The type of issue, e.g. vuln, license
  • issue_title - The issue title of the vulnerability, e.g. Denial os Service (DoS). Can be the CVE if the vulnerability is not named by Snyk
  • ignored - The issue is ignored in Snyk.
  • upgradeable - The issue can be fixed by upgrading to a later version of the dependency.
  • patchable - The issue is patchable through Snyk.
  • monitored - The project is actively monitored by Snyk.

Here is an example.

snyk_vulnerabilities_total{organization="my-org",project="my-app",severity="critical",issue_type="vuln",issue_title="Remote Code Execution",ignored="false",upgradeable="false",patchable="false",monitored="true"} 1.0
snyk_vulnerabilities_total{organization="my-org",project="my-app",severity="high",issue_type="vuln",issue_title="Privilege Escalation",ignored="false",upgradeable="false",patchable="false",monitored="true"} 1.0
snyk_vulnerabilities_total{organization="my-org",project="my-app",severity="low",issue_type="vuln",issue_title="Sandbox (chroot) Escape",ignored="true",upgradeable="false",patchable="false",monitored="false"} 2.0
snyk_vulnerabilities_total{organization="my-org",project="my-app",severity="medium",issue_type="license",issue_title="MPL-2.0 license",ignored="true",upgradeable="false",patchable="false",monitored="true"} 1

Build

The exporter can be build using the standard Go tool chain if you have it available.

go build

You can build inside a docker image as well. This produces a snyk_exporter image that can run with the binary as entry point.

docker build -t snyk_exporter .

This is useful if the exporter is to be depoyled in Kubernetes or other dockerized environments.

Here is an example of running the exporter locally.

$ docker run -p9532:9532 snyk_exporter --snyk.api-token <api-token>
time="2019-01-11T09:42:34Z" level=info msg="Starting Snyk exporter for all organization for token" source="main.go:55"
time="2019-01-11T09:42:34Z" level=info msg="Listening on :9532" source="main.go:63"
time="2019-01-11T09:42:35Z" level=info msg="Running Snyk API scraper for organizations: <omitted>" source="main.go:106"

Deployment

To deploy the exporter in Kubernetes, you can find a simple Kubernetes deployment and secret yaml in the examples folder. You have to add your snyk token in the secrets.yaml and/or the snyk organizations that you want to get metrics from in the args section of the deployment.yaml. If you don't specify a snyk-organization, the exporter will scrape all organizations the token provides access to. The examples assumes that you have a namespace in kubernetes named: monitoring.

It further assumes that you have kubernetes service discovery configured for you Prometheus instance and a target that will gather metrics from pods, similar to this:

- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)

To deploy it to your kubernetes cluster run the following commands:

kubectl apply -f examples/secrets.yaml
kubectl apply -f examples/deployment.yaml

The exporter expose http endpoints that can be used by kubernetes probes:

  • /healthz - used for liveness probe, always returns healthy, status code 200.
  • /ready - used for readiness probe, return true and status code 200 after the first scrape completed. Otherwise, it returns false, with status code 503.

Development

The project uses Go modules so you need Go version >=1.11 to run it. Run builds and tests with the standard Go tool chain.

go build
go test

Credits

This exporter is written with inspiration from dnanexus/prometheus_snyk_exporter.

Main difference is the aggregations are done by Prometheus instead of in the exporter. It also scrapes the Snyk API asyncronously, ie. not when Prometheus tries to scrape the metrics.

snyk_exporter's People

Contributors

crevil avatar dantarian avatar dependabot[bot] avatar hoeg avatar kaspernissen avatar lunar-renovate avatar mahlunar avatar nixboot avatar omerlh avatar opdude avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

snyk_exporter's Issues

Organization name can change when using --snyk.organization and all organizations

If you specify an organization through the flags, that name will be used for the metrics. Here you need to specify the organizations original name, opposed to the display name.

We tag metrics with the original name in these cases.

If you run the exporter without the organization flag, we export all organizations related to the token. Here we lavel metrics with the display name instead which can lead to confusing behaviour.

snyk_vulnerabilities_total{organization="my-org",project="my-app",severity="low",type="Sandbox (chroot) Escape"} 2.0
snyk_vulnerabilities_total{organization="my-org",project="My App",severity="low",type="Sandbox (chroot) Escape"} 2.0

Application exits on timeouts

It should be expected to see request timeouts once in a while and the application should just ignore these as another scrape will occur in the near future anyway.

Here is an example of the log line just before an exit.

time="2019-02-23T10:34:32Z" level=error msg="Snyk exporter exited due to error: Get https://snyk.io/api/v1/org/UUID/projects: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" source="main.go:83"

package the binary in a docker container

Wrapping the binary in a docker container would make the usability of the project better. A simple example of a docker run with the arguments to deploy and run this container.

Upgrading to v1.3.1 produces unexpected crashes

Describe the bug
Upgrading the exporter from v1.2.0->v1.3.1 is causing it to crash multiple times a day. The exporter is running on kubernetes and crashes when it's liveness probe tries to access /healthz and gets the following error - net/http: request canceled (Client.Timeout exceeded while awaiting headers).

To Reproduce
Steps to reproduce the behavior:

  1. Publish the exporter as a k8s pod, with /healthz configured as the liveness probe
  2. Wait for a few hours until the pod crashes.

Expected behavior
The container health check should either return a healthy response or produce a descriptive error of the cause for crashing.

Application (please complete the following information):

  • Distribution: Docker image
  • Version 1.3.1

Log output (if applicable):
This log record is printed just before the pod crashes, but it's probably the sigterm received from k8s causing it -

time="2019-07-15T07:24:58Z" level=error msg="Snyk exporter exited due to error: received os signal 'terminated'" source="main.go:119"

Additional context
Exporter is running fine on v1.2.0. I will try to bisect the issue to a specific release and perhaps narrow it down to a specific feature by gradually incrementing the exporter version and update when I have more info.

Add support for isUpgradeable/isPatchable

Snyk API exposes 2 properties that can help devs understand if the issue is actionable - isUpgradeable and isPatchable. I think this can be nice to expose them also on the exporter.

Don't exit on errors

Describe the bug
We see that the exporter exits regularly due to some intermediate errors or issues at the Snyk APIs. This is generally an odd behaviour and should instead just be logged, as the exporter is not failing in its operation.

Here is a common error we see due to connection closing:

Get https://snyk.io/api/v1/org/UUID/projects: read tcp IP:36810-\u003eIP:443: read: connection reset by peer

To Reproduce

Add a test case for above error to TestPoll.

{
	name: "tcp connection reset by peer",
	collectorErr: &url.Error{
		Op:  "GET",
		URL: "/url",
		Err: &net.OpError{
			Op:  "read",
			Net: "tcp",
			Addr: &net.IPAddr{
				IP: net.IPv4zero,
			},
			Err: &net.OpError{
				Op:  "read",
				Err: syscall.ECONNRESET,
			},
		},
	},
	output: nil,
},

Expected behavior
The error should just be logged as above but the application should keep on running.

Application (please complete the following information):

  • Distribution: Docker image
  • Version 1.4.0

Log output (if applicable):

{"level":"error","msg":"Snyk exporter exited due to error: organization NAME (UUID): Get https://snyk.io/api/v1/org/UUID/projects: read tcp IP:36810-\u003eIP:443: read: connection reset by peer","source":"main.go:121","time":"2019-10-04T10:08:23Z"}

Duplicate metrics upon ignore

Hi,

After ignoring a vulnerability the exporter adds another gauge with the ignore: true label, but doesn't remove the old metric with ignore: false.

It requires server restart to get only the updated metric.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

  • Update actions/setup-go action to v4
  • Update goreleaser/goreleaser-action action to v4

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

Other Branches

These updates are pending. To force PRs open, click the checkbox below.

  • Update actions/checkout action to v3.5.3

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

dockerfile
Dockerfile
  • golang 1.19.3
Dockerfile-goreleaser
  • alpine 3.16.3
github-actions
.github/workflows/build.yml
  • actions/checkout v3.1.0
  • actions/setup-go v3
  • actions/cache v3
.github/workflows/codeql-analysis.yml
  • actions/checkout v3.1.0
  • github/codeql-action v2
  • github/codeql-action v2
  • github/codeql-action v2
.github/workflows/release-drafter.yml
  • release-drafter/release-drafter v5
.github/workflows/release.yml
  • actions/checkout v3.1.0
  • actions/setup-go v3
  • actions/cache v3
  • docker/login-action v2
  • goreleaser/goreleaser-action v3
gomod
go.mod
  • github.com/prometheus/client_golang v1.8.0
  • github.com/prometheus/common v0.15.0
  • gopkg.in/alecthomas/kingpin.v2 v2.2.6

Add /healthz endpoint for kubernetes probes

For easy kubernetes monitoring, we need to provide readyness and liveness probes. The /healthz should always return true, I would like to have /ready that returns true only if the initial scrap completed. So the pod will get traffic only when it finish scarping, and expose the correct metrics

Filter ignored issues

Currently, the request to get issues does not filter ignored issues:

postData := issuesPostData{
		Filters: issueFilters{
			Severities: []string{
				"high", "medium", "low",
			},
		},
	}

See the API documentation for more details. I'm working on a fix PR.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.