With the current state of affairs, a collector has to know exactly the names of the metrics (including the label names for each) it will expose. The only way around that is the SetMetricFamilyInjectionHook function. However, it requires the user to assemble a dto.MetricFamily protobuf, i.e. it is very low-level.

Two approaches come to my mind (which we might want to implement both):

(1) A "prefix" registration: A collector can register a whole namespace (by prefix, could be simply implemented by putting "prefix*" as the name in the descriptor). It would still register for a fixed set of label names. That registration would disallow any other registration matching the prefix. With this approach, full consistency can still be guaranteed. The planned scrape of selected metrics would be a bit more expensive as we have to call the collector whenever a metric is requested that matches the prefix, but that's fundamentally unavoidable if we have dynamic metric names.

(2) An "unchecked" registration: A collector is allowed to collect anything (possibly with the restriction on a name prefix aka namespace). Since metrics with any label set could be collected, consistency guarantees are gone. This would be similar to the SetMetricFamilyInjectionHook, but on a higher level, i.e. the user can collect normal Metric implementations and does not have to bother with low-level protobufs.

Currently, Fingerprint.LoadFromMetric crashes if the Metric contains an empty label (see prometheus/prometheus#374). Should we support explicit empty labels?

Summaries and histograms implicitly create time series named xxx_count and xxx_sum. In addition, histograms implicitly create time series named xxx_bucket. While this should not strictly be a concern of the client (in principle, a future version of the Prometheus server could treat summaries and histograms as 1st class types that do not get dissected into a number of series with the names described above), we should nevertheless avoid name collisions. Thus:

Once a summary or histogram named xxx is registered, any metric named xxx_count, xxx_sum, or xxx_bucket (in case of histograms) must be rejected (and vice versa).

In anticipation of prometheus/pushgateway#18, we want the functions pushing to pushgateway to look like

Push(job string, grouping map[string]string, url string)

grouping can be empty or nil, in which case, only the job is used. Initially, the only supported grouping other than that will be {"instance": "foo"}. Once the pushgateway supports arbitrary grouping, the grouping can contain anything except job. To avoid creating inconsistent metrics on the pushgateway (which will lead to an error on the pushgateway itself), one of the label names used in the grouping, and neither job, must be used by any of the pushed metrics directly. (The functions will check for that.)

There will be a convenience function to push with the own hostname as instance:

prometheus.Push("my_job", prometheus.GroupByHostname(), "pushgateway:9091")

I would like to track process stats per machine. Currently I am forced to rewrite entire ProcessCollector logic.

Dedicated ProcessCollector options would be enough imo.

type ProcessCollectorOpts struct {
    PID int
    Namespace string
    Subsystem string
    ConstLabels Labels
}

Otherwise you will not get any time series at all for _count and _sum before the first observation. @discordianfish encountered that when instrumenting Docker...

Hi,

I just realized (after quite some time debugging) that I can't just call registry.Register() multiple times with the same metric name. I expected this to be idempotent but that doesn't seem to be the case. More over, the behavior is quite surprising: All metrics just become stale, although calling Set() for a metric is still working fine - it's just not updating the exposed state.

As a example, this is a fix to not call Register() multiple times in my docker-exporter:

discordianfish/docker-exporter@778cbb9

Before that, all metrics were stale after the first iteration. Now it seems to work fine.

I want to setup multiple endpoints on one host that a Prometheus server can scrape. I want to register a separate set of metrics for each endpoint, but there is only one (?) global registry for registering collectors. Is this correct or am I overlooking something?

Similar to the Java simple client, we want a top level function "push this collector to a pushgateway". That would also work around the problem that the existing Push() will push everything in the default registry, which by now includes a lot of default metrics.

process_goroutines is a Go specific metrics. It should probably be renamed to go_goroutines in https://github.com/prometheus/client_golang/blob/master/prometheus/go_collector.go#L19 to match go_gc_duration_seconds.

The Collector interface describes Describe() as:

Describe sends the super-set of all possible descriptors of metrics
collected by this Collector to the provided channel and returns once
the last descriptor has been sent.

In reality this often is hard to achieve, therefor we have lot of exporters (e.g graphite exporter) simply not doing that but returning just a subset of possible metric descriptors.

Even to me it's still unclear why this is necessary and how does it help with consistency given that it doesn't enforce the described requirements.

It would be great if InstrumentHandler could not only track completed requests by http_requests_total, but also have a metric of how many requests were received, e.g. http_requests_received. Then one could calculate http_requests_received - http_requests_total to get the number of concurrent requests that have not been processed yet.

Just as time, our custom duration format should also be in the model. Especially for its parsing and marshalling functionality.

We should drop y and w as suffixes. They are rarely used and ambiguous. d is a unit sufficient to express long time intervals, i.e. everybody has a good notion of what 7, 14, 90, or 360 days are.

Overview

• client_golang: Near top-level directory with documentation, license, and friends.
  • client: Almost everything that is inside of present client_golang.
  • examples: Everything inside of client_golang/examples.

Requirements

• Maintaining API compatibility with legacy clients for one month. We will keep skeletons that divert to the authoritative types and spit out warnings once.

This test is always failing in our linux environment using go 1.4.2

input := []map[string]string{nil, {}}

var ms runtime.MemStats
runtime.ReadMemStats(&ms)

alloc := ms.Alloc

for _, labels := range input {
LabelsToSignature(labels)
}

runtime.ReadMemStats(&ms)

if got := ms.Alloc; alloc != got {
t.Fatal("expected LabelsToSignature with empty labels not to perform allocations")
}

I'm writing a proxy that ingests Heroku router/runtime metrics logs and converts it to prometheus-ingestable metrics. (See my fork of lumbermill.)

I thought it'd be nice to output metrics in the same form as the ones used elsewhere in Prometheus, ex http_request_size_bytes with the relevant upstream job, instance and endpoint tags. (Though ideally I'd like to include HTTP Method and response code).

Trying to register a handler for this:

httpResponseSizeBytes = prometheus.NewSummaryVec(prometheus.SummaryOpts{
    Name: "http_request_size_bytes",
    Help: "The HTTP response sizes in bytes.",
}, []string {"job", "instance", "handler"})

func init () {
    prometheus.MustRegister(httpResponseSizeBytes)
}

Conflicts with the existing handler for the very same data:

panic: a previously registered descriptor with the same fully-qualified name as Desc{fqName: "http_request_size_bytes", help: "The HTTP request sizes in bytes.", constLabels: {handler="prometheus"}, variableLabels: []} has different label names or a different help string

I've tried unregistering the existing SummaryVec, but to no avail:

func drain(...) {
    prometheus.Unregister(prometheus.NewSummary(prometheus.SummaryOpts{
        Name:        "http_response_size_bytes",
        Help: "The HTTP response sizes in bytes.",
        ConstLabels: prometheus.Labels{"handler": "prometheus"},
    }))
    prometheus.MustRegister(httpResponseSizeBytes)
}

Detects that a handler with a different set of labels has been there previously, and more or less gives me the same panic as above.

Will sync.atomic help? Investigate.

Currently we have different methods to extract signatures from labels.

func LabelsToSignature(labels map[string]string) uint64
func SignatureForLabels(m Metric, labels LabelNames) uint64
func SignatureWithoutLabels(m Metric, labels map[LabelName]struct{}) uint64

Those should all accept model.LabelSet. I can easily convert from Metric to LabelSet but have to do a full copy to go from Metric or LabelSet to map[string]string.
LabelsToSignature is used in the text parser. We are going out of our way to use all our custom types everywhere. There we are doing the inverse. Is this for a particular reason?

For SignatureWithoutLabels:

In the general case callers will have a slice of labels rather than a map[string]struct{} set. The implementation transforms the map into a slice again anyway. The method works just fine, even if there were duplicates.
I see no good reason to enforce this. Callers should make sure the labels are unique (e.g., ensured by the parser).

I'd also consider making the labelname slices variadic.

Not super urgent but I feel like those exported functions were created on demand of local internal uses and from the outside they seem a bit arbitrary.

From reading the code, it seems like it's not possible to create summaries without any quantiles at the moment:

https://github.com/prometheus/client_golang/blob/master/prometheus/summary.go#L183-L185

If that len(...) == 0 check was changed to a ... == nil, then I guess it should work (one would then be able to pass in an empty, but constructed slice of objectives).

The same is true for histograms (although a summary/histogram without quantiles/buckets would behave identically to each other):

https://github.com/prometheus/client_golang/blob/master/prometheus/histogram.go#L184-L186

I seem to either misunderstand how the MaxAge/decay in summaries works or there might be a problem.

Latencies:

Observations:

As you can see from the second graph, there were actually no requests/observations for tens of minutes. Still the quantile timeseries were not reset at any point.

I was under the assumption, all values will be reset after MaxAge.

It's the third link in the README.md of this project:

https://github.com/prometheus/client_golang/blame/master/README.md#L11

For federation we can provide match[] query parameters with metric selectors to ask for specific metrics. By default none are shown.

Regular scrape targets should provide the same options but show all metrics by default.

AFAICS the new protobuf-based extraction processor simply overwrites the target's base labels on a sample if the target sets a conflicting label itself:

https://github.com/prometheus/client_golang/blob/master/extraction/metricfamilyprocessor.go#L76

...whereas the JSON-based processors avoid such collisions by adding an exporter_ prefix to the conflicting target labels (as specified in https://github.com/prometheus/prometheus/wiki/Automatic-Labels-and-Synthetic-Metrics):

https://github.com/prometheus/client_golang/blob/master/extraction/processor0_0_2.go#L71

This should probably be fixed for the protobuf processor as well.

As we known, people who are familier with the linux OS should be used to read data from native file on the system.
I have read the client_golang/prometheus/README.md file , that shows:
func main() {
http.Handle("/metrics", prometheus.Handler())

indexed.Inc()
size.Set(5)

http.ListenAndServe(":8080", nil)

}

In the example in the README.md, the data is exported to the localhost accroding the http package.
Is there a way to exported the data to a native file according the prometheus/client_golang, which api should be use? Or, is there a example for it?

thanks very much.

First of all, there is the public Registry interface tracked in #46

But there is more:

Two things come to my mind here, from the email thread:

(1)
Another utility function could compare two collectors (or a collector
to a list of template metrics) and verify that they return the same
set of metrics. In that way, you could simply create the expected
metrics and verify that the collector to be tested collects the same
metrics.
[...]
But one of my suggestions was to have a comparison utility function
that would abstract away the order (as the Prometheus registry does
it, too).
Something like

testutil.Collects(myCollector, []prometheus.Metric{
    prometheus.MustNewConstMetric(...),
}

(2)

func readCounter(m prometheus.Counter) float64 {
    pb := &dto.Metric{}
    m.Write(pb)
    return pb.GetCounter().GetValue()
}

Could be done in a generic way, like this:

package testutil

func ToFloat64(m prometheus.Metric) float64 {
    pb := &dto.Metric{}
    m.Write(pb)
    if pb.Gauge != nil {
        return pb.Gauge.GetValue()
    }
    if pb.Counter != nil {
        return pb.Counter.GetValue()
    }
    if pb.Untyped != nil {
        return pb.Untyped.GetValue()
    }
    return math.NaN()
}

The Makefile is only used to run the examples and tests. It's usefulness has been disputed. Even in our own public instructions to prepare for Prometheus workshops, we tell the users to run go get and go build instead of make.

So we could either get rid of the Makefile (with the only downside that the CI will fail all of a sudden if a breaking change of a dependency happens - which could also be considered an advantage), or we should at least bring the Makefile in line with other repos, i.e. use the common Makefile from the utils repo.

Similar to the ways we already have for gauges and counters to mirror an existing metric from a 3rd party system into client_golang, we need a way to generate summaries and histograms with predefined values for _count, _sum, and, if applicable, quantiles or buckets, respectively.

Not only the name, also the help string (and obviously type) of standard metrics exposed by every library (like process_open_fds) should be consistent. There are deviations at the moment, only detected by the pushgateway so far.

Check consistency and fix either here in client_golang or in the affected other library.

When including the pkg in a binary, additionally to the flags defined by my binary there is a list of unrelated ones:

  -gocheck.b=false: Run benchmarks
  -gocheck.btime=1s: approximate run time for each benchmark
  -gocheck.f="": Regular expression selecting what to run
  -gocheck.v=false: Verbose mode
  -gocheck.vv=false: Super verbose mode (disables output caching)
  -telemetry.abortonmisuse=false: abort if a semantic misuse is encountered (bool).
  -telemetry.debugregistration=false: display information about the metric registration process (bool).
  -telemetry.useaggressivesanitychecks=false: perform expensive validation of metrics (bool).
  -test.bench="": regular expression to select benchmarks to run
  -test.benchtime=1: approximate run time for each benchmark, in seconds
  -test.cpu="": comma-separated list of number of CPUs to use for each test
  -test.cpuprofile="": write a cpu profile to the named file during execution
  -test.memprofile="": write a memory profile to the named file after execution
  -test.memprofilerate=0: if >=0, sets runtime.MemProfileRate
  -test.parallel=1: maximum test parallelism
  -test.run="": regular expression to select tests and examples to run
  -test.short=false: run smaller test suite to save time
  -test.timeout=0: if positive, sets an aggregate time limit for all tests
  -test.v=false: verbose: print additional output

Given that the flags prefixed with telemetry are the knobs important to control the behaviour of the exporter, the rest seem unintentional. I haven't found the root of the problem yet. Is it a shared understanding that those should go away?

Currently, only the status code is included. But the body contains often a message that is helpful for understanding the problem (like a parse error in the pushed metrics).

Like the auto HTTP handler instrumentation, we want a similar tool for the client side of HTTP, done as a round tripper.

There exist an internal implementation at SoundCloud already. Take it from there.

I'm experimenting with using this library to instrument the core components of Kubernetes, and have found that it breaks compilation for a couple of the platforms that we target: linux/arm and linux/386 (although to be honest I'm not sure why we build for 386). I may be able to work around the issue within Kubernetes by fixing some unneeded dependencies, but wanted to separately report the issue in case it's not just due to my ineptitude.

This is the exact error I see:
Godeps/_workspace/src/github.com/prometheus/client_golang/prometheus/process_collector_procfs.go:40: p.NewStat undefined (type procfs.Proc has no field or method NewStat)

For context, we've vendored in client_golang and its dependencies using go dep with the most recent versions from github as of last Friday. For more context, here's a PR in which I attempt to add a couple metrics to the Kubernetes API server. You can reproduce the test of whether it builds on all platforms by running build/release.sh without the change I have that removes linux/arm and linux/386 from the list of target platforms.

I can help look into this if it's of interest to you, but without looking into it much I have no immediate guess at what the problem is.

They shouldn't be required anymore...

We need functionality to make allow simple serving of metrics via HTTP handlers that hides details such as content type negotiation, buffering, sorting, etc.

The prometheus golang client is very useful as a generic library to measure and expose metrics, even when the consumer is not prometheus. (It's being used in this way in the Kubernetes project.) In this case, or in any case where prometheus is not the sole consumer of an application's metrics, being able to consume the metrics from the HTTP endpoint as JSON dramatically helps the ability for tooling to consume and parse the metrics outputted by the golang client.

prom2json provides this functionality as a Go command-line tool, which only makes the functionality accessible to tools in other languages if you package and install the tool and call it in between HTTP calls to the client library. Integration directly into the client's metrics server would make things much easier.

x{a="bb",b="c} and x{a="b",bb="c"} have the same fingerprint with the current fingerprinting method.
Reason is that we have a separator between label name and label value, but not between whole label pairs.

This is easy to fix, but invalidates all storage. :-(((

Reported via the mailing list:

I went ahead and tried your simple example from the Go client and added a const label to the counter and the gauge. However, the counter "hd_errors" does not expose the const label but the gauge "cpu_temperature" does.

Am I missing something?

var (
    cpuTemp = prometheus.NewGauge(prometheus.GaugeOpts{
        Name:        "cpu_temperature_celsius",
        Help:        "Current temperature of the CPU.",
        ConstLabels: prometheus.Labels{"version": "1234"},
    })
    hdFailures = prometheus.NewCounter(prometheus.CounterOpts{
        Name:        "hd_errors_total",
        Help:        "Number of hard-disk errors.",
        ConstLabels: prometheus.Labels{"version": "1234"},
    })
)

# HELP cpu_temperature_celsius Current temperature of the CPU. 
# TYPE cpu_temperature_celsius gauge 
cpu_temperature_celsius{version="1234"} 65.3

# HELP hd_errors_total Number of hard-disk errors. 
# TYPE hd_errors_total counter 
hd_errors_total 1 

... rest omitted for brevity ...

docs: http://godoc.org/github.com/prometheus/client_golang/prometheus#Opts
full code: https://gist.github.com/magiconair/c1cf6860968bf0dea600#file-main-go
full output: https://gist.github.com/magiconair/c1cf6860968bf0dea600#file-out

I’m using this simple test program:

package main

import (
    "fmt"
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
)

func main() {
    http.Handle("/metrics", prometheus.Handler())
    go http.ListenAndServe(":8089", nil)

    i := 0
    for {
        prometheus.Register(prometheus.NewCounter(prometheus.CounterOpts{
            Subsystem: "promrace",
            Name:      fmt.Sprintf("counter_%d", i),
            Help:      "counter",
        }))
        i++
        time.Sleep(1 * time.Second)
    }
}

According to the example in https://godoc.org/github.com/prometheus/client_golang/prometheus#Register, registering new counters should work just fine after the HTTP server is already up and running.

However, when running the test program with go run -race and accessing /metrics repeatedly, e.g. in a browser, or with a benchmark program, I get warnings about data races:

==================
WARNING: DATA RACE
Read by goroutine 17:
  github.com/prometheus/client_golang/prometheus.(*registry).writePB()
      /home/michael/gocode/src/github.com/prometheus/client_golang/prometheus/registry.go:403 +0x85
  github.com/prometheus/client_golang/prometheus.(*registry).ServeHTTP()
      /home/michael/gocode/src/github.com/prometheus/client_golang/prometheus/registry.go:383 +0x15b
  net/http.Handler.ServeHTTP·fm()
      /home/michael/gocode/src/github.com/prometheus/client_golang/prometheus/http.go:58 +0x64
  github.com/prometheus/client_golang/prometheus.func·007()
      /home/michael/gocode/src/github.com/prometheus/client_golang/prometheus/http.go:144 +0x241
  net/http.HandlerFunc.ServeHTTP()
      /home/michael/go/src/net/http/server.go:1265 +0x4e
  net/http.(*ServeMux).ServeHTTP()
      /home/michael/go/src/net/http/server.go:1541 +0x20c
  net/http.serverHandler.ServeHTTP()
      /home/michael/go/src/net/http/server.go:1703 +0x1f6
  net/http.(*conn).serve()
      /home/michael/go/src/net/http/server.go:1204 +0x1087

Previous write by main goroutine:
  runtime.mapassign1()
      /home/michael/go/src/runtime/hashmap.go:376 +0x0
  github.com/prometheus/client_golang/prometheus.(*registry).Register()
      /home/michael/gocode/src/github.com/prometheus/client_golang/prometheus/registry.go:301 +0x1123
  github.com/prometheus/client_golang/prometheus.Register()
      /home/michael/gocode/src/github.com/prometheus/client_golang/prometheus/registry.go:114 +0x6d
  main.main()
      /home/michael/gocode/src/promrace/promrace.go:21 +0x288

Goroutine 17 (running) created at:
  net/http.(*Server).Serve()
      /home/michael/go/src/net/http/server.go:1751 +0x3cd
  net/http.(*Server).ListenAndServe()
      /home/michael/go/src/net/http/server.go:1718 +0x188
  net/http.ListenAndServe()
      /home/michael/go/src/net/http/server.go:1808 +0x103
==================

Hi,

While working on Debian packages for Prometheus, I've found a circular dependency between prometheus and client_golang. Prometheus depends on the whole client_golang, while the latter depends on utility/test/error.go from prometheus for some tests.

As we want to make writing exporters easier with the next version we should provide some helpers to generate a landing page. They are basically copy/paste for the existing exporters.

Hi,

We are migrating from influxdb + grafana to prometheus + grafana. But we have a problem on our Go scripts to expose/send our data to prometheus.

What i need:

Metric: Server memory (server-memory)
    [
    Label: server   Value: "vps001"
    Label: free   Value: 123
    ]

    [
    Label: server   Value: "vps001"
    Label: total   Value: 500
    ]

    [
    Label: server   Value: "vps001"
    Label: used   Value: 12
    ]

But i dont know how to do it with go client. My current code is:

{
    memoryGauge := prometheus.NewGauge(prometheus.GaugeOpts{
        Namespace: "prsolucoes",
        Name: "server_memory",
        Help: "Free memory of server",
        ConstLabels: prometheus.Labels{"type": "free"},
    })

    prometheus.MustRegister(memoryGauge)
    memoryGauge.Set(float64(v.Free));
}
{
    memoryGauge := prometheus.NewGauge(prometheus.GaugeOpts{
        Namespace: "prsolucoes",
        Name: "server_memory",
        Help: "Used memory of server",
        ConstLabels: prometheus.Labels{"type": "used"},
    })

    prometheus.MustRegister(memoryGauge)
    memoryGauge.Set(float64(v.Used));
}
{
    memoryGauge := prometheus.NewGauge(prometheus.GaugeOpts{
        Namespace: "prsolucoes",
        Name: "server_memory",
        Help: "Total memory of server",
        ConstLabels: prometheus.Labels{"type": "total"},
    })

    prometheus.MustRegister(memoryGauge)
    memoryGauge.Set(float64(v.Total));
}

It is not a bug, but after lost my day searching and not found the answer, i prefer post here.

...and presumably the same is true for buckets in Histograms.

See prometheus/pushgateway#51 for a detailed example.

To me the following code is simpler, more idiomatic, and gives the same results as the timer API:

func foo() {
    defer func(began time.Time) {
        xyz.IncrementBy(map[string]string{}, time.Since(began))
    }(time.Now())

    // do some work
}

Also, as @matttproud notes the current timer code fails in the presence of a panic, while this does not.

For comparison, see the current registry handler vs. a version using the above pattern.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:    14.04
Codename:   trusty
$ go version
go version go1.2.1 linux/amd64
$ go get github.com/prometheus/client_golang
/home/tmp/src/github.com/prometheus/client_golang/model/signature.go:32: undefined: sync.Pool

looks like sync.Pool was added in Go1.3, https://golang.org/doc/go1.3

For testing purposes and for advanced configuration (and possibly other uses), we want a public Registry interface with the possibility of plugging together implementations that behave according to the user's requirements (error handling, which standard metrics to export, http instrumentation, exposition - which might or might not be via HTTP, ...).

The Registry interface should be lean, possibly with parts of the functionality delegated to other interfaces or even modules (HTTP serving and/or instrumentation, Pushgateway handling, ...).

Currently, there is only one global registry, and it is not exposed to the user. It is used only via global convenience methods. We want to avoid a further explosion of the number of those functions. The good news is that for as long we support the existing functions, we can change everything behind it at will. Basic idea is to keep the existing convenience functions (with the option of deprecating those that are not used a lot), while providing advanced functionality via a more elegant but also more complex to use API (where you create a Registry implementation first that is configured to your needs and then register metrics there and decide about HTTP exposition and such). For inspiration, see #38 (code and discussion).

The convenience functions to keep are:

• EnableCollectChecks
• UninstrumentedHandler
• Handler
• InstrumentHandler
• InstrumentHandlerFunc
• InstrumentHandlerWithOpts
• InstrumentHandlerFuncWithOpts
• PanicOnCollectError
• Push
• PushAdd
• SetMetricFamilyInjectionHook
• Unregister
• Registe
• RegisterOrGet
• MustRegister
• MustRegisterOrGet

Examples:
https://github.com/mesosphere/mesos_exporter/blob/master/master.go#L75
https://github.com/google/cadvisor/blob/master/metrics/prometheus.go#L91

Is there a better way?

Prometheus's exposition format has an optional timestamp field which can be used to set the time a metric was collected (with some caveats as to the danger). This interface should be exposed by the client library so it can be set.

A classic use case is when a service might poll during the collection of the metric, but you want to prevent flooding by limiting the maximum poll rate to some value - in these cases, it would be desirable to send the exact time the metric was collected last while still serving the metric. This allows many scrapes, but only one poll of the underlying metric.

If the /metrics endpoint is scraped too often concurrently, too much resources are claimed and the monitored binary might OOM because of a rogue monitoring server...

Reducing the resource usage of an individual scrape (e.g. by streaming more and buffering less as we do currently) increases the number of scrapes that can be performed in parallel, but doesn't fundamentally solve the problem. Adding "one more test monitoring server" could still crash production binaries. The HTTP exposition needs to be configurable in terms of max scrapes in flight and scrape timeout.