cistern / cistern Goto Github PK

Event aggregation and indexing system

License: MIT License

Go 91.61% JavaScript 7.63% CSS 0.44% Shell 0.33%

go cloudwatch-logs vpc-flow-logs cistern events time-series aws

cistern's Introduction

Cistern is an event aggregation and indexing system. Cistern consumes VPC Flow Logs and JSON events from AWS CloudWatch Logs and exposes a SQL-like querying interface.

Supported sources

CloudWatch Logs
- VPC Flow Logs
- JSON CloudWatch Logs events

Coming soon:

sFlow v5

Documentation

The official documentation is available on the Cistern website.

License

MIT (see LICENSE)

cistern's People

Contributors

Stargazers

Watchers

Forkers

karlpilkington gitter-badger jmptrader digideskio

cistern's Issues

Add basic flow detection

Just need to be able to classify and aggregate flows

Panic

panic: runtime error: slice bounds out of range

goroutine 28 [running]:
runtime.panic(0x6a0860, 0x86648f)
    /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/PreetamJinka/protodecode.DecodeTCP(0xc2080174c2, 0x28, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/preetam/go/src/github.com/PreetamJinka/protodecode/tcp.go:90 +0x418
main.(*RawPacketProcessor).Process(0xc2080385c0)
    /home/preetam/git/cistern/rawpacketprocessor.go:59 +0x303
created by main.(*Pipeline).Run
    /home/preetam/git/cistern/pipeline.go:21 +0xc3

goroutine 16 [chan receive]:
main.main()
    /home/preetam/git/cistern/main.go:85 +0x9b1

goroutine 19 [finalizer wait, 2 minutes]:
runtime.park(0x41ba30, 0x86a490, 0x868729)
    /usr/local/go/src/pkg/runtime/proc.c:1354 +0x89
runtime.parkunlock(0x86a490, 0x868729)
    /usr/local/go/src/pkg/runtime/proc.c:1370 +0x3b
runfinq()
    /usr/local/go/src/pkg/runtime/mgc0.c:2624 +0xcf
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1430

goroutine 20 [IO wait]:
net.runtime_pollWait(0x7f015c867698, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc20802a140, 0x72, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc20802a140, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc20802a0e0, 0xc2080a0000, 0x2710, 0x2710, 0x0, 0x7f015c8662a8, 0xb)
    /usr/local/go/src/pkg/net/fd_unix.go:232 +0x30e
net.(*conn).Read(0xc208032038, 0xc2080a0000, 0x2710, 0x2710, 0x2710, 0x0, 0x0)
    /usr/local/go/src/pkg/net/net.go:122 +0xe7
github.com/PreetamJinka/udpchan.func·003()
    /home/preetam/go/src/github.com/PreetamJinka/udpchan/udpchan.go:66 +0x101
created by github.com/PreetamJinka/udpchan.Listen
    /home/preetam/go/src/github.com/PreetamJinka/udpchan/udpchan.go:74 +0x1d5

goroutine 30 [chan receive]:
main.func·005()
    /home/preetam/git/cistern/rawpacketprocessor.go:36 +0x72
created by main.(*RawPacketProcessor).Process
    /home/preetam/git/cistern/rawpacketprocessor.go:40 +0x68

goroutine 23 [IO wait, 2 minutes]:
net.runtime_pollWait(0x7f015c8675e8, 0x72, 0x0)
    /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc20802a060, 0x72, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc20802a060, 0x0, 0x0)
    /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc20802a000, 0x769380, 0x0, 0x7f015c8662a8, 0xb)
    /usr/local/go/src/pkg/net/fd_unix.go:410 +0x2fe
net.(*TCPListener).AcceptTCP(0xc208032070, 0x7f015c6e0e18, 0x0, 0x0)
    /usr/local/go/src/pkg/net/tcpsock_posix.go:233 +0x59
net/http.tcpKeepAliveListener.Accept(0xc208032070, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1949 +0x6f
net/http.(*Server).Serve(0xc20802e3c0, 0x7f015c867920, 0xc208032070, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1700 +0x91
net/http.(*Server).ListenAndServe(0xc20802e3c0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1690 +0x11d
net/http.ListenAndServe(0x6d54b0, 0x5, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/pkg/net/http/server.go:1780 +0x79
created by main.main
    /home/preetam/git/cistern/main.go:45 +0x8cb

goroutine 24 [chan send]:
main.func·004()
    /home/preetam/git/cistern/main.go:77 +0x2cf
created by main.main
    /home/preetam/git/cistern/main.go:82 +0x954

goroutine 25 [chan receive]:
main.(*HostProcessor).Process(0xc208038560)
    /home/preetam/git/cistern/pipeline.go:77 +0x6e
created by main.(*Pipeline).Run
    /home/preetam/git/cistern/pipeline.go:21 +0xc3

goroutine 26 [chan receive]:
main.(*GenericIfaceProcessor).Process(0xc208038580)
    /home/preetam/git/cistern/pipeline.go:165 +0x6e
created by main.(*Pipeline).Run
    /home/preetam/git/cistern/pipeline.go:21 +0xc3

goroutine 27 [chan receive]:
main.(*BlackholeProcessor).Process(0xc208032060)
    /home/preetam/git/cistern/pipeline.go:47 +0x53
created by main.(*Pipeline).Run
    /home/preetam/git/cistern/pipeline.go:24 +0x10f

goroutine 29 [chan receive]:
main.(*BlackholeProcessor).Process(0xc208032068)
    /home/preetam/git/cistern/pipeline.go:47 +0x53
created by main.(*Pipeline).Run
    /home/preetam/git/cistern/pipeline.go:24 +0x10f

cisternproject.com is down

Hi, cisternproject.com seems down with cisternproject.com’s server DNS address could not be found error.

Refactor flow aggregation

Flows are identified with the following fields:

protocol
address_a
port_a
address_b
port_b

The first address is whichever one is first when ordered byte-wise. This is good for identifying flows (which can go in either direction), but the statistics aggregation should happen separately.

The last thing is that we should probably ignore "flows" with only one entry. We can't calculate derivatives with only one entry, so it doesn't make sense to rank them either.

Metrics state being lost on restarts

It looks like all of the metrics are disappearing whenever I restart.

Support adding events over HTTP

Use Honeycomb's API spec.

https://honeycomb.io/docs/reference/api/#events

Get rid of submodules

Things will start to get really weird if there are cross-package references. jd should be used.

Rewrite

Probably should rewrite everything to clean up the code.

Snapshotted metrics are incorrect

Right now, snapshotting a metric so it can be inserted into the series engine means capturing the current value of a metric. This isn't right because it's getting the instantaneous value of the metric and not something like the average since the last time it was snapshotted.

If we're getting counter records every 15 seconds but we snapshot every 30 seconds, we're only recording the rate over the past 15 seconds. It should instead be the rate over the past 30 seconds, which is the interval since we last snapshotted.

The derivative code needs to be refactored to support the correct behavior. See https://github.com/PreetamJinka/cistern/blob/c7cc1009f0f0dc24430d6795d7c2eaa8e46767d1/state/metrics/state.go#L13.

Add more filter operators

Greater than
Less than
Regular expressions match
Symbolic operators (>, >=, <, <=, !=, =, etc.)

Error reporting from the UI

The UI doesn't report any errors right now.

time.Now() called too often

This one is particularly bad. We can just update a device-level timestamp whenever it receives a new datagram.

Panic

unexpected fault address 0x7f1a2bca9525
fatal error: fault
[signal 0xb code=0x1 addr=0x7f1a2bca9525 pc=0x43149b]

goroutine 28 [running]:
runtime.throw(0x836b02)
    /usr/local/go/src/pkg/runtime/panic.c:520 +0x69 fp=0xc20809faa0
runtime.sigpanic()
    /usr/local/go/src/pkg/runtime/os_linux.c:240 +0x13f fp=0xc20809fab8
runtime.cmpbody()
    /usr/local/go/src/pkg/runtime/asm_amd64.s:1226 +0xcb fp=0xc20809fac0
bytes.Compare(0x7f1a2bca9525, 0x2b, 0x557af3, 0xc2080a0368, 0x4, 0x8, 0x3c)
    /usr/local/go/src/pkg/runtime/asm_amd64.s:1148 +0x19 fp=0xc20809fac8
main.(*MetricStorage).String(0xc208032038, 0x0, 0x0)
    /home/preetam/git/cistern/state.go:54 +0x1d0 fp=0xc20809fce0
main.func·001(0x7f1a2d499008, 0xc2080400a0, 0xc208028270)
    /home/preetam/git/cistern/http.go:9 +0x29 fp=0xc20809fd20
net/http.HandlerFunc.ServeHTTP(0xc208000280, 0x7f1a2d499008, 0xc2080400a0, 0xc208028270)
    /usr/local/go/src/pkg/net/http/server.go:1237 +0x40 fp=0xc20809fd40
net/http.serverHandler.ServeHTTP(0xc20802e2a0, 0x7f1a2d499008, 0xc2080400a0, 0xc208028270)
    /usr/local/go/src/pkg/net/http/server.go:1675 +0x18c fp=0xc20809fd88
net/http.(*conn).serve(0xc20804c280)
    /usr/local/go/src/pkg/net/http/server.go:1176 +0x9a1 fp=0xc20809ffa0
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1430 fp=0xc20809ffa8
created by net/http.(*Server).Serve
    /usr/local/go/src/pkg/net/http/server.go:1723 +0x2ea

Document v0.2.0 features

Complete before release.

Graphite support

We should be able to export metrics data using the ~~StatsD~~ Graphite protocol.

Use external time series DB

Cistern should not be managing time series data. It should connect to an sdb instance and query it.

Missing data

I think I'm missing some metrics, and sometimes hosts don't show up.

Query multiple metrics

Use updated sFlow version

[queries] limit not enforced when there is no group by or aggregates

Probably should be done within this block.

Create API spec

Decode/encode MIB strings

Rethink pipelines

Support new host CPU metrics

See Cistern/sflow#21.

Refactor classification and discovery of devices

(Details coming soon...)

Use Bolt

Add minimal dashboard

This should be a dashboard with...

Fields to set a time range, and a field for a query string
Sets of time series charts (optional).
- One chart per aggregation.
- One path plotted for each group.
A table with summary data

Add group IDs to query results

E.g.

{
  "summary": [
    {
      "_group_id": "6",
      "protocol": 6,
      "sum(bytes)": 10326988
    },
    {
      "_group_id": "17",
      "protocol": 17,
      "sum(bytes)": 456
    },
    {
      "_group_id": "1",
      "protocol": 1,
      "sum(bytes)": 136
    }
  ],
  "series": [
    {
      "_group_id": "6",
      "_ts": "2017-07-25T00:00:00Z",
      "protocol": 6,
      "sum(bytes)": 2497
    },
    {
      "_group_id": "17",
      "_ts": "2017-07-26T00:00:00Z",
      "protocol": 17,
      "sum(bytes)": 76
    }
  ]
}

Helps with processing the output.


root@ubuntu:~# ./cistern
2017/01/04 17:27:04 main.go:54: Cistern version 0.0.4 starting
2017/01/04 17:27:04 clock.go:13:   Starting internal clock
2017/01/04 17:27:04 main.go:57:   Attempting to load configuration file at /opt/cistern/config.json
2017/01/04 17:27:04 main.go:60: ✗ Could not load configuration: `open /opt/cistern/config.json: no such file or directory`
2017/01/04 17:27:04 main.go:71: ✓ Successfully loaded configuration
2017/01/04 17:27:04 service.go:54: listening for sFlow datagrams on :6343
2017/01/04 17:27:04 service.go:59: listening for AppFlow datagrams on :6344
2017/01/04 17:27:04 main.go:90: ✓ Successfully started network service
2017/01/04 17:27:09 service.go:73: 10.0.3.15 is unknown. Registering new source.
2017/01/04 17:27:09 service.go:82: Source{10.0.3.15} needs class "sflow".
2017/01/04 17:27:09 source.go:34: registering class sflow for Source{10.0.3.15}
2017/01/04 17:27:09 source.go:66:   Source{10.0.3.15} does not have class "switch-counters" registered
2017/01/04 17:27:09 source.go:34: registering class switch-counters for Source{10.0.3.15}
2017/01/04 17:27:09 source.go:66:   Source{10.0.3.15} does not have class "metrics" registered
2017/01/04 17:27:09 source.go:34: registering class metrics for Source{10.0.3.15}
2017/01/04 17:27:09 engine.go:51: [Series engine] Writing 14 observations
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4a98de]

goroutine 1 [running]:
panic(0x7b6180, 0xc82000a0f0)
	/usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/Cistern/cistern/state/series.(*Engine).writeObservations(0x0, 0xc8200da000, 0xe, 0x10)
	/Users/alexanderturner/Code/gocode/src/github.com/Cistern/cistern/state/series/engine.go:52 +0xde
github.com/Cistern/cistern/state/series.(*Engine).Process(0x0, 0xc820016a80)
	/Users/alexanderturner/Code/gocode/src/github.com/Cistern/cistern/state/series/engine.go:47 +0xae
main.main()
	/Users/alexanderturner/Code/gocode/src/github.com/Cistern/cistern/main.go:96 +0x1094

Use SQLite for general purpose data

By general purpose data I mean things like

Users
Authentication parameters
Events
Configurations

etc.