segmentio / cwlogs Goto Github PK
View Code? Open in Web Editor NEWCLI tool for reading logs from Cloudwatch Logs
License: MIT License
CLI tool for reading logs from Cloudwatch Logs
License: MIT License
It's taking several minutes to get any logs back. This makes the tool pretty unusable, as I can stumble around the CloudWatch's clunky UI more quickly than that.
∴ time aws-vault exec prod -- cwlogs fetch integration-drip-v1
^C
real 2m24.997s
user 0m8.905s
sys 0m0.771s
Is there anything we can do to improve performance here?
The --until
flag doesn't seem to be working. I've tried both --until 20m
and --until 2017-07-05T09:00:00
but it always fetches until now.
To run cwlogs I have to export the AWS_REGION
variable. It should use what's configured in the AWS profile I'm using, or default to us-east-1
.
I was grabbing logs from a service and hit this:
panic: reflect: call of reflect.Value.Type on zero Value [recovered]
panic: reflect: call of reflect.Value.Type on zero Value
goroutine 1 [running]:
text/template.errRecover(0xc42062f858)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:146 +0x62
panic(0x144e640, 0xc42032bfe0)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/panic.go:489 +0x2cf
reflect.Value.Type(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/reflect/value.go:1688 +0x21f
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.FlattenPrefixedToResult(0x0, 0x0, 0xc42032bf60, 0x17, 0xc4206e8450)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:72 +0xee
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.FlattenPrefixedToResult(0x14545c0, 0xc42067cf30, 0xc4211b9630, 0xc, 0xc4206e8450)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:81 +0x588
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.FlattenPrefixedToResult(0x1471960, 0xc42067ced0, 0x0, 0x0, 0xc4206e8450)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:81 +0x588
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.FlattenPrefixed(0x1471960, 0xc42067ced0, 0x0, 0x0, 0x13e5d40)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:56 +0x87
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.Flatten(0x1471960, 0xc42067ced0, 0x104d693)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:51 +0x47
github.com/segmentio/cwlogs/lib.Event.DataFlat(0x7, 0xed19e9e47, 0xf42400, 0x0, 0xc4211a8b20, 0xc, 0x0, 0x0, 0x0, 0x0, ...)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/lib/event.go:76 +0x3a
github.com/segmentio/cwlogs/lib.(*Event).DataFlat(0xc420a17800, 0x0)
<autogenerated>:5 +0x80
reflect.Value.call(0x14b2560, 0xc420a17800, 0x693, 0x14cc766, 0x4, 0x174d078, 0x0, 0x0, 0x5522, 0x13eb262, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0x14b2560, 0xc420a17800, 0x693, 0x174d078, 0x0, 0x0, 0xc4203da840, 0xc4203da840, 0xc420976260)
/usr/local/Cellar/go/1.8.3/libexec/src/reflect/value.go:302 +0xa4
text/template.(*state).evalCall(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x14b2560, 0xc420a17800, 0x693, 0x1703d00, 0xc42016c660, 0x14e38dd, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:668 +0x51f
text/template.(*state).evalField(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x14e38dd, 0x8, 0x1703d00, 0xc42016c660, 0xc420014980, 0x1, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:558 +0xe18
text/template.(*state).evalFieldChain(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x14b2560, 0xc420a17800, 0x99, 0x1703d00, 0xc42016c660, 0xc420014850, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:526 +0x22b
text/template.(*state).evalFieldNode(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0xc42016c660, 0xc420014980, 0x1, 0x1, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:490 +0x118
text/template.(*state).evalCommand(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0xc42016c630, 0x0, 0x0, 0x0, 0x16, 0x3, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:428 +0x837
text/template.(*state).evalPipeline(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0xc420168230, 0x0, 0xc42062f5c0, 0x10aeabb)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:406 +0xf2
text/template.(*state).walkRange(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0xc4201704c0)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:315 +0xec
text/template.(*state).walk(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x1703fa0, 0xc4201704c0)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:243 +0x4f1
text/template.(*state).walk(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x1703e20, 0xc420125470)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:240 +0x139
text/template.(*Template).execute(0xc42023a1c0, 0x16fda20, 0xc42000e018, 0x14b2560, 0xc420a17800, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:195 +0x20a
text/template.(*Template).Execute(0xc42023a1c0, 0x16fda20, 0xc42000e018, 0x14b2560, 0xc420a17800, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:178 +0x53
github.com/segmentio/cwlogs/cmd.fetch(0x172d5c0, 0xc420016e80, 0x1, 0x4, 0x0, 0x0)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/cmd/fetch.go:125 +0x537
github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra.(*Command).execute(0x172d5c0, 0xc420016e40, 0x4, 0x4, 0x172d5c0, 0xc420016e40)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra/command.go:650 +0x457
github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x172da40, 0x14d2a5b, 0x13, 0xc4200f9f68)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra/command.go:729 +0x339
github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra.(*Command).Execute(0x172da40, 0xc420010118, 0x0)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra/command.go:688 +0x2b
github.com/segmentio/cwlogs/cmd.Execute()
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/cmd/root.go:45 +0x31
main.main()
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/main.go:6 +0x20
It looks like if log events are in JSON format, cwlogs expects the JSON to be structured in this format:
type Event struct {
Level Level `json:"level"`
Time time.Time `json:"time"`
Info EventInfo `json:"info"`
Data EventData `json:"data"`
Message string `json:"message"`
}
https://github.com/segmentio/ecs-logs-go/blob/46096d8f38d354840bbc51bf2d425416a907dd6b/event.go#L63
Our project outputs logs with a timeMillis
epoch value and it took a while to get to the bottom of why the time was displaying incorrectly.
Does documentation exist somewhere on this event format? Also, I think that the timestamp should be populated for the event based on the AWS event timestamp if the json timestamp isn't found.
I noticed that cwlogs orders/sorts log lines incorrectly:
[ e184f96e ] 02-23 18:52:03 INFO - beginning to drop messages
[ f0bddc5b ] 02-23 18:48:53 INFO - stats/datadog: sending metrics with a buffer of size 1024 B
Is it expected behaviour, or does the log lines from all streams should be sorted together?
First off, thank you for putting this project together. Its the most performant way I've found to examine CloudWatch logs.
I understand that CloudWatch does not provide a great native solution for filtering streams by name if you are not doing a prefix match. Therefore cwlogs has implemented its own in-process matching, but it caps the number of streams it searches for a particular query to 100.
This number is too low for my purposes and was wondering if this project would consider exposing that limit as a configurable option?
If the project is willing to do so, I'd be happy to help contribute, but I'm new to Go so would require some guidance.
Alternatively, please consider upping the limit to 500. Searching 500 streams in my tests still allow the query to return in under 2 seconds.
👋 just heads up, govendor
is deprecated. I am curious about the maintenance status of this project. Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.