segmentio / cwlogs Goto Github PK
View Code? Open in Web Editor NEWCLI tool for reading logs from Cloudwatch Logs
License: MIT License
cwlogs's Introduction
cwlogs's People
Contributors
Stargazers
Watchers
cwlogs's Issues
--since default value is unintuitive
It's taking several minutes to get any logs back. This makes the tool pretty unusable, as I can stumble around the CloudWatch's clunky UI more quickly than that.
∴ time aws-vault exec prod -- cwlogs fetch integration-drip-v1
^C
real 2m24.997s
user 0m8.905s
sys 0m0.771s
Is there anything we can do to improve performance here?
--until flag doesn't work
The --until flag doesn't seem to be working. I've tried both --until 20m and --until 2017-07-05T09:00:00 but it always fetches until now.
Use AWS region from configured profile or default
To run cwlogs I have to export the AWS_REGION variable. It should use what's configured in the AWS profile I'm using, or default to us-east-1.
panic: reflect: call of reflect.Value.Type on zero Value
I was grabbing logs from a service and hit this:
panic: reflect: call of reflect.Value.Type on zero Value [recovered]
panic: reflect: call of reflect.Value.Type on zero Value
goroutine 1 [running]:
text/template.errRecover(0xc42062f858)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:146 +0x62
panic(0x144e640, 0xc42032bfe0)
/usr/local/Cellar/go/1.8.3/libexec/src/runtime/panic.go:489 +0x2cf
reflect.Value.Type(0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/reflect/value.go:1688 +0x21f
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.FlattenPrefixedToResult(0x0, 0x0, 0xc42032bf60, 0x17, 0xc4206e8450)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:72 +0xee
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.FlattenPrefixedToResult(0x14545c0, 0xc42067cf30, 0xc4211b9630, 0xc, 0xc4206e8450)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:81 +0x588
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.FlattenPrefixedToResult(0x1471960, 0xc42067ced0, 0x0, 0x0, 0xc4206e8450)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:81 +0x588
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.FlattenPrefixed(0x1471960, 0xc42067ced0, 0x0, 0x0, 0x13e5d40)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:56 +0x87
github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows.Flatten(0x1471960, 0xc42067ced0, 0x104d693)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/doublerebel/bellows/main.go:51 +0x47
github.com/segmentio/cwlogs/lib.Event.DataFlat(0x7, 0xed19e9e47, 0xf42400, 0x0, 0xc4211a8b20, 0xc, 0x0, 0x0, 0x0, 0x0, ...)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/lib/event.go:76 +0x3a
github.com/segmentio/cwlogs/lib.(*Event).DataFlat(0xc420a17800, 0x0)
<autogenerated>:5 +0x80
reflect.Value.call(0x14b2560, 0xc420a17800, 0x693, 0x14cc766, 0x4, 0x174d078, 0x0, 0x0, 0x5522, 0x13eb262, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0x14b2560, 0xc420a17800, 0x693, 0x174d078, 0x0, 0x0, 0xc4203da840, 0xc4203da840, 0xc420976260)
/usr/local/Cellar/go/1.8.3/libexec/src/reflect/value.go:302 +0xa4
text/template.(*state).evalCall(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x14b2560, 0xc420a17800, 0x693, 0x1703d00, 0xc42016c660, 0x14e38dd, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:668 +0x51f
text/template.(*state).evalField(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x14e38dd, 0x8, 0x1703d00, 0xc42016c660, 0xc420014980, 0x1, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:558 +0xe18
text/template.(*state).evalFieldChain(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x14b2560, 0xc420a17800, 0x99, 0x1703d00, 0xc42016c660, 0xc420014850, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:526 +0x22b
text/template.(*state).evalFieldNode(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0xc42016c660, 0xc420014980, 0x1, 0x1, 0x0, 0x0, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:490 +0x118
text/template.(*state).evalCommand(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0xc42016c630, 0x0, 0x0, 0x0, 0x16, 0x3, ...)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:428 +0x837
text/template.(*state).evalPipeline(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0xc420168230, 0x0, 0xc42062f5c0, 0x10aeabb)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:406 +0xf2
text/template.(*state).walkRange(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0xc4201704c0)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:315 +0xec
text/template.(*state).walk(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x1703fa0, 0xc4201704c0)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:243 +0x4f1
text/template.(*state).walk(0xc42062f7d8, 0x14b2560, 0xc420a17800, 0x99, 0x1703e20, 0xc420125470)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:240 +0x139
text/template.(*Template).execute(0xc42023a1c0, 0x16fda20, 0xc42000e018, 0x14b2560, 0xc420a17800, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:195 +0x20a
text/template.(*Template).Execute(0xc42023a1c0, 0x16fda20, 0xc42000e018, 0x14b2560, 0xc420a17800, 0x0, 0x0)
/usr/local/Cellar/go/1.8.3/libexec/src/text/template/exec.go:178 +0x53
github.com/segmentio/cwlogs/cmd.fetch(0x172d5c0, 0xc420016e80, 0x1, 0x4, 0x0, 0x0)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/cmd/fetch.go:125 +0x537
github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra.(*Command).execute(0x172d5c0, 0xc420016e40, 0x4, 0x4, 0x172d5c0, 0xc420016e40)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra/command.go:650 +0x457
github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x172da40, 0x14d2a5b, 0x13, 0xc4200f9f68)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra/command.go:729 +0x339
github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra.(*Command).Execute(0x172da40, 0xc420010118, 0x0)
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/vendor/github.com/spf13/cobra/command.go:688 +0x2b
github.com/segmentio/cwlogs/cmd.Execute()
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/cmd/root.go:45 +0x31
main.main()
/Users/stephenmathieson/dev/src/github.com/segmentio/cwlogs/main.go:6 +0x20
Expected JSON format - timestamp
It looks like if log events are in JSON format, cwlogs expects the JSON to be structured in this format:
type Event struct {
Level Level `json:"level"`
Time time.Time `json:"time"`
Info EventInfo `json:"info"`
Data EventData `json:"data"`
Message string `json:"message"`
}
https://github.com/segmentio/ecs-logs-go/blob/46096d8f38d354840bbc51bf2d425416a907dd6b/event.go#L63
Our project outputs logs with a timeMillis epoch value and it took a while to get to the bottom of why the time was displaying incorrectly.
Does documentation exist somewhere on this event format? Also, I think that the timestamp should be populated for the event based on the AWS event timestamp if the json timestamp isn't found.
ordering of log lines
I noticed that cwlogs orders/sorts log lines incorrectly:
[ e184f96e ] 02-23 18:52:03 INFO - beginning to drop messages
[ f0bddc5b ] 02-23 18:48:53 INFO - stats/datadog: sending metrics with a buffer of size 1024 B
Is it expected behaviour, or does the log lines from all streams should be sorted together?
Make MaxStreams configurable
First off, thank you for putting this project together. Its the most performant way I've found to examine CloudWatch logs.
I understand that CloudWatch does not provide a great native solution for filtering streams by name if you are not doing a prefix match. Therefore cwlogs has implemented its own in-process matching, but it caps the number of streams it searches for a particular query to 100.
This number is too low for my purposes and was wondering if this project would consider exposing that limit as a configurable option?
If the project is willing to do so, I'd be happy to help contribute, but I'm new to Go so would require some guidance.
Alternatively, please consider upping the limit to 500. Searching 500 streams in my tests still allow the query to return in under 2 seconds.
status of the project?
👋 just heads up, govendor is deprecated. I am curious about the maintenance status of this project. Thanks!
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.