Light

tkuchiki / alp Goto Github PK

View Code? Open in Web Editor NEW

660.0 660.0 34.0 635 KB

Access Log Profiler

License: MIT License

Go 99.30% Makefile 0.67% Dockerfile 0.03%

access-logs golang json ltsv profiler regexp

alp's Introduction

Hi there 👋

I'm interested in:

timezone
- https://github.com/tkuchiki/go-timezone
- https://github.com/tkuchiki/timezones
access log profiling
- https://github.com/tkuchiki/alp
- https://github.com/tkuchiki/wireray

I ❤️ Shell Script and Go.

Thank you for reading!

alp's People

Contributors

Stargazers

Watchers

alp's Issues

Fix https://github.com/tkuchiki/alp/pull/48#discussion_r676537188

add HTTP status code to output

diff command is broken

Hi @tkuchiki ! Thank you for the nice plugin.

Currently, diff command seems to be broken.

$ go run ./cmd/alp diff hoge.yaml fuga.yaml
2023/08/13 11:05:57 flag accessed but not defined: config
exit status 1

-c option causes panic when some key is omitted

Hello!
If the configuration file does not contain ltsv, json, regexp, or pcap, it will cause nil pointer dereference error

❯ ./cli/alp/alp json -c 'example/config.yml'
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x1006cd7d0]

goroutine 1 [running]:
github.com/tkuchiki/alp/options.LoadOptionsFromReader(0x10089af28, 0x14000010bd8, 0x0, 0x0, 0x14000010bd8)
	/Users/hosshii/Workspace/github.com/tkuchiki/alp/options/option.go:583 +0x600
github.com/tkuchiki/alp.(*Profiler).Run(0x14000139780, 0x14000020090, 0x3, 0x3, 0x0, 0x0)
	/Users/hosshii/Workspace/github.com/tkuchiki/alp/alp.go:138 +0x730
main.main()
	/Users/hosshii/Workspace/github.com/tkuchiki/alp/cli/alp/main.go:13 +0x9c

I think this is because this field is pointer type, and the field become nil if configuration is omitted.

`-c` option no longer working

Hi, I found an error when using -c option. --config works though.

$ alp -c alp.yaml ltsv --file /var/log/nginx/access.log
2023/11/09 09:58:41 unknown shorthand flag: 'c' in -c
$ alp --version
1.0.21

Maybe we should update README here?

Lines 541 to 543 in d91a23d

    
           - `-c, --config` 
        
               - The configuration file 
        
               - YAML

Can you also distribute ARM64 binaries?

Currently, you distribute AMD64 and MIPS binaries. I would like you to distribute ARM64 binaries.

Also, because it is difficult to handle zip files on Linux, I would like you to distribute them in gzip format if possible.

alp dies while parsing invalid URL escape

/tmp/bad.log

time:02/Jan/2016:07:11:42 +0000 host:114.172.184.92 forwardedfor:-  req:GET / HTTP/1.1  status:200  method:GET  uri:/   size:31506  referer:-   ua:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    reqtime:0.022   cache:- runtime:-   apptime:-   vhost:dqn.sakusakutto.jp
time:02/Jan/2016:07:11:42 +0000 host:114.172.184.92 forwardedfor:-  req:GET /css/main.css?1000 HTTP/1.1 status:304  method:GET  uri:/css/main.css?1000  size:0  referer:http://dqn.sakusakutto.jp/  ua:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    reqtime:0.000   cache:- runtime:-   apptime:-   vhost:dqn.sakusakutto.jp
time:02/Jan/2016:07:11:42 +0000 host:114.172.184.92 forwardedfor:-  req:GET /images/kali.jpg HTTP/1.1   status:304  method:GET  uri:/images/kali.jpg    size:0  referer:http://dqn.sakusakutto.jp/  ua:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    reqtime:0.000   cache:- runtime:-   apptime:-   vhost:dqn.sakusakutto.jp
time:02/Jan/2016:07:11:46 +0000 host:185.130.5.224  forwardedfor:-  req:GET /server-status?HTTP_POST=%\x22%6346#%#/&#736%\x22#423|;&HTTP_CGI_GET=GRESYYK\x22K&J\x22#L523D2G23H23 HTTP/1.0   status:404  method:GET  uri:/server-status?HTTP_POST=%\x22%6346#%#/&#736%\x22#423|;&HTTP_CGI_GET=GRESYYK\x22K&J\x22#L523D2G23H23    size:168    referer:-   ua:apache 0day by @hxmonsegur   reqtime:1.206   cache:- runtime:-   apptime:-   vhost:dqn.sakusakutto.jp
time:02/Jan/2016:07:12:07 +0000 host:119.241.121.77 forwardedfor:-  req:GET /2015/12/vagrantfiles_for_php7.html HTTP/1.1    status:200  method:GET  uri:/2015/12/vagrantfiles_for_php7.html size:3255   referer:http://dqn.sakusakutto.jp/2015/12/centos7_yum_install_php70.html    ua:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36    reqtime:0.000   cache:- runtime:-   apptime:-   vhost:dqn.sakusakutto.jp
time:02/Jan/2016:07:12:40 +0000 host:182.118.22.219 forwardedfor:-  req:GET /2013/04/c_kr_stack.html HTTP/1.1   status:200  method:GET  uri:/2013/04/c_kr_stack.html    size:3486   referer:http://dqn.sakusakutto.jp/2013/04/c_kr_stack.html   ua:Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)   reqtime:0.000   cache:- runtime:-   apptime:-   vhost:dqn.sakusakutto.jp

$ alp --version
0.2.1
$ alp --apptime-label reqtime -f  /tmp/bad.log
2016/01/02 07:20:28 parse /server-status?HTTP_POST=%\x22%6346#%#/&#736%\x22#423|;&HTTP_CGI_GET=GRESYYK\x22K&J\x22#L523D2G23H23: invalid URL escape "%#/"

IMHO, it would be more helpful if alp skipped invalid records , instead of aborting the whole process.

typo in README.md

I found a typo in README.md.

Download from https://github.com/tkuchiki/alp/releasse
-> Download from https://github.com/tkuchiki/alp/releases

Custom percentiles

Feature Suggestion: vhost filtering

現状 vhost によるフィルタリングを grep vhost:example.com foo.log | alp ltsv としているのですが、これがフィルタで行えたら便利だと思いました。

`alp json --dump` does not work 1.0.13 rel

👋 while upgrading alp to the latest release, 1.0.13, I ran into some regression test failure issue (--dump stops working for alp json), thanks!

error log as below:

==> /opt/homebrew/Cellar/alp/1.0.13/bin/alp json --file=/private/tmp/alp-test-20230722-91695-180mec3/json_access.log --dump=/private/tmp/alp-test-20230722-91695-180mec3/dump.yml
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD |        URI        |  MIN   |  MAX   |  SUM   |  AVG   |  P90   |  P95   |  P99   | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 1     | 0   | 1   | 0   | 0   | 0   | POST   | /hoge/piyo        | 0.234  | 0.234  | 0.234  | 0.234  | 0.234  | 0.234  | 0.234  | 0.000  | 34.000    | 34.000    | 34.000    | 34.000    |
| 1     | 0   | 1   | 0   | 0   | 0   | GET    | /diary/entry/1234 | 0.135  | 0.135  | 0.135  | 0.135  | 0.135  | 0.135  | 0.135  | 0.000  | 15.000    | 15.000    | 15.000    | 15.000    |
| 1     | 0   | 1   | 0   | 0   | 0   | GET    | /diary/entry/5678 | 0.432  | 0.432  | 0.432  | 0.432  | 0.432  | 0.432  | 0.432  | 0.000  | 30.000    | 30.000    | 30.000    | 30.000    |
| 1     | 0   | 0   | 0   | 0   | 1   | GET    | /foo/bar/5xx      | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 0.000  | 15.000    | 15.000    | 15.000    | 15.000    |
| 1     | 0   | 1   | 0   | 0   | 0   | GET    | /req              | 0.321  | 0.321  | 0.321  | 0.321  | 0.321  | 0.321  | 0.321  | 0.000  | 15.000    | 15.000    | 15.000    | 15.000    |
| 2     | 0   | 2   | 0   | 0   | 0   | GET    | /foo/bar          | 0.123  | 0.123  | 0.246  | 0.123  | 0.123  | 0.123  | 0.123  | 0.000  | 56.000    | 56.000    | 112.000   | 56.000    |
| 5     | 0   | 5   | 0   | 0   | 0   | POST   | /foo/bar          | 0.057  | 0.234  | 0.548  | 0.110  | 0.234  | 0.234  | 0.234  | 0.065  | 12.000    | 34.000    | 126.000   | 25.200    |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+

let me know if I need to provide more info.

relates to Homebrew/homebrew-core#137306

Migrate to cobra

add test

Failure on timezone.GetOffset while using go get

Hello,
I'm trying to compile alp from sources and a straight go get fails for me with the following output:

i7:~$ go get -u github.com/tkuchiki/alp
# github.com/tkuchiki/parsetime
.go/src/github.com/tkuchiki/parsetime/parsetime.go:63:20: undefined: timezone.GetOffset
.go/src/github.com/tkuchiki/parsetime/parsetime.go:117:17: undefined: timezone.GetOffset
i7:~$ go version
go version go1.15.9 linux/amd64
i7:~$

Am I doing sth wrong or this is a genuine error? Thank you !

alp 1.0.0 does not show the correct result for H20 log

Thank you for your great tool! 😄 I just encountered a small problem.

version

v1.0.0-rc1 or above

Problem

When I use alp v1.0.0 -rc1 for H2o ltsv log, it does not show the result like below.

[vagrant@localhost ~]$ sudo alp ltsv --file /var/log/h2o/access.log
+-------+-----+-----+-----+-----+-----+--------+-----+-----+-----+-----+-----+----+-----+-----+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-----+-----+-----+-----+-----+----+-----+-----+--------+-----------+-----------+-----------+-----------+
+-------+-----+-----+-----+-----+-----+--------+-----+-----+-----+-----+-----+----+-----+-----+--------+-----------+-----------+-----------+-----------+

When I use alp v0.4.0 to the same log file instead, it shows the correct result.

[vagrant@localhost ~]$ sudo alp --file=/var/log/h2o/access.log
+-------+--------+-----------------------------------------+-----+-----+-----+-----+-----+--------+--------+--------+--------+--------+--------+--------+--------+--------------+--------------+--------------+--------------+
| COUNT | METHOD |                   URI                   | 1XX | 2XX | 3XX | 4XX | 5XX |  MIN   |  MAX   |  SUM   |  AVG   |   P1   |  P50   |  P99   | STDDEV |  MIN(BODY)   |  MAX(BODY)   |  SUM(BODY)   |  AVG(BODY)   |
+-------+--------+-----------------------------------------+-----+-----+-----+-----+-----+--------+--------+--------+--------+--------+--------+--------+--------+--------------+--------------+--------------+--------------+
|     1 | DELETE | /api/events/11/sheets/C/60/reservation  |   0 |   0 |   0 |   1 |   0 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |       26.000 |       26.000 |       26.000 |       26.000 |
|    29 | GET    | /css/admin.css                          |   0 |  29 |   0 |   0 |   0 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |      684.000 |      684.000 |    19836.000 |      684.000 |
|     1 | DELETE | /api/events/11/sheets/C/115/reservation |   0 |   0 |   0 |   1 |   0 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |       26.000 |       26.000 |       26.000 |       26.000 |
|    29 | GET    | /css/layout.css                         |   0 |  29 |   0 |   0 |   0 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |  0.000 |      707.000 |      707.000 |    20503.000 |      707.

/etc/h2o/h2o.conf

[vagrant@localhost ~]$ cat /etc/h2o/h2o.conf
user: isucon

access-log:
  path: /var/log/h2o/access.log
  format: "time:%t\thost:%h\tua:\"%{User-agent}i\"\tstatus:%s\treq:%r\turi:%U\treqtime:%{duration}x\tsize:%b\tmethod:%m\t"
error-log: /var/log/h2o/error.log
pid-file: /var/run/h2o/h2o.pid

hosts:
  "localhost:80":
    listen:
      port: 80
      host: 0.0.0.0
    paths:
      "/favicon.ico":
        file.file: /home/isucon/torb/webapp/static/favicon.ico
      "/css":
        file.dir: /home/isucon/torb/webapp/static/css
      "/img":
        file.dir: /home/isucon/torb/webapp/static/img
      "/js":
        file.dir: /home/isucon/torb/webapp/static/js
      "/":
        proxy.reverse.url: http://127.0.0.1:8080/
        proxy.preserve-host: ON

/var/log/h2o/access.log

[vagrant@localhost ~]$ sudo head /var/log/h2o/access.log

time:[24/Aug/2019:07:13:12 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:204	req:GET /initialize HTTP/1.1	uri:/initialize	reqtime:1.606702	size:0	method:GET
time:[24/Aug/2019:07:13:13 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:200	req:GET /css/admin.css HTTP/1.1	uri:/css/admin.css	reqtime:0	size:684	method:GET
time:[24/Aug/2019:07:13:13 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:200	req:GET /css/bootstrap.min.css HTTP/1.1	uri:/css/bootstrap.min.css	reqtime:0	size:140930	method:GET
time:[24/Aug/2019:07:13:13 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:200	req:GET /css/layout.css HTTP/1.1	uri:/css/layout.css	reqtime:0	size:707	method:GET
time:[24/Aug/2019:07:13:13 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:200	req:GET /favicon.ico HTTP/1.1	uri:/favicon.ico	reqtime:0	size:1092	method:GET
time:[24/Aug/2019:07:13:13 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:200	req:GET /js/admin.js HTTP/1.1	uri:/js/admin.js	reqtime:0	size:8454	method:GET
time:[24/Aug/2019:07:13:13 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:200	req:GET /js/app.js HTTP/1.1	uri:/js/app.js	reqtime:0	size:10204	method:GET
time:[24/Aug/2019:07:13:13 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:200	req:GET /js/bootstrap-waitingfor.min.js HTTP/1.1	uri:/js/bootstrap-waitingfor.min.js	reqtime:0size:2074	method:GET
time:[24/Aug/2019:07:13:13 +0000]	host:172.28.128.9	ua:"isucon8q-benchmarker"	status:200	req:GET /js/bootstrap.bundle.min.js HTTP/1.1	uri:/js/bootstrap.bundle.min.js	reqtime:0	size:70682	method:GET

refactor

ref: #5

Setup Github Actions

Create Pull Request to Homebrew
Release binary

Suggestion to clarify dependencies

Situation

I tried to build alp but got following error:

.go/src/github.com/tkuchiki/alp/main.go:572: undefined: parsetime.Parse
.go/src/github.com/tkuchiki/alp/main.go:589: undefined: parsetime.Parse
.go/src/github.com/tkuchiki/alp/main.go:625: undefined: parsetime.Log

I think the reason is that alp is not supporting latest github.com/tkuchiki/parsetime package.

Problem

There is no information about supported version of dependency packages.

Solution(Suggestion)

If you feel negative to use some dependency manager like Godep to make it easier to reproduce build dependencies, at least could you consider to clarify dependency information which includes supported version (or revision)?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.