Code Monkey home page Code Monkey logo

ngxtop's Introduction

ngxtop - real-time metrics for nginx server (and others)

ngxtop parses your nginx access log and outputs useful, top-like, metrics of your nginx server. So you can tell what is happening with your server in real-time.

ngxtop is designed to run in a short-period time just like the top command for troubleshooting and monitoring your Nginx server at the moment. If you need a long running monitoring process or storing your webserver stats in external monitoring / graphing system, you can try Luameter.

ngxtop tries to determine the correct location and format of nginx access log file by default, so you can just run ngxtop and having a close look at all requests coming to your nginx server. But it does not limit you to nginx and the default top view. ngxtop is flexible enough for you to configure and change most of its behaviours. You can query for different things, specify your log and format, even parse remote Apache common access log with ease. See sample usages below for some ideas about what you can do with it.

Installation

pip install ngxtop

Note: ngxtop is primarily developed and tested with python2 but also supports python3.

Usage

Usage:
    ngxtop [options]
    ngxtop [options] (print|top|avg|sum) <var>
    ngxtop info

Options:
    -l <file>, --access-log <file>  access log file to parse.
    -f <format>, --log-format <format>  log format as specify in log_format directive.
    --no-follow  ngxtop default behavior is to ignore current lines in log
                     and only watch for new lines as they are written to the access log.
                     Use this flag to tell ngxtop to process the current content of the access log instead.
    -t <seconds>, --interval <seconds>  report interval when running in follow mode [default: 2.0]

    -g <var>, --group-by <var>  group by variable [default: request_path]
    -w <var>, --having <expr>  having clause [default: 1]
    -o <var>, --order-by <var>  order of output for default query [default: count]
    -n <number>, --limit <number>  limit the number of records included in report for top command [default: 10]
    -a <exp> ..., --a <exp> ...  add exp (must be aggregation exp: sum, avg, min, max, etc.) into output

    -v, --verbose  more verbose output
    -d, --debug  print every line and parsed record
    -h, --help  print this help message.
    --version  print version information.

    Advanced / experimental options:
    -c <file>, --config <file>  allow ngxtop to parse nginx config file for log format and location.
    -i <filter-expression>, --filter <filter-expression>  filter in, records satisfied given expression are processed.
    -p <filter-expression>, --pre-filter <filter-expression> in-filter expression to check in pre-parsing phase.

Samples

Default output

$ ngxtop
running for 411 seconds, 64332 records processed: 156.60 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|   64332 |         2775.251 | 61262 |  2994 |    71 |     5 |

Detailed:
| request_path                             |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /abc/xyz/xxxx                            |   20946 |          434.693 | 20935 |     0 |    11 |     0 |
| /xxxxx.json                              |    5633 |         1483.723 |  5633 |     0 |     0 |     0 |
| /xxxxx/xxx/xxxxxxxxxxxxx                 |    3629 |         6835.499 |  3626 |     0 |     3 |     0 |
| /xxxxx/xxx/xxxxxxxx                      |    3627 |        15971.885 |  3623 |     0 |     4 |     0 |
| /xxxxx/xxx/xxxxxxx                       |    3624 |         7830.236 |  3621 |     0 |     3 |     0 |
| /static/js/minified/utils.min.js         |    3031 |         1781.155 |  2104 |   927 |     0 |     0 |
| /static/js/minified/xxxxxxx.min.v1.js    |    2889 |         2210.235 |  2068 |   821 |     0 |     0 |
| /static/tracking/js/xxxxxxxx.js          |    2594 |         1325.681 |  1927 |   667 |     0 |     0 |
| /xxxxx/xxx.html                          |    2521 |          573.597 |  2520 |     0 |     1 |     0 |
| /xxxxx/xxxx.json                         |    1840 |          800.542 |  1839 |     0 |     1 |     0 |

View top source IPs of clients

$ ngxtop top remote_addr
running for 20 seconds, 3215 records processed: 159.62 req/sec

top remote_addr
| remote_addr     |   count |
|-----------------+---------|
| 118.173.177.161 |      20 |
| 110.78.145.3    |      16 |
| 171.7.153.7     |      16 |
| 180.183.67.155  |      16 |
| 183.89.65.9     |      16 |
| 202.28.182.5    |      16 |
| 1.47.170.12     |      15 |
| 119.46.184.2    |      15 |
| 125.26.135.219  |      15 |
| 125.26.213.203  |      15 |

List 4xx or 5xx responses together with HTTP referer

$ ngxtop -i 'status >= 400' print request status http_referer
running for 2 seconds, 28 records processed: 13.95 req/sec

request, status, http_referer:
| request   |   status | http_referer   |
|-----------+----------+----------------|
| -         |      400 | -              |

Parse apache log from remote server with common format

$ ssh user@remote_server tail -f /var/log/apache2/access.log | ngxtop -f common
running for 20 seconds, 1068 records processed: 53.01 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    1068 |        28026.763 |  1029 |    20 |    19 |     0 |

Detailed:
| request_path                             |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /xxxxxxxxxx                              |     199 |        55150.402 |   199 |     0 |     0 |     0 |
| /xxxxxxxx/xxxxx                          |     167 |        47591.826 |   167 |     0 |     0 |     0 |
| /xxxxxxxxxxxxx/xxxxxx                    |      25 |         7432.200 |    25 |     0 |     0 |     0 |
| /xxxx/xxxxx/x/xxxxxxxxxxxxx/xxxxxxx      |      22 |          698.727 |    22 |     0 |     0 |     0 |
| /xxxx/xxxxx/x/xxxxxxxxxxxxx/xxxxxx       |      19 |         7431.632 |    19 |     0 |     0 |     0 |
| /xxxxx/xxxxx/                            |      18 |         7840.889 |    18 |     0 |     0 |     0 |
| /xxxxxxxx/xxxxxxxxxxxxxxxxx              |      15 |         7356.000 |    15 |     0 |     0 |     0 |
| /xxxxxxxxxxx/xxxxxxxx                    |      15 |         9978.800 |    15 |     0 |     0 |     0 |
| /xxxxx/                                  |      14 |            0.000 |     0 |    14 |     0 |     0 |
| /xxxxxxxxxx/xxxxxxxx/xxxxx               |      13 |        20530.154 |    13 |     0 |     0 |     0 |

ngxtop's People

Contributors

amuraru avatar amyreese avatar bndr avatar haguenau avatar jlec avatar jleclanche avatar lebinh avatar mcortizo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ngxtop's Issues

log_format of apache access logs

Hi!

I have log like this:

192.168.7.40 - - [31/Mar/2014:04:02:02 +0400] "POST /my/url HTTP/1.0" 200 16 "-" "Zend_Http_Client"

It successfully parsed by ngxtop, but if I have this format:

192.168.7.40 192.168.7.41 - [31/Mar/2014:04:02:02 +0400] "POST /my/url HTTP/1.0" 200 16 "-" "Zend_Http_Client"

It does not parse.

How can I parsed this log? I tried to use --log-format but it did not help me.

Log Records

Does ngxtop save records for its own use , somewhere?

I mean how can it take out the top 20 requests ?
by
ngxtop -n 20

ngxtop reports no ips or data from nginx log ?

ngxtop latest version installed via pip on CentOS 6.6 64bit

ngxtop --version
xstat 0.1

command used

ngxtop -l /home/nginx/domains/domain.com/log/access.log.norotate --group-by remote_addr --no-follow

results in output of

running for 0 seconds, 0 records processed: 0.00 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       0 |                  |     0 |     0 |     0 |     0 |

Detailed:
| remote_addr   | count   | avg_bytes_sent   | 2xx   | 3xx   | 4xx   | 5xx   |
|---------------+---------+------------------+-------+-------+-------+-------|

log file contents only 5 lines of

 IPADDRESS - - [14/Jun/2015:00:52:57 +0000] GET /url HTTP/1.1 "200" 31 "-" "curl/7.29.0" "-" "-" "8" "1" "0.000"

sqlite3.ProgrammingError: library routine called out of sequence

I'm using python 2.7.6
ngxtop --config /usr/local/nginx/conf/nginx.conf
It works for 933 seconds, then I got this

Traceback (most recent call last):
  File "/opt/python2.7.6/bin/ngxtop", line 11, in <module>
    sys.exit(main())
  File "/opt/python2.7.6/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 446, in main
    process(args)
  File "/opt/python2.7.6/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 429, in process
    total = process_log(source, pattern, processor, arguments)
  File "/opt/python2.7.6/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 334, in process_log
    total = processor.process(records)
  File "/opt/python2.7.6/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 282, in process
    cursor.execute(insert, r)
sqlite3.ProgrammingError: library routine called out of sequence

Multiple group by columns

I'd like to show requests by hostname and request path, so I tried:

ngxtop -g http_host,request_path

which gives me

running for 8 seconds, 0 records processed: 0.00 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       0 |                  |     0 |     0 |     0 |     0 |

Detailed:

Traceback (most recent call last):
  File "/usr/bin/ngxtop", line 9, in <module>
    load_entry_point('ngxtop==0.0.1', 'console_scripts', 'ngxtop')()
  File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 446, in main
    process(args)
  File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 429, in process
    total = process_log(source, pattern, processor, arguments)
  File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 334, in process_log
    total = processor.process(records)
  File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 282, in process
    cursor.execute(insert, r)
sqlite3.OperationalError: no such column: request_path

So then I tried nginxtop -g http_host -g request_path, which just gives me back the usage information. Grouping by either http_host or request_path works fine, it's just the combination together that I can't seem to get working.

Got error messages in centos 6.5

Centos 6.5

Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)

./ngxtop.py
Traceback (most recent call last):
  File "./ngxtop.py", line 440, in <module>
    main(args)
  File "./ngxtop.py", line 402, in main
    config = get_nginx_conf_path()
  File "./ngxtop.py", line 117, in get_nginx_conf_path
    proc = subprocess.Popen(['nginx', '-V'], stderr=subprocess.PIPE)
  File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Wrong if syntax

Hello!

[michael_xiii@new-host ~]$ sudo /usr/bin/pip-python install ngxtop
Downloading/unpacking ngxtop
  Downloading ngxtop-0.0.2.tar.gz
  Running setup.py egg_info for package ngxtop
Downloading/unpacking docopt (from ngxtop)
  Downloading docopt-0.6.2.tar.gz
  Running setup.py egg_info for package docopt
    Traceback (most recent call last):
      File "<string>", line 14, in ?
      File "/home/michael_xiii/build/docopt/setup.py", line 3, in ?
        from docopt import __version__
      File "docopt.py", line 49
        uniq = list(set(self.flat())) if uniq is None else uniq
                                                ^
    SyntaxError: invalid syntax
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in ?

  File "/home/michael_xiii/build/docopt/setup.py", line 3, in ?

    from docopt import __version__

  File "docopt.py", line 49

    uniq = list(set(self.flat())) if uniq is None else uniq

                                            ^

SyntaxError: invalid syntax

----------------------------------------
Command python setup.py egg_info failed with error code 1

Python 2.4.3 (#1, Jan 9 2013, 06:47:03)

uname -a

Linux XXX 2.6.18-238.19.1.el5.028stab092.2 #1 SMP Thu Jul 21 19:23:22 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux

Error while installing

Not able to install. appreciate help.
pip.log.zip

Downloading/unpacking ngxtop
  Downloading ngxtop-0.0.2-py2.py3-none-any.whl
Downloading/unpacking pyparsing (from ngxtop)
  Downloading pyparsing-2.1.1-py2.py3-none-any.whl
Downloading/unpacking docopt (from ngxtop)
  Downloading docopt-0.6.2.tar.gz
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1198, in prepare_files
    do_download,
  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1376, in unpack_url
    self.session,
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 582, in unpack_http_url
    unpack_file(temp_location, location, content_type, link)
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 643, in unpack_file
    untar_file(filename, location)
  File "/usr/lib/python2.7/dist-packages/pip/util.py", line 551, in untar_file
    os.makedirs(location)
  File "/usr/lib/python2.7/os.py", line 157, in makers
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/tmp/pip_build_root/docopt'

Storing debug log for failure in /root/.pip/pip.log

Improper field splitting for a record

Invocation like:
$ ngnxtop -l path_to/my_access.log

The acces.log has a normal format, like in nginx.conf: "access_log path_to/my_access.log;"

running for 10 seconds, 0 records processed: 0.00 req/sec

Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 0 | | 0 | 0 | 0 | 0 |

Detailed:

Traceback (most recent call last):
File "/usr/local/bin/ngxtop", line 9, in
load_entry_point('ngxtop==0.0.1', 'console_scripts', 'ngxtop')()
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 446, in main
process(args)
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 429, in process
total = process_log(source, pattern, processor, arguments)
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 334, in process_log
total = processor.process(records)
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 281, in process
for r in records:
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 215, in add_field
for item in dict_sequence:
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 205, in map_field
for item in dict_sequence:
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 205, in map_field
for item in dict_sequence:
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 215, in add_field
for item in dict_sequence:
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 215, in add_field
for item in dict_sequence:
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 206, in map_field
item[field] = func(item.get(field, None))
File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 245, in to_int
return int(value) if value and value != '-' else 0
ValueError: invalid literal for int() with base 10: '"Mozilla/5.0 (X11; Ubuntu;'

access.log detection / parsing

Hello,

After
$ pip install docopt tabulate

I have:

$ python -V
Python 2.7.4

$ grep access_log /etc/nginx/nginx.conf
access_log /var/log/nginx/access.log;

but log file is not detected:

$ python ngxtop.py info
configuration file:
/etc/nginx/nginx.conf
access log file:
logs/access.log
access log format:
combined
available variables:

What's more, explicitly setting the log file breaks:

$ python ngxtop.py --access-log /var/log/nginx/access.log
Traceback (most recent call last):
File "ngxtop.py", line 437, in
main(args)
File "ngxtop.py", line 420, in main
total = process_log(source, pattern, processor, arguments)
File "ngxtop.py", line 325, in process_log
total = processor.process(records)
File "ngxtop.py", line 272, in process
for r in records:
File "ngxtop.py", line 206, in add_field
for item in dict_sequence:
File "ngxtop.py", line 196, in map_field
for item in dict_sequence:
File "ngxtop.py", line 196, in map_field
for item in dict_sequence:
File "ngxtop.py", line 206, in add_field
for item in dict_sequence:
File "ngxtop.py", line 206, in add_field
for item in dict_sequence:
File "ngxtop.py", line 196, in map_field
for item in dict_sequence:
File "ngxtop.py", line 245, in
records = (m.groupdict() for m in matches if m is not None)
File "ngxtop.py", line 244, in
matches = (pattern.match(l) for l in lines)
File "ngxtop.py", line 181, in follow
with open(the_file) as f:
IOError: [Errno 2] No such file or directory: 'logs/access.log'

thanks!

How to custom my nginx log format?

Hello,
My nginx log is defined, like :

log_format    main   '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for" $request_length $request_time $bytes_sent ' 
                     '"$upstream_cache_status"';

What should I use ngxtop with this log_format, I don't discover an option for this situation.

Thanks & Regards

can not catch nginx-wallarm

nginx-wallarm -v
nginx version: nginx/1.8.1
nginx
command not found
ngxtop
Error: Access log file or format was not set and nginx config file cannot be detected. Perhaps nginx is not in your PATH?

ngxtop can't parse old logs (output is messed)

ngxtop is a nice tool, but I don't see a way to use it for old logs - while it does show the output nicely during log parsing, the output is garbled when log parsing finishes - see below for an example.

It would be nice if ngxtop had some kind of CSV/table/text output for such cases.

cat some-old-log | ngxtop

running for 0 seconds, 1000 records processed: 20227.36 req/sec

                                                           Summary:
                                                                   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
                                                                                                                                 |---------+------------------+-------+-------+-------+-------|
   |    1000 |          223.154 |   999 |     0 |     1 |     0 |

                                                                 Detailed:
                                                                          | request_path            |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
                                                                                                                                                                  |-------------------------+---------+------------------+-------+-------+-------+-------|
                                                              | /v2/api/execute.php     |     985 |          223.951 |   985 |     0 |     0 |     0 |
                                                                                                                                                      | /v2/api/healthcheck.php |      15 |          170.800 |    14 |     0 |     1 |     0 |

feature suggestion: specify an ID format to obtain metrics per "request type"

Use case

a REST api has requests like :

/x/y/2da625ba-1199-4e9f-96d9-c80394981952
/x/y/5d46145c-4399-3f4a-fdaa-a40f94934534

ngxtop will consider them as two distinct request.

If I could invoke ngxtop wit a regex that "detects" IDs in the REST paths,
ex: idformat=........-....-................

then the above two calls would become the same, /x/y/xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx
then metrics could be produced per call "type".

ngxtop no data gathering from access log

Can you help me what problem is?
no data summary from access log

ngxtop

running for 42 seconds, 0 records processed: 0.00 req/sec

Summary:
| count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|---------+------------------+-------+-------+-------+-------|
| 0 || 0 | 0 | 0 | 0 |

Detailed:
| request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |
|----------------+---------+------------------+-------+-------+-------+-------|246

ngxtop fail to work with comment out log_format directive.

use ngxtop -c conf/nginx.conf -l logs/access.log to specify the log_format. When nginx.conf with commented out log_format as following, I cann't not get the expected result.
# log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
log_format main '$remote_addr - $remote_user [$time_local] '
'"$request" $status $request_length $request_time $body_bytes_sent '
'"$http_referer" "$http_user_agent" $server_addr $upstream_addr $host $upstream_cache_status';

After I delete the commented lines, It works as expected.

sre_constants.error: redefinition of group name 'remote_addr' as group 17; was group 3

Traceback (most recent call last):
  File "/usr/local/bin/ngxtop", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 385, in main
    process(args)
  File "/usr/local/lib/python2.7/dist-packages/ngxtop/ngxtop.py", line 367, in process
    pattern = build_pattern(log_format)
  File "/usr/local/lib/python2.7/dist-packages/ngxtop/config_parser.py", line 137, in build_pattern
    return re.compile(pattern)
  File "/usr/lib/python2.7/re.py", line 190, in compile
    return _compile(pattern, flags)
  File "/usr/lib/python2.7/re.py", line 242, in _compile
    raise error, v # invalid expression
sre_constants.error: redefinition of group name 'remote_addr' as group 17; was group 3

--access-log doesn't seem to be working

ngxtop -l /var/log/nginx/access.log
Traceback (most recent call last):
  File "/home/adys/bin/ngxtop", line 437, in <module>
    main(args)
  File "/home/adys/bin/ngxtop", line 420, in main
    total = process_log(source, pattern, processor, arguments)
  File "/home/adys/bin/ngxtop", line 325, in process_log
    total = processor.process(records)
  File "/home/adys/bin/ngxtop", line 272, in process
    for r in records:
  File "/home/adys/bin/ngxtop", line 206, in add_field
    for item in dict_sequence:
  File "/home/adys/bin/ngxtop", line 196, in map_field
    for item in dict_sequence:
  File "/home/adys/bin/ngxtop", line 196, in map_field
    for item in dict_sequence:
  File "/home/adys/bin/ngxtop", line 206, in add_field
    for item in dict_sequence:
  File "/home/adys/bin/ngxtop", line 206, in add_field
    for item in dict_sequence:
  File "/home/adys/bin/ngxtop", line 196, in map_field
    for item in dict_sequence:
  File "/home/adys/bin/ngxtop", line 245, in <genexpr>
    records = (m.groupdict() for m in matches if m is not None)
  File "/home/adys/bin/ngxtop", line 244, in <genexpr>
    matches = (pattern.match(l) for l in lines)
  File "/home/adys/bin/ngxtop", line 181, in follow
    with open(the_file) as f:
IOError: [Errno 2] No such file or directory: 'logs/access.log'

Catchall

Hi !
I'm discovering ngxtop and i feel great about it !
My vhost error log files are split but if i want the overall stats, the catchall caracter doesn't work :
ngxtop -l /var/log/nginx/*/access_log

  • stands for my multiple vhosts.

thx a lot !

No matching distribution found for sqlite-devel

I run the that command via ssh
yum -y install python-pip; pip install ngxtop sqlite-devel

And it says:

Installed:
python-pip.noarch 0:7.1.0-1.el7
Complete!
/usr/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
You are using pip version 7.1.0, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting ngxtop
/usr/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading ngxtop-0.0.2-py2.py3-none-any.whl
Collecting sqlite-devel
Could not find a version that satisfies the requirement sqlite-devel (from versions: )
No matching distribution found for sqlite-devel

I think, a sqlite-devel problem.

P.S: Operation system is centos 7 64bit

Any way to filter the request_path ?

Hello

Is possible that any way to filter the request_path? For example I want to see analyze the just ".css" files (hit count /4xx 3xx 2xx errors)

Are there any parameter ?

why not data?

this is my command line: /usr/local/bin/ngxtop -l /opt/logs/access.log -f '[$local]' top remote_addr -t 1

but,there is not any data as follow. why? thanks.

running for 136 seconds, 0 records processed: 0.00 req/sec

top remote_addr
| remote_addr | count |
|---------------+---------|

Broken on python 2.7.6 (works on 2.7.5)

I checked (with pyenv):
works perfectly with python 2.7.5
when changing to python 2.7.6 broke with this error:

Traceback (most recent call last):
File "/home/alon/.pyenv/versions/2.7.6/bin/ngxtop", line 9, in
load_entry_point('ngxtop==0.0.2', 'console_scripts', 'ngxtop')()
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 348, in load_entry_point
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2311, in load_entry_point
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 2025, in load
File "/home/alon/.pyenv/versions/2.7.6/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 61, in
import curses
File "/home/alon/.pyenv/versions/2.7.6/lib/python2.7/curses/init.py", line 15, in
from _curses import *
ImportError: No module named _curses

BESIDES: great tool! love to use it

ImportError: No module named 'config_parser' under python 3.4

I run ngxtop in virtualenv: can not find config_parser

(py3virtualenv)[root@xxxx ~]# ngxtop
Traceback (most recent call last):
File "/root/py3virtualenv/bin/ngxtop", line 7, in
from ngxtop.ngxtop import main
File "/root/py3virtualenv/lib/python3.4/site-packages/ngxtop/ngxtop.py", line 77, in
from config_parser import detect_log_config, detect_config_path, extract_variables, build_pattern
ImportError: No module named 'config_parser'

ngxtop with nginx-boilerplate logs

I'm trying to use ngxtop on logs from https://github.com/nginx-boilerplate/nginx-boilerplate

But with no luck. No errors. Just 0 records processed: 0.00 req/sec

I used default boilerplate configs, tried to move log format into my sites config file, used every option from ngxtop...

Tried several times, on site access logs, on default access logs... Nothing.

Looks like something wrong with me... Or ngxtop doesnt work with that nginx boilerplate...

Can someone test it ?

tail log via ssh not working

ssh remote tail -f /var/log/apache2/access.log | ngxtop -f common
Traceback (most recent call last):
File "/usr/bin/ngxtop", line 9, in
load_entry_point('ngxtop==0.0.2', 'console_scripts', 'ngxtop')()
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 385, in main
process(args)
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 370, in process
process_log(source, pattern, processor, arguments)
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 267, in process_log
processor.process(records)
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 215, in process
for r in records:
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 148, in add_field
for item in dict_sequence:
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 135, in map_field
for item in dict_sequence:
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 135, in map_field
for item in dict_sequence:
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 148, in add_field
for item in dict_sequence:
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 148, in add_field
for item in dict_sequence:
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 135, in map_field
for item in dict_sequence:
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 187, in
records = (m.groupdict() for m in matches if m is not None)
File "/usr/lib/python2.6/site-packages/ngxtop-0.0.2-py2.6.egg/ngxtop/ngxtop.py", line 186, in
matches = (pattern.match(l) for l in lines)
IOError: [Errno 4] Interrupted system call

Run as an unpriviledged user on Debian

Their were two quirks I noticed, when trying this on Debian Wheezy as a non-root user with read access to the system logfiles.

nginx is located in /usr/sbin/, which usually is not in $PATH, so it won't find the binary.

The log location could somehow not be determined, I haven't dug through the code much. Had it fall back to '/var/log/nginx/access.log' instead of 'log/access.log'. It detected the nginx.conf, however it wasn't able to extract the access_log path.

% grep "access" /etc/nginx/nginx.conf
access_log /var/log/nginx/access.log;

RuntimeError: maximum recursion depth exceeded while calling a Python object

After running for a while, it raised RuntimeError: maximum recursion depth exceeded while calling a Python object.

File "/usr/local/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 233, in report
result = tabulate.tabulate(cursor.fetchall(), headers=columns, tablefmt='orgtbl', floatfmt='.3f')
File "/usr/local/lib/python2.7/site-packages/tabulate.py", line 753, in tabulate
coltypes = list(map(_column_type, cols)) File "/usr/local/lib/python2.7/site-packages/tabulate.py", line 426, in _column_type
types = [_type(s, has_invisible) for s in strings ]
File "/usr/local/lib/python2.7/site-packages/tabulate.py", line 260, in _type (isinstance(string, _text_type) or isinstance(string, _binary_type)):
RuntimeError: maximum recursion depth exceeded while calling a Python object

Can't specify config

I run a CentOS 6.3 box with nginx installed into /opt/nginx and there doesn't seem to be a way to specify where my conf file is, like there is to specify where my log file is.

Traceback (most recent call last):
  File "/usr/bin/ngxtop", line 9, in <module>
    load_entry_point('ngxtop==0.0.1', 'console_scripts', 'ngxtop')()
  File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 446, in main
    process(args)
  File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 409, in process
    access_log, log_format = extract_nginx_conf(config, access_log)
  File "/usr/lib/python2.6/site-packages/ngxtop/ngxtop.py", line 142, in extract_nginx_conf
    with open(path) as conf_file:
IOError: [Errno 2] No such file or directory: '/etc/nginx/nginx.conf'

proxy_cache_path monitoring

It would be great to monitor which file are taking up the most space. If possible, which ones are requested most.

-i argument does not filter in some cases

I run this :

ngxtop --no-follow -i 'upstream_response_time >=2' print upstream_response_time request

and all rows are returned. the -i filter has no effect.

nginx configuration file:
  /etc/nginx/nginx.conf
access log file:
  /var/log/nginx-access.log
access log format:
  $remote_addr [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_time $upstream_response_time $pipe
available variables:
  body_bytes_sent, http_referer, http_user_agent, pipe, remote_addr, request, request_time, status, time_local, upstream_response_time

Custom logformat

Hi !

I enconter an issue with a custum access log format :

$ ngxtop -c /etc/nginx/site-available/unicorn.conf
Error: Incorrect format name set in config for access log file "blablabla/access_log"
My Nginx vhost :
access_log blablabla/access_log vhosts

And my log format is :
log_format vhosts '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';

Thanks a lot !

No module named 'config_parser'

Installed on ARM (Arch) and got this error. Could it be Python 3 related?

$ ngxtop
Traceback (most recent call last):
  File "/usr/bin/ngxtop", line 7, in <module>
    from ngxtop.ngxtop import main
  File "/usr/lib/python3.4/site-packages/ngxtop/ngxtop.py", line 77, in <module>
    from config_parser import detect_log_config, detect_config_path, extract_variables, build_pattern
ImportError: No module named 'config_parser'
$ python -V
Python 3.4.0
$ python2 -V
Python 2.7.6

FreeBSD version

Good day, can we run ngxtop on FreeBSD systems ?
Best regards.

some sort of python problem

Trying to run ngxtop gives me this:

(2:1977)# ngxtop
Traceback (most recent call last):
File "/usr/local/bin/ngxtop", line 9, in
load_entry_point('ngxtop==0.0.2', 'console_scripts', 'ngxtop')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 343, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2309, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2015, in load
entry = import(self.module_name, globals(),globals(), ['name'])
ImportError: No module named ngxtop

I figured out how to run it though but would need a permanent solution to this:

(2:1978)# python2 /usr/local/bin/ngxtop.py

Any ideas?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.