Code Monkey home page Code Monkey logo

aiomonitor's Introduction

aiomonitor

GitHub Actions status for the main branch

codecov.io status for the main branch

Latest PyPI package version

Downloads count

Documentation Status

aiomonitor is a module that adds monitor and cli capabilities for asyncio applications. Idea and code were borrowed from curio project. Task monitor that runs concurrently to the asyncio loop (or fast drop-in replacement uvloop) in a separate thread as result monitor will work even if the event loop is blocked for some reason.

This library provides a python console using aioconsole module. It is possible to execute asynchronous commands inside your running application. Extensible with you own commands, in the style of the standard library's cmd module

An example to run the aiomonitor shell

Installation

Installation process is simple, just:

$ pip install aiomonitor

Example

Monitor has context manager interface:

import aiomonitor

async def main():
    loop = asyncio.get_running_loop()
    run_forever = loop.create_future()
    with aiomonitor.start_monitor(loop):
        await run_forever

try:
    asyncio.run(main())
except KeyboardInterrupt:
    pass

Now from separate terminal it is possible to connect to the application:

$ telnet localhost 20101

or the included python client:

$ python -m aiomonitor.cli

Tutorial

Let's create a simple aiohttp application, and see how aiomonitor can be integrated with it.

import asyncio

import aiomonitor
from aiohttp import web

# Simple handler that returns response after 100s
async def simple(request):
    print('Start sleeping')
    await asyncio.sleep(100)
    return web.Response(text="Simple answer")

loop = asyncio.get_event_loop()
# create application and register route
app = web.Application()
app.router.add_get('/simple', simple)

# it is possible to pass a dictionary with local variables
# to the python console environment
host, port = "localhost", 8090
locals_ = {"port": port, "host": host}
# init monitor just before run_app
with aiomonitor.start_monitor(loop=loop, locals=locals_):
    # run application with built-in aiohttp run_app function
    web.run_app(app, port=port, host=host, loop=loop)

Let's save this code in file simple_srv.py, so we can run it with the following command:

$ python simple_srv.py
======== Running on http://localhost:8090 ========
(Press CTRL+C to quit)

And now one can connect to a running application from a separate terminal, with the telnet command, and aiomonitor will immediately respond with prompt:

$ telnet localhost 20101
Asyncio Monitor: 1 tasks running
Type help for commands
monitor >>>

Now you can type commands, for instance, help:

monitor >>> help
Usage: help [OPTIONS] COMMAND [ARGS]...

  To see the usage of each command, run them with "--help" option.

Commands:
  cancel                  Cancel an indicated task
  console                 Switch to async Python REPL
  exit (q,quit)           Leave the monitor client session
  help (?,h)              Show the list of commands
  ps (p)                  Show task table
  ps-terminated (pst,pt)  List recently terminated/cancelled tasks
  signal                  Send a Unix signal
  stacktrace (st,stack)   Print a stack trace from the event loop thread
  where (w)               Show stack frames and the task creation chain of a task
  where-terminated (wt)   Show stack frames and the termination/cancellation chain of a task

aiomonitor also supports async python console inside a running event loop so you can explore the state of your application:

monitor >>> console
Python 3.10.7 (main, Sep  9 2022, 12:31:20) [Clang 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
---
This console is running in an asyncio event loop.
It allows you to wait for coroutines using the 'await' syntax.
Try: await asyncio.sleep(1, result=3)
---
>>> await asyncio.sleep(1, result=3)
3
>>>

To leave the console type exit() or press Ctrl+D:

>>> exit()

โœ“ The console session is closed.
monitor >>>

Extension

Additional console variables

You may add more variables that can be directly referenced in the console command. Refer the console-variables example code

Custom console commands

aiomonitor is very easy to extend with your own console commands. Refer the extension example code

Requirements

aiomonitor's People

Contributors

achimnol avatar agronholm avatar alefteris avatar apatrushev avatar asvetlov avatar bmerry avatar dependabot[bot] avatar hellysmile avatar imgbot[bot] avatar jettify avatar kinow avatar mtomilov avatar pre-commit-ci[bot] avatar pyup-bot avatar rohitsanj avatar shadchin avatar wbarnha avatar webknjaz avatar yggdr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aiomonitor's Issues

Expose monitor over HTTP

  1. Expose REST api for command executions using aiohttp
  2. Expose web page with form and execute button

Monitor hangs on context manager exit

While fixing issue #162 solved the problem of starting the monitor in an already running loop, it has difficulties shutting down because it first schedules a future to be run on the loop but waits on that future on that loop's own thread, so the loop never has a chance to complete the task.

Initial Update

Hi ๐Ÿ‘Š

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! ๐Ÿค–

Endless loop after exiting console

  1. Connect to remote server with socat/nc
  2. Run console command
  3. Exit from console with ctrl-d/cmd-d

See lots of messages:

monitor >>> Leaving monitor. Hit Ctrl-C to exit

Change the default port numbers to avoid the ephemeral/dynamic range

The current defaults are:

  • monitor (termui): 50101
  • monitor (webui): 50201
  • console: 50102

Since these are inside the default ephemeral/dynamic port range used in Linux and many other operating systems, there are rare cases that aiomonitor causes failure of the application when the host is busy enough to handle many concurrent client connections.
(I actually experienced this situation in one of my customer sites....)

ref: https://en.wikipedia.org/wiki/Ephemeral_port

I'd like to change them to:

  • monitor (termui): 20101
  • monitor (webui): 20102
  • console: 20103

As this is a potentially breaking change for some users who have been relying on the defaults, I'm leaving this as an explicit issue.

Integration with OpenTelemetry, Prometheus, etc.

In both PyCon US and KR talks, there were folks to suggest integration with OpenTelemetry.
https://opentelemetry.io/
https://opentelemetry.io/docs/instrumentation/python/
https://opentelemetry.io/docs/instrumentation/python/manual/

It would be nice to expose some basic metrics like the number of currently running tasks as telemetry stats.

Let's explore which metrics could be useful and how to expose them.

A QnA comment after PyCon APAC 2023 talk:

It would be nice if we could "tag" tasks and get statistics of task grouped by the same tag in metric monitoring systems.

Add the read-only mode

To prevent accidental cancellation of critical tasks when used in production, let's add a read-only mode for both termui and webui.

This is a QnA comment after PyCon APAC 2023 talk.

Traceback on process shutdown

My code looks like this, which is taken from the example in the aiomonitor README:

def main() -> None:  # pragma: no cover
    if "NO_UVLOOP" not in os.environ:
        asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
    logging.basicConfig(
        level=getattr(logging, os.environ.get("LOG_LEVEL", "INFO"))
    )
    logging.getLogger("watchgod.main").setLevel(logging.INFO)
    app = create_web_app()
    loop = asyncio.get_event_loop()
    with aiomonitor.start_monitor(loop=loop, locals={"app": app}):
        web.run_app(
            app, host="0.0.0.0", port=int(os.environ.get("LISTEN_PORT", "80"))
        )

At process shutdown time (after it receives SIGTERM), this traceback always happens:

RuntimeError: Event loop is closed
  File "python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "betslipdisp/__main__.py", line 138, in <module>
    main()
  File "betslipdisp/__main__.py", line 133, in main
    app, host="0.0.0.0", port=int(os.environ.get("LISTEN_PORT", "80"))
  File "aiomonitor/monitor.py", line 126, in __exit__
    self.close()
  File "aiomonitor/monitor.py", line 133, in close
    close_console_server(self._console_server, self._loop)
  File "aiomonitor/utils.py", line 84, in close_console_server
    loop.run_until_complete(coro)
  File "uvloop/loop.pyx", line 1420, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 634, in uvloop.loop.Loop._check_closed

I guess it's because the aiomonitor context manager __exit__ runs after aiohttp has already closed the event loop.

I have aiohttp 3.4.4 and aiomonitor 0.4.0, CPython 3.7, uvloop 0.11.3.

Unable to start monitor with already running event loop.

I am calling the start_monitor function after initializing the loop and I get the following trace:

Traceback (most recent call last): File "/opt/project/code/framework/taurine/daemon.py", line 142, in start_monitor self.monitor = aiomonitor.start_monitor(self.loop, **self.monitor_config) File "/usr/local/lib/python3.7/site-packages/aiomonitor/monitor.py", line 411, in start_monitor m.start() File "/usr/local/lib/python3.7/site-packages/aiomonitor/monitor.py", line 112, in start self._host, self._console_port, self._locals, self._loop) File "/usr/local/lib/python3.7/site-packages/aiomonitor/utils.py", line 73, in init_console_server server = loop.run_until_complete(coro) # type: Server File "/usr/local/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete self.run_forever() File "/usr/local/lib/python3.7/asyncio/base_events.py", line 526, in run_forever raise RuntimeError('This event loop is already running') RuntimeError: This event loop is already running

My use case is that I want to be able to turn monitoring on/off while my python daemon is running. This is a regression from 0.3.1.
I believe the offending code is in the init_console_server function that tries to do loop.run_until_complete. this was previously asyncio.run_coroutine_threadsafe

Announce: The new maintainer

As I have presented aiomonitor-ng (next generation), which is a fork of this project, as a session talk and a poster in PyCon US 2023, @webknjaz has promoted me as the maintainer of this project.

aiomonitor-ng adds:

  • A completely rewritten terminal UI using prompt_toolkit and Click including auto-completion of commands and task IDs
  • Support for concurrent client connections (#140 is resolved!)
  • Task creation chain tracker
  • Task termination history
  • Task cancellation chain tracker
  • Filtering of long task lists using the task name and coro name

I'm going to do the following things during/after the PyCon US 2023 sprints:

  • Backport the changes of aiomonitor-ng
  • Rename the master branch to main
  • Revamp the CI/CD and fix the broken test suite (let it use pytest-asyncio)
  • Add a minimal web UI support in addition to the terminal UI (#84)
  • Diagnose an issue which causes negative side-effects to the application event loop when the telnet server is broken (more details will follow as a separate issue)
  • Pagination support for long task lists (#326)
  • Integrate pystack as an additional option to inspect the deeper side of task stacks

Sprinters and contributors are also welcome!

Add `grep`-wise option on `ps`

Hello, I am glad to have found and be using this library.

However, when dealing with 1,000 tasks at once, I find it difficult to monitor the output using 'ps' on 'monitor'.
Would it be beneficial to have output cutting tools such as grep, tail, or head?

Thanks

v0.4.1 PyPI release doesn't work (PyPI package differs to Github files)

Thanks for releasing v0.4.1 so quickly. Unfortunately the package installed from PyPI has a new error when importing. At least on Python 3.5.6 I haven't tried newer versions.

(venvp3) a@b:~/venvp3$ pip list
Package        Version
-------------- -------
aioconsole     0.1.11 
aiomonitor     0.4.1  
pip            18.1   
setuptools     40.6.3 
terminaltables 3.1.0  
wheel          0.32.3 

(venvp3) a@b:~/venvp3$ python
Python 3.5.6 (default, Aug  4 2018, 00:00:00) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import aiomonitor
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/a/venvp3/lib/python3.5/site-packages/aiomonitor/__init__.py", line 23, in <module>
    from .monitor import (Monitor, start_monitor,
  File "/home/a/venvp3/lib/python3.5/site-packages/aiomonitor/monitor.py", line 20, in <module>
    from .utils import (_format_stack, cancel_task, task_by_id, console_proxy,
  File "/home/a/venvp3/lib/python3.5/site-packages/aiomonitor/utils.py", line 18, in <module>
    def _get_stack(task: asyncio.Task[Any]) -> List[Any]:
TypeError: 'type' object is not subscriptable
>>> 

The code giving the error doesn't exist in the file on Github. Somehow the code on PyPI differs:

a@b:~/tmp$ curl https://files.pythonhosted.org/packages/fe/df/3e01a3683c407edfb6659b0ff9bb9745403609f466861bea246a7962ba5b/aiomonitor-0.4.1.tar.gz -o aiomonitor-0.4.1.tar.gz
a@b:~/tmp$ tar zxvf aiomonitor-0.4.1.tar.gz 
x aiomonitor-0.4.1/
x aiomonitor-0.4.1/CHANGES.txt
x aiomonitor-0.4.1/LICENSE
x aiomonitor-0.4.1/MANIFEST.in
x aiomonitor-0.4.1/PKG-INFO
x aiomonitor-0.4.1/README.rst
x aiomonitor-0.4.1/aiomonitor/
x aiomonitor-0.4.1/aiomonitor/__init__.py
x aiomonitor-0.4.1/aiomonitor/cli.py
x aiomonitor-0.4.1/aiomonitor/monitor.py
x aiomonitor-0.4.1/aiomonitor/mypy_types.py
x aiomonitor-0.4.1/aiomonitor/utils.py
x aiomonitor-0.4.1/aiomonitor.egg-info/
x aiomonitor-0.4.1/aiomonitor.egg-info/PKG-INFO
x aiomonitor-0.4.1/aiomonitor.egg-info/SOURCES.txt
x aiomonitor-0.4.1/aiomonitor.egg-info/dependency_links.txt
x aiomonitor-0.4.1/aiomonitor.egg-info/requires.txt
x aiomonitor-0.4.1/aiomonitor.egg-info/top_level.txt
x aiomonitor-0.4.1/setup.cfg
x aiomonitor-0.4.1/setup.py

a@b:~/tmp$ git clone [email protected]:aio-libs/aiomonitor.git aiomonitor-clone
Cloning into 'aiomonitor-clone'...
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 979 (delta 22), reused 15 (delta 8), pack-reused 935
Receiving objects: 100% (979/979), 1.28 MiB | 536.00 KiB/s, done.
Resolving deltas: 100% (538/538), done.
Checking connectivity... done.
a@b:~/tmp/aiomonitor-clone$ git checkout v0.4.1
Note: checking out 'v0.4.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at fa47d78... bump version

a@b:~/tmp/aiomonitor-clone$ cd ..
a@b:~/tmp$ git diff aiomonitor-0.4.1/aiomonitor aiomonitor-clone/aiomonitor
diff --git a/aiomonitor-0.4.1/aiomonitor/utils.py b/aiomonitor-clone/aiomonitor/utils.py
index 2aea5da..e37b169 100644
--- a/aiomonitor-0.4.1/aiomonitor/utils.py
+++ b/aiomonitor-clone/aiomonitor/utils.py
@@ -15,7 +15,7 @@ from .mypy_types import Loop, OptLocals
 Server = asyncio.AbstractServer  # noqa
 
 
-def _get_stack(task: asyncio.Task[Any]) -> List[Any]:
+def _get_stack(task: asyncio.Task) -> List[Any]:
     frames = []  # type: List[Any]
     coro = task._coro  # type: ignore
     while coro:
@@ -28,7 +28,7 @@ def _get_stack(task: asyncio.Task[Any]) -> List[Any]:
     return frames
 
 
-def _format_stack(task: asyncio.Task[Any]) -> str:
+def _format_stack(task: asyncio.Task) -> str:
     extracted_list = []
     checked = set()  # type: Set[str]
     for f in _get_stack(task):
@@ -49,12 +49,12 @@ def _format_stack(task: asyncio.Task[Any]) -> str:
     return resp
 
 
-def task_by_id(taskid: int, loop: Loop) -> Optional[asyncio.Task[Any]]:
+def task_by_id(taskid: int, loop: Loop) -> Optional[asyncio.Task]:
     tasks = asyncio.Task.all_tasks(loop=loop)
     return next(filter(lambda t: id(t) == taskid, tasks), None)
 
 
-async def cancel_task(task: asyncio.Task[Any]) -> None:
+async def cancel_task(task: asyncio.Task) -> None:
     with contextlib.suppress(asyncio.CancelledError):
         task.cancel()
         await task

Rewrite the docs

Currently https://aiomonitor.aio-libs.org is from the old version.
Let's update to reflect the additions and changes from the aiomonitor-ng, such as:

  • Update screencasts and screenshots
  • Introduce the new commands: where-terminated, ps-terminated
  • Introduce the changed method for adding custom commands to the monitor
  • Describe the difference between monitor.console_locals and locals kwarg in the monitor constructor (console_locals is to inject the local variables after instantiation)

Relax aiohttp dependency

Describe the bug

Is it straightforward to relax the constraint on the aiohttp dependency? We are getting the following error from pip check:

aiomonitor 0.6.0 has requirement aiohttp~=3.8.5, but you have aiohttp 3.9.1. 

Please note that typically aiohttp~=3.8.5 would be fine but as of recently, aiohttp<=3.9 has a critical security vulnerability and containers that use it don't pass the relevant scans.

To Reproduce

Attempt to install a newer versions of aiohttp alongside aiomonitor: pip install aiohttp==3.9.1 aiomonitor==0.6.0.

Result: ERROR: Cannot install aiohttp==3.9.1 and aiomonitor==0.6.0 because these package versions have conflicting dependencies.

Expected behavior

The expected behavior is that the dependency constraint be aiohttp>=3.8.5

Logs/tracebacks

`ERROR: Cannot install aiohttp==3.9.1 and aiomonitor==0.6.0 because these package versions have conflicting dependencies.`

Python Version

3.10 and 3.11

aiomonitor Version

0.6.0

OS

MacOS

Additional context

Thank you for the help!

Code of Conduct

  • I agree to follow the aio-libs Code of Conduct

console command: AttributeError: '_SpecialForm' object has no attribute 'setdefault'

Test program:

#!/usr/bin/env python

import asyncio
import aiomonitor

async def amain():
    while True:
        await asyncio.sleep(5)
        print('async ticking')

loop = asyncio.get_event_loop()
with aiomonitor.start_monitor(loop):
    loop.run_until_complete(amain())

While this is running, I connect to it with: nc localhost 50101 and type console. Immediate crash with:

Task exception was never retrieved
future: <Task finished coro=<handle_connect() done, defined at /home/abc/venv/lib/python3.7/site-packages/aioconsole/server.py:8> exception=AttributeError("'_SpecialForm' object has no attribute 'setdefault'")>
Traceback (most recent call last):
  File "/home/abc/venv/lib/python3.7/site-packages/aioconsole/server.py", line 11, in handle_connect
    interface = factory(streams=streams)
  File "/home/abc/venv/lib/python3.7/site-packages/aiomonitor/utils.py", line 73, in _factory
    locals=locals, streams=streams, loop=loop)
  File "/home/abc/venv/lib/python3.7/site-packages/aioconsole/code.py", line 56, in __init__
    self.locals.setdefault('asyncio', asyncio)
  File "/usr/lib/python3.7/typing.py", line 698, in __getattr__
    return getattr(self.__origin__, attr)
AttributeError: '_SpecialForm' object has no attribute 'setdefault'

Python 3.7.1
aioconsole 0.1.11
aiomonitor 0.3.1
uvloop not installed
Linux

Monitoring of multiple events in one interface

Is your feature request related to a problem?

No response

Describe the solution you'd like

In my program, I use asyncio in multiprocessing. I don't want to create a separate interface for each process. Can I monitor them all in a single interface?

Describe alternatives you've considered

Maybe the web server can be separate from the aiomonitor class, and the web server starts separately. The aiomonitor instance just sends messages to it.

Additional context

No response

Code of Conduct

  • I agree to follow the aio-libs Code of Conduct

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.