highvolt-dev / tmo-monitor Goto Github PK

Have you considered allowing for an email or some other options that just basically says the gateway was rebooted? Email would have the timestamp. Might indicate if the issue is overheating. telegram-cli would have been perfect except it gives an error on pi and needs to have 2 lines commented out in the source then recompiled.

Since firmware version 1.2103.00.0338 - #57 fixed #56 for cases where there are no extra hashing iterations required for the password. Need to support cases where multiple iterations are expected.

Substantial work from @AndrewPardoe has lead to a new upcoming major version release, 2.0. This issue is intended to track any feedback regarding this release.

Specifically:

• .env file and configuration override nuances
• logging behavior
• ping behavior (the ping implementation was slightly changed when extracting latency information for logging) - need to confirm that no regressions are introduced, particularly with Windows in the case of a "Network not reachable" edge case.

The script is not working with the Nokia Firmware Upgrade to version 1.2104.00.0286

Using Nokia 5G "trashcan" on firmware 1.2104.00.0286

2022/06/15 00:20:38 [DEBUG] Starting new HTTP connection (1): 192.168.12.1:80
2022/06/15 00:20:39 [DEBUG] http://192.168.12.1:80 "GET /dashboard_device_info_status_web_app.cgi HTTP/1.1" 200 649
2022/06/15 00:20:39 [INFO] Reboot requested.
2022/06/15 00:20:39 [INFO] Rebooting.
2022/06/15 00:20:39 [DEBUG] Starting new HTTP connection (1): 192.168.12.1:80
2022/06/15 00:20:39 [DEBUG] http://192.168.12.1:80 "GET /login_web_app.cgi?nonce HTTP/1.1" 200 369
2022/06/15 00:20:39 [DEBUG] Starting new HTTP connection (1): 192.168.12.1:80
2022/06/15 00:20:39 [DEBUG] http://192.168.12.1:80 "POST /login_web_app.cgi HTTP/1.1" 200 32
2022/06/15 00:20:39 [CRITICAL] Could not post reboot request, exiting.
(program terminates here)

Doesn't seem to be able to reboot the router, so it just exits?

Also all of this happens within a few seconds, it's not like it sends an initial reboot command then waits for the router, then fails.

FYI, I created a Docker container version of your tool at https://github.com/hugoh/docker-tmo-monitor. The image is published at https://hub.docker.com/r/hugohh/tmo-monitor.

It's been working fine for me. I'm happy to provide a pull request. It will entail Docker Hub and GitHub Actions to perform the automated builds.

Thank you for your tool.

At given times of the day and especially in bad weather you can experience "drop outs" that can last for 20-30 seconds. Right now this will trigger a reboot that adds another 2 minutes into the process. If the monitor was to do 4 pings 10 seconds apart before triggering a reboot this would allow for those moments of drop out or latency.

This seemed to be different enough from the "enhance ping" issue where it talks about multiple hosts vs multiple pings across a timeframe interval.

Per https://tmo.report/2021/12/this-is-the-new-t-mobile-5g-home-internet-gateway/ there is likely a new 5G Gateway coming for T-Mobile Home Internet. Ideally this script should be compatible with more than just the Nokia 5G Gateway.

Add linting rules for better consistency of code. Incorporate in GitHub Actions.

Add pre-defined band groups, e.g. low-band, mid-band, Sprint, T-Mobile, anchor bands (i.e. B2/B66), etc.

Follow-up to #10

Bootloop prevention is currently implemented with an uptime check.

Add statefulness to prevent boot loops so that a given number of consecutive failures or a failure of a certain duration causes different behavior than endlessly rebooting.

Add diagnostic logging with loglevel support. Separate from #11

day = uptime // (24 * 3600)
uptime = uptime % (24 * 3600)
hour = uptime // 3600
uptime %= 3600
minutes = uptime // 60
uptime %= 60
seconds = uptime

Not sure where to insert this, without possibly breaking stuff.

When I remove the first six characters (T-Mobile ID), then divide the remaining 8 digit number by 256, the tower ID matched what cellmapper.net showed for my connected tower. Discard the numbers after the decimal point, to get the correct enBID. I would be amenable to letting you ssh into my Linux Box, running Fedora 35, if you care to test it with a live unit. I do appreciate this effort you are putting out, for all of us who are using the Arc box. Using CGI on LTE tab, as the source, peel off the first 6 characters, then divide the remaining 8 digit number by 256, drop the characters after the decimal point, should give cell tower enBID that match the stuff reported to cellmapper.net. Hope this helps. Great work on this tool.

It would be nice to log to syslog in addition to a file.

I started the implementation at https://github.com/hugoh/tmo-monitor/tree/log-to-syslog but I'm struggling to get logging's SysLogHandler to do what I want. I need to work more on it.

capture_output kwarg for subprocess.run was introduced in Python 3.7 - instead use stdout=subprocess.PIPE, stderr=subprocess.PIPE

Reported by reddit user /u/Tractor_Boy_500

I had issues when my Linux system didn't apparently handle IPv6 pings to Google very well, and for some reason tmo_monitor/gateway/base.py wanted to use IPv6 ping. It would be really nice to either explicitly force ping -4 by default, or allow the user to specify ping -4 as a command line option.

In my configuration I wanted to make double sure it didn't use IPv6 ping, so I just hardcoded base.py to add the -4 flag...

edit2: oops 🤦 I completely missed that the .env file now contains flags to reboot based on 4G/5G bands, default false. Might be worth adding these flags as CLI overrides since this is a change from the default behavior of v1? I can send a PR if the maintainers are open to it :)

original issue below
Just pulled the latest version and restarting based on 5G bands isn't working. My router is camping on n71 but running with -5 n41 doesn't cause a restart like it does on the v1 branch.

edit: I'm running MacOS 12.0.1

Logs:

> ./tmo-monitor.py admin [redacted] -5 n41 --print-config --log-all --logfile /dev/stdout
Password:
Script configuration:
  Login info:
    Username: admin
    Password: [redacted]
  Ping configuration:
    Host: google.com
    Count: 1
    Interval: 10
  Connection configuration:
    Secondary band: ['n41']
  Reboot settings:
    Reboot now: False
    Skip reboot: False
    Min uptime: 90
  Reboot on:
  General settings:
    Log file: /dev/stdout
    Log all: True
    Log delta: False

eNB ID check passed, on 27120.
Camping on B66.
Camping on n71.
No reboot necessary.
4G: B66 | 5G: n71 | eNB ID: 27120 | Avg Ping:  ms | Uptime: 223 sec
2022/01/08 02:36:36 [INFO] 4G: B66 | 5G: n71 | eNB ID: 27120 | Avg Ping:  ms | Uptime: 223 sec

Thanks for all your work on this! My trashcan is finally usable 🙏

Hey there! First, thanks for the script!

I wanted to add (local) logging, but before I hacked on it I figured I'd add support for .env files and clean up the argument parsing a bit. I tried not to change the semantics but I haven't thoroughly tested yet.

What I've got is a pretty major refactor that's just one commit behind your repo's status.

https://github.com/AndrewPardoe/tmo-monitor/blob/config/tmo-monitor.py

If you like the refactor, I'm happy to integrate your multiple ping logic & submit a PR to merge everything back. If you don't like the refactor, I'll probably maintain a fork. I'm still testing so I haven't created a PR yet.

Please LMK.

I downloaded the 2.0.4 zip release and upgraded over my older working release.

pi@p314:~/tmo-monitor-main/bin $ ./tmo-monitor.py
Traceback (most recent call last):
File "./tmo-monitor.py", line 8, in
import tailer
ModuleNotFoundError: No module named 'tailer'

Add statefulness to prevent boot loops so that a given number of consecutive failures or a failure of a certain duration causes different behavior than endlessly rebooting.

As an alternative connectivity test, use http://google.com/generate_204 and check the status code. Allow custom host and status code.

Executing:

tmo-monitor.py --ping-count 3 -4 B2 -5 n41 -5 n71 --model ARCKVD21

When the Ark device isn't connected to a 5g band, the JSON no longer contains a 5g section in the output, resulting in the following error:

2022/08/05 10:04:00 [DEBUG] Starting new HTTP connection (1): 192.168.12.1:80
2022/08/05 10:04:00 [DEBUG] http://192.168.12.1:80 "GET /TMI/v1/gateway?get=all HTTP/1.1" 200 906
Traceback (most recent call last):
File "/home/treddy/.local/bin/tmo-monitor.py", line 83, in
signal_info = gw_control.get_signal_info()
File "/home/treddy/.local/lib/python3.9/site-packages/tmo_monitor/gateway/arcadyan.py", line 69, in get_signal_info
nr_info = info['signal']['5g']['bands']
KeyError: '5g'

Output from gateway endpoint:

{
"device": {
"friendlyName": "5G Gateway",
"hardwareVersion": "R01",
"isEnabled": true,
"isMeshSupported": true,
"macId": "",
"manufacturer": "Arcadyan",
"manufacturerOUI": "001A2A",
"model": "KVD21",
"name": "5G Gateway",
"role": "gateway",
"serial": "",
"softwareVersion": "1.00.16",
"type": "HSID",
"updateState": "latest"
},
"signal": {
"4g": {
"bands": [
"b12"
],
"bars": 5.0,
"cid": 22,
"eNBID": ,
"rsrp": -73,
"rsrq": -6,
"rssi": -67,
"sinr": 17
},
"generic": {
"apn": "FBB.HOME",
"hasIPv6": true,
"registration": "registered",
"roaming": false
}
},
"time": {
"daylightSavings": {
"isUsed": false
},
"localTime": 1659687085,
"localTimeZone": "<-06>6",
"upTime": 306
}
}

macOS 12.4
Python 3.9.13
tmo-monitor 2.0.0 beta6

All dependencies installed, as far as I know. The main script crashes with:

$ ./bin/tmo-monitor.py Password: 2022/06/20 18:42:29 [DEBUG] Starting new HTTP connection (1): 192.168.12.1:80 2022/06/20 18:42:29 [DEBUG] http://192.168.12.1:80 "GET /fastmile_radio_status_web_app.cgi HTTP/1.1" 200 3318 Traceback (most recent call last): File "/Users/Administrator/Downloads/tmo-monitor-2.0.0-beta6/./bin/tmo-monitor.py", line 83, in <module> signal_info = gw_control.get_signal_info() File "/usr/local/lib/python3.9/site-packages/tmo_monitor/gateway/nokia.py", line 141, in get_signal_info info = signal_request.json() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Sorry, Python noob here.

On an uppdated RaspberryPi
Python 3.9.2
pip 20.3.4 from /usr/lib/python3/dist-packages/pip (python 3.9)
pip3 install .
yields
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.

Help

Do I need to download the project and unpack it in the home directory?

Using a distinct status code when exiting after rebooting can allow monitoring tools to introspect that a reboot occurred without scraping logs.

When running latest main on macOS 12.0.1, ping checks fail with:

PING google.com (142.250.191.238): 56 data bytes
64 bytes from 142.250.191.238: icmp_seq=0 ttl=110 time=239.159 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 239.159/239.159/239.159/0.000 ms

Traceback (most recent call last):
  File "/Users/adelikat/src/tmo-monitor/./tmo-monitor.py", line 417, in <module>
    ping_ms = tc_control.ping(config.ping['ping_host'], config.ping['ping_count'],
  File "/Users/adelikat/src/tmo-monitor/./tmo-monitor.py", line 157, in ping
    result = ping_time(i)
  File "/Users/adelikat/src/tmo-monitor/./tmo-monitor.py", line 154, in ping_time
    return round(float(ping_ms.group(1)))
AttributeError: 'NoneType' object has no attribute 'group'

This happens because macOS prints stddev instead of mdev.

When the Arcadyan gateway returns the status of the signal, it doesn't always provide both 4g and 5g bands if not connected. For example, my gateway pulls this currently:

{ "device": { "friendlyName": "5G Gateway", "hardwareVersion": "R01", "isEnabled": true, "isMeshSupported": true, "macId": MACADDRESS, "manufacturer": "Arcadyan", "manufacturerOUI": "001A2A", "model": "KVD21", "name": "5G Gateway", "role": "gateway", "serial": SERIAL, "softwareVersion": "1.00.16", "type": "HSID", "updateState": "latest" }, "signal": { "4g": { "bands": [ "b66" ], "bars": 2.0, "cid": 3, "eNBID": 778528, "rsrp": -116, "rsrq": -13, "rssi": -103, "sinr": 0 }

When this happens, there is a key error for ['signal']['5g']['bands'].

I have a pi with a fresh install of the latest version.

What do I need to do to install and then run this script?

I can run pip3 and, after a while, it returns a string of command line options.

I am a noob at python.

Have you looked at this...

https://github.com/geerlingguy/internet-pi

I have been running it for 3 months. It works with any ISP.

My TMHI was terrible during the day until about 3 weeks ago. Speeds are not great but very usable.

Thanks for sharing your work.

Dennis

internet=pi output...

The ping command on Windows has different output than on Linux which causes parsing issues:

Approximate round trip times in milli-seconds:
    Minimum = 39ms, Maximum = 46ms, Average = 42ms

Running on a raspberry pi, It is not seeing the .env file when I run python3 bin/tmo-monitor.py

I put the .env in the bin folder, then second guessed myself and put in in the tmo-monitor-2.0.0-beta4 folder. No luck either way.

When the script is not ran interactively and no password is supplied, need to output an error.

IPv4 addresses are supported at ping hosts (e.g., 8.8.8.8), but IPv6 (e.g., 2001:4860:4860::8888) isn't on Linux as it requires the use of ping6 instead of ping.

It would be nice to support both.

The script is not currently compatible with Windows due to its ping implementation. Conditionally translate ping flags to Windows equivalents. Check output for indications that Destination Host Unreachable.

For now, running with --skip-ping will work on Windows if that feature is not needed.

Provide method(s) for credentials to be re-used.

Because web api authentication uses hashing and app api authentication uses cleartext passwords, cannot simply just use hashed value.

Options include storing password in cleartext or encrypted at rest and providing the decryption key at runtime, e.g. with an environment variable. If storing in cleartext, should ensure read permissions are as strict as possible.

I am running the latest code. This morning, my internet dropped out. When I looked at logs I got the following:

2022/01/11 07:00:04 [INFO] 4G: B2 | 5G: n41 | eNB ID: 28646 | Avg Ping: 44 ms | Uptime: 158930 sec
2022/01/11 07:05:01 [CRITICAL] Could not post login request, exiting.
2022/01/11 07:05:01 [CRITICAL] Could not query site info, exiting.
2022/01/11 07:06:09 [ERROR] Could not ping google.com.
2022/01/11 07:06:09 [CRITICAL] Could not query modem uptime, exiting.
2022/01/11 07:06:17 [CRITICAL] Could not post login request, exiting.
2022/01/11 07:06:17 [CRITICAL] Could not query site info, exiting.
2022/01/11 07:06:43 [CRITICAL] Could not post login request, exiting.
2022/01/11 07:06:43 [CRITICAL] Could not query site info, exiting.

Any ideas why? I was able to log into my gateway from my laptop, so it was still on my local network. As soon as the reboot was complete, everything was fine and things were being logged like normal again.

2022/01/11 07:10:03 [INFO] 4G: B66 | 5G: n41 | eNB ID: 28646 | Avg Ping: 59 ms | Uptime: 94 sec
2022/01/11 07:12:46 [INFO] 4G: B66 | 5G: n41 | eNB ID: 28646 | Avg Ping: 84 ms | Uptime: 257 sec

@AndrewPardoe did you accidentally have extraneous dependencies included into requirements.txt when you ran pip freeze?

E.g. pandas and numpy

Add --silent flag to suppress output to stdout. This will prevent cron from needing to redirect output.

Handle DNS failures in a more explicit way.

Pinging an IP address (e.g. 1.1.1.1 or 8.8.8.8) can indicate whether there is network connectivity regardless of DNS functionality.

Likewise, more explicitly checking that DNS resolution is working may be a desirable but separate health check.

Add CGI check (from Mobile App endpoints) or introspect tower based on connected bands and/or PCI/etc (disambiguate Sprint vs Native LTE B41).

Allow specifying multiple ping hosts, with the ability to reboot if either any or all of them fail (flags can specify the behavior).

I grabbed the latest version and the logging is not working. Where it was on 12/30. Also the print configuration on the latest version is not working.

It looks like T-Mobile started blocking pings, and so the connection looks down even when it's not. This broke this tool for me – I'm assuming for others too.

We should add a different mechanism, such as DNS lookup or HTTP request.

Submit to PyPI for expanded user reach and a more accessible route to using the script via pip install.

The update to firmware version 1.2103.00.0338 has broken the reboot API POST request.

I see there are no unit tests.

Before I jump in head first writing tests, would you find it useful for this project?

I tried various choices and it looks to me like every time it's still only doing 1 ping.

pi@p314:~ $ /home/pi/tmo-monitor-main/tmo-monitor.py --ping-count 3 --skip-bands admin password
PING google.com (142.250.64.238) 56(84) bytes of data.
64 bytes from mia07s57-in-f14.1e100.net (142.250.64.238): icmp_seq=1 ttl=109 time=47.3 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 47.275/47.275/47.275/0.000 ms

No reboot necessary.
pi@p314:~ $ /home/pi/tmo-monitor-main/tmo-monitor.py --skip-bands --ping-count=1 admin password
PING google.com (142.250.64.238) 56(84) bytes of data.
64 bytes from mia07s57-in-f14.1e100.net (142.250.64.238): icmp_seq=1 ttl=109 time=54.8 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 54.781/54.781/54.781/0.000 ms

No reboot necessary.
pi@p314:~ $ /home/pi/tmo-monitor-main/tmo-monitor.py --skip-bands --ping-count=3 admin password
PING google.com (142.250.64.238) 56(84) bytes of data.
64 bytes from mia07s57-in-f14.1e100.net (142.250.64.238): icmp_seq=1 ttl=109 time=39.6 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 39.575/39.575/39.575/0.000 ms

Add logging of status checks so that the output can be used for analysis or with time series databases, graphing, and monitoring tools.

Add ability to specify output format (e.g. json)

Add tests and setup GitHub Actions to automatically run test suite to detect regressions.

Need to mock API responses.

The README needs to explain the currently implemented logging options/configuration.