matthewwall / mtools Goto Github PK

Been trying to troubleshoot frequent hangs of btmon in server mode. btmon runs from one to a few hours before it stops outputting to influxdb. I can't see a specific pattern and the log output isn't helpful from what I can see.

I tried getting more details by doing a python trace:

sudo python -m trace --trace /usr/local/mtools/bin/btmon.py --config-file=/usr/local/mtools/etc/btmon.cfg --debug

(tail of trace output attached)

I have three GEM panels - one sending directly to btmon via its wifi interface. The other two are serial attached to a dashbox, which is forwarding to btmon. All hitting the same btmon port running in a single instance. The only suspicion I have is that the wifi-connected GEM has a spotty wifi signal from time to time. I'm not sure how that would lead to btmon hanging eventually, but it's a thought if it gets stuck mid-packet or something and can't recover. I haven't had any luck improving the wifi signal.

Any assistance in troubleshooting this is greatly appreciated - I don't see anything so far that tell me where to go next.

btmon_trace.txt

Edited Oct. 9. 2016: Since I wrote the issue below, I moved btmon from an older original Raspi1 to a Raspi3. I also changed the trigger power from 60W to 16000W and set upload period from 60s to 15s to match my ECM-1240 sample period setting of 15s. I increased the trigger power so I will always get samples at 15s. Since I made those changes, I have not had any further issues.

Here are some screen captures of issues with latest btmon 3.3.0 on post to OEM (--ip --oem). I believe this is the same issue I have seen in previous versions of btmon. You can see my ECM-1240 CH1 and CH2 watts are periodically spiking at 65536 and the aux channels at 256W. Thanks.

Hello!
It looks like InfluxDB is only supported with HTTP. Please add HTTPS support.

2022/08/11 20:46:44 Exception in InfluxDBProcessor: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))
Traceback (most recent call last):
  File "/usr/local/bin/btmon.py", line 2247, in process
    p.process_compiled(self.packet_collector.packet_buffer)
  File "/usr/local/bin/btmon.py", line 2836, in process_compiled
    self.process_calculated(packets)
  File "/usr/local/bin/btmon.py", line 4304, in process_calculated
    client.write_points(series, tags=self.tags)
  File "/usr/lib/python2.7/dist-packages/influxdb/client.py", line 490, in write_points
    tags=tags, protocol=protocol)
  File "/usr/lib/python2.7/dist-packages/influxdb/client.py", line 551, in _write_points
    protocol=protocol
  File "/usr/lib/python2.7/dist-packages/influxdb/client.py", line 327, in write
    headers=headers
  File "/usr/lib/python2.7/dist-packages/influxdb/client.py", line 267, in request
    timeout=self._timeout
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 686, in send
    r.content
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 828, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 753, in generate
    raise ChunkedEncodingError(e)
ChunkedEncodingError: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))

Thanks,
Peter

I have btmon.py successfully setup polling data from my Brultech GEM, it then sends it to emoncms and an MQTT server. For the most part, everything works great. However, when a value is zero; it is not published to MQTT. I had tried subscribing to everything; '#' topic and the channel showing power consumption in watts would not be present when it is at 0. However, the accumulated watt-hours consumed for that same channel is displayed correctly.

In emoncms, I can see the 0.0 data being posted, and everything graphs out correctly.

Could this be a bug, or a configuration problem on my part?

I am using btmon 3.2.0 with IP in and MQTT out and it is working, but I have a few issues/comments:

I have OBFUSCATE_SERIALS = 1, but the serial is not Obfuscated. I believe it works for OEM output. I can Obfuscate the serial with MQTT mapping, but that's just a workaround.

It would be useful if there was an option for all ECM-1240 data in one payload.

I set INCLUDE_CURRENT = 1, assuming the current would be output, but I saw in the comments it is only for GEM?

I noticed a timestamp is not included in MQTT out, even if upload period is set to sampling time. Can it be added?

Thanks.

I think the following are no longer functional, at least from a public/personal API perspective:

• Wattzon
• MyEnersave
• PeoplePower
• Bidgely
• MyEragy
• Xively

I can send a pull request to nuke them all if I'm correct and if that's helpful, not that removing dead code is rocket science. ;) It is almost 1000 lines of dead code.

[root@lisa ~]# ./btmon.py --ip --ip-host=10.88.64.85 --ip-port=8000 --ip-mode=client --device-type=gem -p -v
2017/07/01 13:56:45 btmon: 3.2.0
2017/07/01 13:56:45 python: 2.7.5 (default, Nov 6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]
2017/07/01 13:56:45 platform: linux2
2017/07/01 13:56:45 device type: gem
2017/07/01 13:56:45 device list: ['']
2017/07/01 13:56:45 packet format: gem48ptbin
2017/07/01 13:56:45 schema: counters
2017/07/01 13:56:45 SOCKET: timeout: 60
2017/07/01 13:56:45 SOCKET: server host: 10.88.64.85
2017/07/01 13:56:45 SOCKET: server port: 8000

I get no output, but if I use a config file and send it to mqtt I see the data. Would be nice if -p would work for testing.

Having issues with btmon in client mode hanging up and giving

expected START_HEADER0 0xfe, got nothing

restarting the btmon.py seems to fix it for awhile but eventually comes back after a few days.

I'd like to write directly to InfluxDB and log with grafana.

I think the format should look like (in InfluxDB line format):

measurement_name XXXX123_ch1_aws=1234.0,XXXX123_ch2_aws=345.0 123456789000000000
measurement_name XXXX789_ch1_aws=1.0,XXXX789_ch2_aws=2.0 123456789000000000

That is, each channel reading should be a field, named for the serial number and channel. (note that since the timestamps and measurement name + tags match, this is a single measurement with 4 fields... i just split it for readability).

An alternative would be

measurement_name,serial=XXXX123 ch1_aws=1234.0,ch2_aws=345.0 123456789000000000
measurement_name,serial=XXXX789 ch1_aws=1.0,ch2_aws=2.0 123456789000000000

which is two separate series. I don't think this is as valuable because there's no reason you'd want to query for all the ch1_aws values across tags.

Another alternative would be

measurement_name,serial=XXXX123,id=ch1_aws value=1234.0 123456789000000000
measurement_name,serial=XXXX123,id=ch2_aws value=345.0 123456789000000000
measurement_name,serial=XXXX789,id=ch1_aws value=1.0 123456789000000000
measurement_name,serial=XXXX789,id=ch2_aws value=2.0 123456789000000000

The problem with this format is that each channel is a separate series, so you can't do interesting things like add two values together. For example, the plugs in my kitchen are on 3 different branch circuits, so I'd like to have a "Kitchen" value that is mean("ch1_aws") + mean("ch2_aws") + mean("ch3_aws") but you can't do that. This is possible with the proposed format above, though.

Hi,

I'm running btmon 3.2.0 and can get data from printing to the console (-p). I would really like to publish to MQTT but I'm getting the following error message:

MQTT Error: paho.mqtt.publish module could not be imported.

sudo python 2btmon.py --serial --serial-port=/dev/ttyUSB2 --mqtt -p
2017/01/12 02:48:19 btmon: 3.2.0
2017/01/12 02:48:19 python: 2.7.9 (default, Sep 17 2016, 20:26:04)
[GCC 4.9.2]
2017/01/12 02:48:19 platform: linux2
2017/01/12 02:48:19 device type: ecm1240
2017/01/12 02:48:19 device list: ['fc']
2017/01/12 02:48:19 packet format: ecm1240bin
2017/01/12 02:48:19 schema: counters
2017/01/12 02:48:19 SERIAL: serial port: /dev/ttyUSB2
2017/01/12 02:48:19 SERIAL: baud rate: 19200
MQTT Error: paho.mqtt.publish module could not be imported.

I thought it might be the version of paho-mqtt but I changed it from 1.2 to 1.1 and I still get the same error.

Any suggestions?

Sharing for others who might have similar problems. I am running mtools on a raspberry pi. When using the systemd templates in the repo, there were a few things that didn't work, so Im sharing my workarounds

1. When copying the "btmon.service" file from /usr/local/mtools/etc/systemd to /etc/systemd/system, it needs to be renamed to "[email protected]". Without this change you will get an error that the service file is missing when you try to enable the btmon instance using the "btmon@" command.
2. My version of raspian uses python3 as the default python interpreter. This wont work with mtools out of the box, and you'll get startup errors due to some syntax differences. Since I have other python scripts running, I didn't want to change the default python to python2 to support the provided systemd service script for btmon. Instead, I edited the [email protected] file and added the path to my python 2 install in the ExecStart command:

ExecStart=/usr/bin/python2.7 /usr/local/mtools/bin/btmon.py --quiet --config-file=/usr/local/mtools/etc/btmon-%I.cfg

Hopefully this will be helpful for others!

The docs talk about a configuration script for the Brultech units, but it is not in this distribution. Was it removed intentionally?

Maye this is a mysql admin problem and not a btmon.py problem, but I really don't know.

I have a client on a rPi collecting data from an ECM1240 and committing it to a remote mysql server:

[mysql]
mysql_out = true
mysql_insert_period = 30
mysql_user = ecmuser
mysql_passwd = ecmpass
mysql_host = 10.0.0.100
mysql_database = ecm
mysql_table = ecmdata2

and on the host running the server, I pick it up with another btmon.py instance, and send it to all the various services:

[source]
mysql_read = true
mysql_poll_interval = 30
mysql_src_host = localhost
mysql_src_user = ecmuser
mysql_src_passwd = ecmpass
mysql_src_database = ecm
mysql_src_table = ecmdata2

If I convert my tables from MyISAM to InnoDB, the read side sees no updates. If I convert back to MyISAM it starts working again. Have I set something up wrong or is one of the btmon clients doing the wrong thing?