No hourly data downloaded when the account includes an inactive zaehlpunkt ("Returned data does not match given zaehlpunkt!") about wienernetzesmartmeter HOT 40 CLOSED

1. You get your current smartmeter imported as a live sensor ✔️

Correct.

2. You get your current smartmeter imported as a statistical sensor and you can see in your Energy dashboard the historical data imported when you go back in time (the current date though will never show data, as the API is not reporting before the next day) ❓

Partly correct. The sensor with _statistics suffix shows 0 kWh for all days and time frames. The other sensor (without the _statisticssuffix) provides daily data which is correct.

3. For the "legacy" smartmeters the API is not responding useful data, so we report it in the logs. Thus the sensor is never created. (Which would be the expected behaviour) ✔️

Wrong. The sensor is created, but just one and not also the _statistics suffix sensor.

Please correct me if anything here is summarized not correctly, or if you would expect a different behaviour. Please confirm if you can see the historical data for the statistics sensor.

I can see the daily data, but not the data per 15 or 60 minutes, although these charts are available on my Wiener Netze dashboard. And based on the logfiles the reason is most likely that HA unfortunately is requesting data for the inactive zählpunkt I have not been living at the past 3 years and then erroring out due to the inability to process the response.

Once again I want to emphasize that it's very difficult to deliver an experience that is for everyone fitting with this integration, as the API underlies sudden breaking changes and we have no other way than reverse engineering it when it breaks. There is no public documentation or public API which we can leverage. To some extent we are as well dependent on what Wiener Netze responds, which can be wrong and sometimes appear to us as random.

I understand and this is why I have been asking how to enable the display of the entire API payload in the logfiles. I'm willing to analyse what's going wrong and report it as exactly as possible, but without knowing the queries HA sends and the response it gets, I can't report more than I did in all my comments above.

Please let me know what I'd need to add in the code to dump the payload into the debug log, thanks.

from wienernetzesmartmeter.

got you. I'll try to implement more verbose logging in my next change (can't really promise when this will be though) and we can dig into it together afterwards.

Sorry for the back and forth and thanks for your patience

from wienernetzesmartmeter.

@DarwinsBuddy Thanks, much appreciated. I'm removing, rebooting & reinstalling as I'm tying this here.

from wienernetzesmartmeter.

@DarwinsBuddy Is there a way I could send you a (redacted) logfile of ~800 lines? There still is an issue, unfortunately. Thanks!

from wienernetzesmartmeter.

And also the consumed energy (KWh) in the home assistant energy board appears with one day delay. This means in HA I see the consumption of Monday as if it had been consumed on Tuesday. Can this be fixed/improved?

Edit:
Obviously this seems to be a HA limitation, that a data event cannot be imported with a timestamp in the past (#195).

from wienernetzesmartmeter.

As an update: deleting all entities and recreating the entities in the integration did not solve the issue. I'm only seeing daily data. The attribute granularity says QUARTER_HOUR, if that helps.

from wienernetzesmartmeter.

Do you see statistics in the sensor that is unavailable? See also the FAQ for more information: #19

If that is not the case and you still see only daily data: do you see hourly data in the wiener netze dashboard? If no - than there is an issue with the smartmeter itself. If yes, please check the debug log from homeassistant

from wienernetzesmartmeter.

reox, thanks. I can see hourly and 15min charts in https://smartmeter-web.wienernetze.at . In HA only the sensor with daily data contains data. the _statistics sensor has a red exclamation mark icon next to it and it says "unavailable". A lovelace card using the code you mention in "Get a graph of the hourly consumption" with my specific sensor name only shows "No statistics found".

Searching for my sensor name or for "wiener" or "netze" in /config/logs does not return anything in the HA core log. Shall I use the following in configuration.yaml (not sure if the name of the component is correct) and report back after 24 hours?

logger:
  default: warning
  logs:
    custom_components.wiener_netze_smart_meter: debug

…or is my next step to remove everything, reboot and start fresh?

from wienernetzesmartmeter.

ah okay, the "unavailable" is fine, but you should see some statistics...

yes, please add the debug logging for the custom component.
I used

logger:
  default: warning
  logs:
    custom_components.wnsm: debug

But I guess that just depends how your component is named?

from wienernetzesmartmeter.

I actually don't know what a "component" is in HA, is it a developer term? I'm using the integration and have not yet seen a way to rename the integration. I don't know where/how to check which components I have running.

from wienernetzesmartmeter.

I think it only matters how it is named in the custom_components folder, but I'm not sure either... It prints some debug output in any case, thus if it is correctly enabled, you should see some output

from wienernetzesmartmeter.

You can enable debug logging in HA Ui. Open the integration there you can enable debug logging.

from wienernetzesmartmeter.

@donkristo I did press that button and 60 seconds later I pressed it again. The browser started to download a 70MB (!!) logfile. This didn't feel helpful at all.

from wienernetzesmartmeter.

That’s weird. When I click the button there is no download at all.

from wienernetzesmartmeter.

After enabling debug logging and rebooting HA I got the following error:

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:573
Integration: Sensor (documentation, issues)
First occurred: 20:17:02 (1 occurrences)
Last logged: 20:17:02

wnsm: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 573, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1212, in async_device_update
    await self.async_update()
  File "/config/custom_components/wnsm/statistics_sensor.py", line 126, in async_update
    await self._import_historical_data(smartmeter)
  File "/config/custom_components/wnsm/statistics_sensor.py", line 140, in _import_historical_data
    recording = await self.get_historic_data(smartmeter)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/base_sensor.py", line 141, in get_historic_data
    response = await self.hass.async_add_executor_job(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/api/client.py", line 442, in historical_data
    raise SmartmeterQueryError("Returned data does not match given zaehlpunkt!")
custom_components.wnsm.api.errors.SmartmeterQueryError: Returned data does not match given zaehlpunkt!

I'm not sure how do deal with the issue "Returned data does not match given zaehlpunkt!"?

from wienernetzesmartmeter.

I see, so I guess you triggered this TODO:

could you paste a anonymized version of the debug output of Returned data: .... that is directly before this error?

from wienernetzesmartmeter.

The problem is then, that if the historic data import fails, also no new data is imported - or possibly runs into a similar issue...

from wienernetzesmartmeter.

Loading the full log grepping for wnsm does seem to show an issue: When logging into my logwien account in the HA wnsm integration, three entities were created. Two of them share the same zählpunkt, one with _statistics as suffix. These are for the place I'm living. Additionally a third entity was created (without _statistics) for the flat I moved out of 3 years ago. I disabled this entity in HA. wnsm seems to want to use it though:

last 4 digits of Zählpunkt of active apartment: 1977
last 4 digits of Zählpunkt of inactive apartment: 8109

To me the log looks like it's querying the data for the active zaehlpunkt but then attempts to store this data in a statistics_sensor for the old (deactivated) zaehlpunkt. Interestingly the address I can see in the log is for my old apartment, although the query in the log line right above queries the zaehlpunkt of the active address.

2024-01-10 20:23:10.541 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration wnsm which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-01-10 20:23:38.666 DEBUG (MainThread) [custom_components.wnsm.statistics_sensor] Last inserted stat: {}
2024-01-10 20:23:38.667 DEBUG (MainThread) [custom_components.wnsm.statistics_sensor] Last inserted stat: {}
2024-01-10 20:23:39.288 DEBUG (SyncWorker_2) [custom_components.wnsm.api.client] Access Token valid until 2024-01-10 20:28:39.288780
2024-01-10 20:23:39.350 DEBUG (SyncWorker_8) [custom_components.wnsm.api.client] Access Token valid until 2024-01-10 20:28:39.350222
2024-01-10 20:23:39.411 DEBUG (SyncWorker_3) [custom_components.wnsm.api.client] Access Token valid until 2024-01-10 20:28:39.411179
2024-01-10 20:23:39.557 DEBUG (SyncWorker_4) [custom_components.wnsm.api.client] Access Token valid until 2024-01-10 20:28:39.557060
2024-01-10 20:23:39.976 DEBUG (SyncWorker_7) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte
2024-01-10 20:23:39.980 DEBUG (SyncWorker_1) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte
2024-01-10 20:23:39.983 DEBUG (SyncWorker_9) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte
2024-01-10 20:23:40.692 DEBUG (SyncWorker_3) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte
2024-01-10 20:23:41.340 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Starting import of historical data. This might take some time.
2024-01-10 20:23:41.345 DEBUG (SyncWorker_4) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2B/1.0/zaehlpunkte/messwerte?zaehlpunkt=AT-REDACTED-1977&datumVon=2021-01-10&datumBis=2024-01-10&wertetyp=QUARTER_HOUR
2024-01-10 20:23:41.423 DEBUG (MainThread) [custom_components.wnsm.statistics_sensor] Smartmeter {'customerId': 'REDACTED', 'zaehlpunktnummer': 'AT-REDACTED-8109', 'equipmentNumber': 'REDACTED', 'deviceId': 'REDACTED', 'street': 'OLD ADDRESS', 'streetNumber': 'REDACTED', 'zip': 'REDACTED', 'city': 'Wien', 'longitude': 'REDACTED', 'latitude': 'REDACTED', 'type': 'TAGSTROM', 'default': False, 'active': False, 'smartMeterReady': False, 'granularity': ''} is not active
2024-01-10 20:23:42.024 DEBUG (SyncWorker_4) [custom_components.wnsm.api.client] Returned data: {'anlage': None, 'geraet': None, 'verbrauchsstelle': None, 'idex': None, 'zaehlpunktnummer': None, 'zaehlpunktname': None}
2024-01-10 20:23:42.159 ERROR (MainThread) [homeassistant.components.sensor] wnsm: Error on device update!
File "/config/custom_components/wnsm/statistics_sensor.py", line 126, in async_update
File "/config/custom_components/wnsm/statistics_sensor.py", line 140, in _import_historical_data
File "/config/custom_components/wnsm/base_sensor.py", line 141, in get_historic_data
File "/config/custom_components/wnsm/api/client.py", line 442, in historical_data
custom_components.wnsm.api.errors.SmartmeterQueryError: Returned data does not match given zaehlpunkt!
2024-01-10 20:23:42.179 DEBUG (SyncWorker_9) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkt/consumptions
2024-01-10 20:23:42.975 DEBUG (SyncWorker_11) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkt/baseInformation
2024-01-10 20:23:43.652 DEBUG (SyncWorker_4) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkt/meterReadings

from wienernetzesmartmeter.

And here's the 3 entities. As you can see, I renamed the sensors for the active apartment to be sure I select the correct sensor in lovelace. I now wonder if renaming might be a no-go in this case?

from wienernetzesmartmeter.

And here are the "Anlagedaten":

from wienernetzesmartmeter.

hmmm okay, the issue is here:

Returned data: {'anlage': None, 'geraet': None, 'verbrauchsstelle': None, 'idex': None, 'zaehlpunktnummer': None, 'zaehlpunktname': None}

I can just speculate here: Either the API has changed or you have some special device...

What you can do is comment out these lines:

await self._import_statistics(smartmeter, start, _sum)

is started.
Another method would be to insert fake statistics into the database.

If any statistical data is present in the sensor, the historical data import will not trigger and it should work (if there are not any other quirks with your sensor...)

However, there is something else:

Smartmeter {[...]} is not active

And if you look in the screenshot carefully, it says "inaktiv seit 15.10.2021".
Thus, another reason might just be that this Zähler is not active?

from wienernetzesmartmeter.

As mentioned, in my account I have listed two Zähler with each a Zählpunktnummer. One is where I lived until 3 years ago (which is inactive), the other is where I currently live (which is active). In https://smartmeter-web.wienernetze.at/#/anlagedaten I can't see any way to remove the old place I lived in, so I can imagine that some API calls might return data for both zaehlpunkte?

And I commented out lines 124-127 and restarted HA, now I'm seeing the following error:

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:573
Integration: Sensor (documentation, issues)
First occurred: 21:43:27 (1 occurrences)
Last logged: 21:43:27

wnsm: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 573, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1212, in async_device_update
    await self.async_update()
  File "/config/custom_components/wnsm/statistics_sensor.py", line 128, in async_update
    await self._import_statistics(smartmeter, start, _sum)
                                              ^^^^^
UnboundLocalError: cannot access local variable 'start' where it is not associated with a value

Please also see my logfile from before:

THE URL IN THE API CALL CONTAINS THE zaehlpuntnummer FOR MY CURRENT APARTMENT:
2024-01-10 20:23:41.345 DEBUG (SyncWorker_4) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2B/1.0/zaehlpunkte/messwerte?zaehlpunkt=AT-REDACTED-1977&datumVon=2021-01-10&datumBis=2024-01-10&wertetyp=QUARTER_HOUR

THIS IS PROBABLY THE RESULT OF THE QUERY, BUT IT CONTAINS THE ADDRESS OF MY OLD (!) APARTMENT:
2024-01-10 20:23:41.423 DEBUG (MainThread) [custom_components.wnsm.statistics_sensor] Smartmeter {'customerId': 'REDACTED', 'zaehlpunktnummer': 'AT-REDACTED-8109', 'equipmentNumber': 'REDACTED', 'deviceId': 'REDACTED', 'street': 'OLD ADDRESS', 'streetNumber': 'REDACTED', 'zip': 'REDACTED', 'city': 'Wien', 'longitude': 'REDACTED', 'latitude': 'REDACTED', 'type': 'TAGSTROM', 'default': False, 'active': False, 'smartMeterReady': False, 'granularity': ''} is not active

THIS IS ALSO MY OLD APARTMENT, I ASSUME:
2024-01-10 20:23:42.024 DEBUG (SyncWorker_4) [custom_components.wnsm.api.client] Returned data: {'anlage': None, 'geraet': None, 'verbrauchsstelle': None, 'idex': None, 'zaehlpunktnummer': None, 'zaehlpunktname': None}

I might be completely wrong, but based on the logfile I can speculate that the API call returns data for all zaehlpunkte associated with the account and not just for the single zaehlpunkt requested. Is that possible?

from wienernetzesmartmeter.

okay, I see. But unfortunately, I don't know how to resolve this, because I cannot really test it with my account :(
You are probably right, that somewhere the old stuff appears and confuses the script. I'm not sure why, because the zählpunktnummer should be an argument to the calls?

The cleanest solution I can think of would be to add a checkbox during integration setup, where you could disable the import of historical data. Apparently this is what crashes your script.

You can try to download the CSV file in the wiener netze dashboard and while doing this check with the developer tools which endpoint is called. Maybe you can spot a difference between the two!

And I commented out lines 124-127 and restarted HA, now I'm seeing the following error:

ah, yes - the start is missing, because it wasn't set before.
You can try to add something like this the line before the async call:

if init_meter:
    start = datetime.now(timezone.utc).replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(hours=24)

from wienernetzesmartmeter.

Thanks for all your assistance. I understand that it's hard to resolve such issues. Is there any code I could add or activate that will put the entire API payload (request & response) into the debug log? I could then redact it where necessary and post it here.

from wienernetzesmartmeter.

Using the wiener netze dashboard I found two API calls that might be relevant:

https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte returns the zaehlpunkte in an array.

https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/messdaten/[REDACTED KUNDENNUMMER]/[REDACTED ZAEHLPUNKTNUMMER]/verbrauch?dateFrom=2024-01-09T23%3A00%3A00.000Z&period=DAY&accumulate=false&offset=0&dayViewResolution=QUARTER-HOUR returns the 15 minute data, e.g. when switching the axis in the chart from daily to 15minutes resolution.

from wienernetzesmartmeter.

ah, yes - the start is missing, because it wasn't set before. You can try to add something like this the line before the async call:

if init_meter:
    start = datetime.now(timezone.utc).replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(hours=24)

This unfortunately leads to:

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:573
Integration: Sensor (documentation, issues)
First occurred: 18:40:26 (1 occurrences)
Last logged: 18:40:26

wnsm: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 573, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1212, in async_device_update
    await self.async_update()
  File "/config/custom_components/wnsm/statistics_sensor.py", line 130, in async_update
    await self._import_statistics(smartmeter, start, _sum)
                                                     ^^^^
UnboundLocalError: cannot access local variable '_sum' where it is not associated with a value

I then tried to add _sum = 0 before the call, this leads to:

Logger: custom_components.wnsm.statistics_sensor
Source: custom_components/wnsm/statistics_sensor.py:217
Integration: WienerNetzeSmartmeter (documentation, issues)
First occurred: 18:44:14 (1 occurrences)
Last logged: 18:44:14

No values in API response! This likely indicates an API error. Original response: {'optIn': True, 'consumptionAverage': 0, 'consumptionMinimum': 0, 'consumptionMaximum': 0}

from wienernetzesmartmeter.

Just noticed, #191 and #205 both seem to be the same issue.

from wienernetzesmartmeter.

@DarwinsBuddy and @reox Unfortunately the issue is not fixed, please reopen the issue. Here's the same error with version 1.4.3 that was released today:

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:573
Integration: Sensor (documentation, issues)
First occurred: 14:09:47 (1 occurrences)
Last logged: 14:09:47

wnsm: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 573, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1216, in async_device_update
    await self.async_update()
  File "/config/custom_components/wnsm/statistics_sensor.py", line 129, in async_update
    await self._import_historical_data(smartmeter)
  File "/config/custom_components/wnsm/statistics_sensor.py", line 143, in _import_historical_data
    recording = await self.get_historic_data(smartmeter)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/base_sensor.py", line 141, in get_historic_data
    response = await self.hass.async_add_executor_job(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/api/client.py", line 442, in historical_data
    raise SmartmeterQueryError("Returned data does not match given zaehlpunkt!")
custom_components.wnsm.api.errors.SmartmeterQueryError: Returned data does not match given zaehlpunkt!

from wienernetzesmartmeter.

If you installed the plugin manually, you might need to clean up the .pyc files upon upgrade.

from wienernetzesmartmeter.

If you installed the plugin manually, you might need to clean up the .pyc files upon upgrade.

Thanks, but I installed it via HACS two weeks ago.

from wienernetzesmartmeter.

@maia let's try to get to the bottom of this. Can you...

• go to your integrations page and activate debug logging for the wnsm integration
• Reload the integration or restart home assistant
• post again your logs here
  ?

from wienernetzesmartmeter.

@maia let's try to get to the bottom of this. Can you...

• go to your integrations page and activate debug logging for the wnsm integration
• Reload the integration or restart home assistant
• post again your logs here
  ?

Thanks. Here's the current debug log after a HA restart. It's exactly the same as 5 days ago with the previous version. The issue is that the way the integration requests the data leads to a payload containing the (non-existant) data for the old address I no longer live in, instead of requesting the data for the current address. I've redacted the data but kept the last 4 digits of the zaehlpunktnummer and added a note if this is the current address or the old address I've not been living in for 3 years.

2024-01-15 10:49:30.894 DEBUG (MainThread) [custom_components.wnsm.statistics_sensor] Last inserted stat: {}
2024-01-15 10:49:30.894 DEBUG (MainThread) [custom_components.wnsm.statistics_sensor] Last inserted stat: {}
2024-01-15 10:49:31.439 DEBUG (SyncWorker_3) [custom_components.wnsm.api.client] Access Token valid until 2024-01-15 10:54:31.439487
2024-01-15 10:49:31.513 DEBUG (SyncWorker_0) [custom_components.wnsm.api.client] Access Token valid until 2024-01-15 10:54:31.513091
2024-01-15 10:49:31.579 DEBUG (SyncWorker_1) [custom_components.wnsm.api.client] Access Token valid until 2024-01-15 10:54:31.579024
2024-01-15 10:49:31.608 DEBUG (SyncWorker_5) [custom_components.wnsm.api.client] Access Token valid until 2024-01-15 10:54:31.608608
2024-01-15 10:49:32.515 DEBUG (SyncWorker_4) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte
2024-01-15 10:49:32.516 DEBUG (SyncWorker_7) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte
2024-01-15 10:49:32.521 DEBUG (SyncWorker_2) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte
2024-01-15 10:49:32.525 DEBUG (SyncWorker_3) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte
2024-01-15 10:49:34.295 DEBUG (SyncWorker_8) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkt/consumptions
2024-01-15 10:49:34.345 WARNING (MainThread) [custom_components.wnsm.statistics_sensor] Starting import of historical data. This might take some time.
2024-01-15 10:49:34.347 DEBUG (SyncWorker_1) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2B/1.0/zaehlpunkte/messwerte?zaehlpunkt=AT-REDACTED-CURRENT-APARTMENT-1977&datumVon=2021-01-15&datumBis=2024-01-15&wertetyp=QUARTER_HOUR
2024-01-15 10:49:34.419 DEBUG (MainThread) [custom_components.wnsm.statistics_sensor] Smartmeter {'customerId': '1200761562', 'zaehlpunktnummer': 'AT-REDACTED-OLD-APARTMENT-8109', 'equipmentNumber': '9-REDACTED-4', 'deviceId': 'LGZ103-REDACTED-5', 'street': 'OLD STREET NAME', 'streetNumber': 'xxx', 'zip': 'xxx', 'city': 'xxx', 'longitude': 'xxx', 'latitude': 'xxx, 'type': 'TAGSTROM', 'default': False, 'active': False, 'smartMeterReady': False, 'granularity': ''} is not active
2024-01-15 10:49:34.641 DEBUG (SyncWorker_1) [custom_components.wnsm.api.client] Returned data: {'anlage': None, 'geraet': None, 'verbrauchsstelle': None, 'idex': None, 'zaehlpunktnummer': None, 'zaehlpunktname': None}
2024-01-15 10:49:34.664 ERROR (MainThread) [homeassistant.components.sensor] wnsm: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 573, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1216, in async_device_update
    await self.async_update()
  File "/config/custom_components/wnsm/statistics_sensor.py", line 129, in async_update
    await self._import_historical_data(smartmeter)
  File "/config/custom_components/wnsm/statistics_sensor.py", line 143, in _import_historical_data
    recording = await self.get_historic_data(smartmeter)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/base_sensor.py", line 141, in get_historic_data
    response = await self.hass.async_add_executor_job(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/wnsm/api/client.py", line 442, in historical_data
    raise SmartmeterQueryError("Returned data does not match given zaehlpunkt!")
custom_components.wnsm.api.errors.SmartmeterQueryError: Returned data does not match given zaehlpunkt!
2024-01-15 10:49:35.257 DEBUG (SyncWorker_5) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkt/baseInformation
2024-01-15 10:49:36.045 DEBUG (SyncWorker_5) [custom_components.wnsm.api.client] REQUEST: https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkt/meterReadings

Also in case it gets lost in the many comments above, using the wiener netze dashboard I found two API calls that might be relevant

• https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte returns the zaehlpunkte in an array.
• https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/messdaten/[REDACTED KUNDENNUMMER]/[REDACTED ZAEHLPUNKTNUMMER]/verbrauch?dateFrom=2024-01-09T23%3A00%3A00.000Z&period=DAY&accumulate=false&offset=0&dayViewResolution=QUARTER-HOUR returns the 15 minute data, e.g. when switching the axis in the chart from daily to 15minutes resolution.

Is it intended that the integration uses …API_B2B/1.0/zaehlpunkte/messwerte?zaehlpunkt= and the wiener netze dashboard uses API_B2C/1.0/messdaten/[REDACTED KUNDENNUMMER]/[REDACTED ZAEHLPUNKTNUMMER]/verbrauch? Could this lead to solving the issue?

from wienernetzesmartmeter.

Ah now I get it. Well for the integration it's not clear from just the responses of the API if a zaehlpunkt is a legacy one carried along by Wiener Netze or not. So unless you've disabled your sensors associated with that Zaehlpunkt you'll get that error. So if you already know that this sensor will not get you any data just disable it and you're fine.

Please respond if that solved your problem, so I can close the issue.

from wienernetzesmartmeter.

@DarwinsBuddy The sensor of the inactive zaehlpunkt has been disabled in HA, as you can see above #208 (comment). It still doesn't work. I also tried enabling it, but that doesn't help either.

from wienernetzesmartmeter.

did you disable both sensors the live and the statistics one?

from wienernetzesmartmeter.

did you disable both sensors the live and the statistics one?

As you can see in the screenshot in the linked comment above, the inactive zaehlpunkt only has one entity, without _statistics sensor, so I only disabled the one that was generated.

from wienernetzesmartmeter.

Let me summarize once more what the issue is here:

1. You get your current smartmeter imported as a live sensor ✔️
2. You get your current smartmeter imported as a statistical sensor and you can see in your Energy dashboard the historical data imported when you go back in time (the current date though will never show data, as the API is not reporting before the next day) ❓
3. For the "legacy" smartmeters the API is not responding useful data, so we report it in the logs. Thus the sensor is never created. (Which would be the expected behaviour) ✔️

Please correct me if anything here is summarized not correctly, or if you would expect a different behaviour. Please confirm if you can see the historical data for the statistics sensor.

Once again I want to emphasize that it's very difficult to deliver an experience that is for everyone fitting with this integration, as the API underlies sudden breaking changes and we have no other way than reverse engineering it when it breaks. There is no public documentation or public API which we can leverage. To some extent we are as well dependent on what Wiener Netze responds, which can be wrong and sometimes appear to us as random.

from wienernetzesmartmeter.

Also in case it gets lost in the many comments above, using the wiener netze dashboard I found two API calls that might be relevant

• https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/zaehlpunkte returns the zaehlpunkte in an array.
• https://api.wstw.at/gateway/WN_SMART_METER_PORTAL_API_B2C/1.0/messdaten/[REDACTED KUNDENNUMMER]/[REDACTED ZAEHLPUNKTNUMMER]/verbrauch?dateFrom=2024-01-09T23%3A00%3A00.000Z&period=DAY&accumulate=false&offset=0&dayViewResolution=QUARTER-HOUR returns the 15 minute data, e.g. when switching the axis in the chart from daily to 15minutes resolution.

Is it intended that the integration uses …API_B2B/1.0/zaehlpunkte/messwerte?zaehlpunkt= and the wiener netze dashboard uses API_B2C/1.0/messdaten/[REDACTED KUNDENNUMMER]/[REDACTED ZAEHLPUNKTNUMMER]/verbrauch? Could this lead to solving the issue?

Regarding your suggestion. It is worth digging into, though I could not reproduce it unfortunately, as my frontend already uses some ominous new endpoint

https://service.wienernetze.at/sm/api/user/messwerte/bewegungsdaten?geschaeftspartner=YYYYYYYYY&zaehlpunktnummer=ATXXXXXXXXXXXXXXXX&rolle=V001&zeitpunktVon=2023-10-31T23%3A00%3A00.000Z&zeitpunktBis=2023-11-30T22%3A59%3A59.999Z&aggregat=NONE for getting the historical data. I added it to the client, but I am currently not confident enough to change to that, because who knows if everyone has access to it currently.

from wienernetzesmartmeter.

I released a new version with some fixes to the statistics import.
Please remove the whole integration once and reinstall it.
I'll close this ticket, please reopen a new ticket when the issue still persists.

@maia I added also extensive logging if we still run into issues. Let's regroup in a separate issues and discuss from there if you still have troubles. :)

from wienernetzesmartmeter.

thanks for the email ☺️

from wienernetzesmartmeter.