Getting multiples HTTP 401 when polling camera.get_battery_level about python-arlo HOT 17 CLOSED

@tchellomello This async idea works really well.

I have made some optimizations in the pyarlo library code. I will post for PR later tomorrow. This optimization reduced the time consumed to subscribe, and fetch each property from 5 seconds to 0.7 seconds.

In addition to that, with the async thread in home assistant, I have added the battery sensors as well as displaying the battery & motion_detection status in the camera attributes/properties. All those are working well without errors as well as not slowing down HASS.

I will try to share that code later this weekend for you to try, if you would be interested.

from python-arlo.

@tchellomello After we register the timer in the async_setup code we have to call the function once manually to update the data. After that, every SCAN_INTERVAL it will update the properties automatically. I haven't done that yet in the branch.

Otherwise for the first time, it will take some time to get the sensor value. If you restart HASS you will see that the sensor gets reading almost immediately because there is some value first available in DB.

Also we cannot reduce to below 15 seconds because, in the update properties function where we are doing this sequence, we are subscribing, then populating around 4 properties and then unsubscribing, which could potentially take more than 10 seconds. If you set the timer to 10 seconds, you might potentially end up calling an instance of the function when previous instance is executing causing such issues.

To avoid that, I am going to change the timer such that it will call the update properties function and then when it is done, it will reset the timer for (current time + 10 seconds). That way we can try to increase the polling frequency and yet populate the parameters properly.

I will update the branch code probably this weekend and send the code across for you to verify.

BTW, I have never seen the urllib3 error message in my HASS with this new code. Not sure if you are seeing it because of reducing the scan interval.

Let me implement these things that I mentioned above and we will give that a try. Most probably that should take care of these issues I believe.

from python-arlo.

@viswa-swami To reproduce that on HA, you can checkout the branch https://github.com/tchellomello/home-assistant/tree/arlo_battery

from python-arlo.

@tchellomello I will pull that and try it out.

I think we can try these 2 options that I have mentioned below first. If that doesn't work out, then we may have to add the HTTPAdapter methodology.

Option-1:
I was working on adding the code to set the "motion_enabled" status from base station to HASS 0.48.1. I have made some change to pyarlo for optimization. We have to call 'subscribe' from the HASS (components/arlo.py) right after login so that HASS (as a client) is subscribed to Arlo system. Then we can just quickly post a query to arlo and get a response relatively quickly instead of each time, subscribing (which in turn starts a thread) and query+response and unsubscribe (which stops the thread).

Earlier when I added the service to enable/disable motion detection, we didn't have this capability so we used a local variable in HASS to set a flag for motion_enabled status.

For that above mentioned setup to work, a minor change was required in pyarlo to not subscribe before posting the query, if already subscribed. I will push that change in a day or so. My old HDD failed in my home system and had to replace/reinstall everything !!!!!!

Once the subscribe is added after the login function in components/arlo.py, that should optimize to some extent.

Option-2:
Another possible option to try is, instead of using the camera.get_battery_level, we can use base_station.get_camera_battery_levels and parse for each camera ourselves in HASS. That way, there needs to be only 1 query sent to arlo system and anyway HASS has access the camera device id. So we can filter there. The dict should not be that big anyway i guess.

from python-arlo.

@tchellomello I took the branch that you indicated, and merged those sensor changes into my code-base... But I am not seeing those Battery sensors getting created at all !!! Is any other change missed in that branch that you gave ?

from python-arlo.

@tchellomello Ignore my previous comment. It was my bad... I didnt add battery to my configuration file and thats why it didn't show up when i tried !!!! Let me try and get back to you

from python-arlo.

@tchellomello Ok, I got the error that you were talking about in your comment. I will try to debug that further tomorrow and let you know if i find anything....

from python-arlo.

@tchellomello I am getting spurious results... not sure why... sometimes I am getting that error that you reported.. while if i restart HASS I am not getting that error... !!!!!!!!! you noticing the same behavior ?

from python-arlo.

@viswa-swami I have not noticed it. I'll try to play with that later today

from python-arlo.

@tchellomello I was trying few experiments with this setup. I was also working on adding some of those parameters are attributes/properties for the camera like brightness, signal strength etc.

From the perspective of HASS, I think it might be a good idea to create a separate thread (or a async_task) whichever would be appropriate in the components/arlo.py which will subscribe and get all those properties from the base station under regular intervals, say every 5 minutes or 30 seconds for example. It will populate the properties in a variable.

The sensors and camera components can just get the value from the respective variable whenever required.

Without this, with multiple sensors, and/or these properties for each camera like brightness, etc, it is slowing down the HASS since each request takes some time to get response from the base station.

I will try to implement this as a dirty code first and try it out. Meanwhile let me know what you think.

I think this will serve as better optimization from the HASS point of view to incorporate the Arlo sensors and properties.

from python-arlo.

@viswa-swami yes, +1 to the async idea.

Yes that is a good idea and will offer a good optimization. I'm running some tests here

from python-arlo.

Yes please share then I can test as well. working for me then we can post it for the HASS version.

Very nice to hear that!! Good job @viswa-swami !

from python-arlo.

@tchellomello Below is the Home Assistant branch that contains the code for arlo async sensors and attributes. This is based on latset 'dev' branch as of today.

The mainly modified files are components/arlo.py, components/camera/arlo.py and components/sensor/arlo.py for the respective functions.

Please note that, you would need the latest pyarlo including the PR #34 that I have posted a short while ago. First install that latest version of pyarlo including PR #34 and then try these Home Assistant code-set. You will be able to see sensor.battery_<camera_name> as well as sensor.signal_<camera_name>.

If you enable the motion detection using camera service (camera.enable_motion_detection), the 'motion_detection' attribute for the camera entity will mention as 'true'. This was available in 0.48.1 HASS itself, but now the status is live from the arlo base-station itself.

I am also planning to add the 'mirror', 'flip' and 'sensitivity' as attributes listed for each camera entity. I will keep adding that in the background while you try this out.

Let me know how it goes once you have tried it.

from python-arlo.

Sorry, forgot to paste the link for the Home Assistant branch:

https://github.com/viswa-swami/home-assistant/tree/newly_added_async_arlo_sensors_properties

from python-arlo.

@viswa-swami pyarlo 0.0.6 published on PyPi.

I tested your HA branch and I also updated it to make use of the new version published on PyPi too.

I got the sensors displayed on the webUI, however it took 90 seconds to the battery and signal sensors populated.
I believe that is because of the SCAN_INTERVAL. Just to test it, I changed it to 10 seconds, and after the HASS were up, it took 10 seconds to update it. Interesting that the other sensors were updated as soon as the HASS started. Do you think that is because of the subscribe process?

When starting HASS, I got some messages like below:

2017-07-20 02:12:59 WARNING (MainThread) [homeassistant.components.sensor] Setup of platform arlo is taking over 10 seconds.
2017-07-20 02:13:06 WARNING (SyncWorker_7) [requests.packages.urllib3.connectionpool] Connection pool is full, discarding connection: arlo.netgear.com

2017-07-20 02:13:22 WARNING (SyncWorker_13) [requests.packages.urllib3.connectionpool] Connection pool is full, discarding connection: arlo.netgear.com

The message is still displayed but more sporadic now.

Also I with the scan_internval set to 10 seconds I hit the problem below:

Exception in thread Thread-10:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mdemello/.virtualenvs/ha-py36/lib/python3.6/site-packages/pyarlo/base_station.py", line 45, in thread_function
    for event in (self.__sseclient).events():
  File "/home/mdemello/.virtualenvs/ha-py36/lib/python3.6/site-packages/sseclient/__init__.py", line 58, in events
    for chunk in self._read():
  File "/home/mdemello/.virtualenvs/ha-py36/lib/python3.6/site-packages/sseclient/__init__.py", line 48, in _read
    for chunk in self._event_source:
TypeError: 'NoneType' object is not iterable

Maybe to address this error we can check if (self.__sseclient).events() with a try condition. What do you think?

from python-arlo.

Interesting that after running for some time, some battery sensors zeroed out.

After some minutes, it got re-populated.

from python-arlo.

The PR #52 should address this problem.

from python-arlo.