StoreOnceG3: get CPU utilization status about python-storage-clients HOT 10 CLOSED

Hello spmatich,
As I understand, you are trying to get CPU statistics from SO Gen3 and SO Gen4 with class StoreOnceG3? Right?

I'll try to reproduce your case.

from python-storage-clients.

yes, I have actually tried to use the Gen4 class but I get the same issue

from python-storage-clients.

I think because it's deprecated endpoint :) This endpoint was introduced for API 2.0 (firmware 3.14), but actual documentation for API 3.1 (firmware 3.18.x) and for 4.x does not contain this enpdoint, but still I'll try to reproduce this case.

from python-storage-clients.

Update: node numbering starts from "1". Your code examples and debug logs use node=0. It's reason why you has received HTTP code 406 "Not Acceptable"

from python-storage-clients.

Could you please run this script for SO4 device?

from hpestorapi import StoreOnceG3
import logging
from pprint import pprint

node = 1
url = f'/cluster/node/{node}/resourceMonitoring/cpu/min/'
params = {'startTime': '2020-05-07T06:00:00Z',
          'endTime': '2020-05-07T06:15:00Z'
          }

if __name__ == "__main__":
    logfmt = ('[%(asctime)s] '
              '%(levelname)-8s '
              '%(filename)-12s:%(lineno)-3d '
              '%(message)s')
    logging.basicConfig(format=logfmt,
                        level=logging.WARNING,
                        filename='messages.log')
    logging.getLogger("hpestorapi.storeonce").setLevel(logging.DEBUG)

    with StoreOnceG3('address', 'login', 'pass') as so:
        so.open()
        status, data = so.get(url, params=params)
        pprint(status, data)

from python-storage-clients.

1. there was a slight issue with that code pprint(status,data) does not seem to like the type
   $ /usr/local/bin/python3 ./testing.py
   Traceback (most recent call last):
   File "./testing.py", line 25, in
   pprint(status, data)
   File "/usr/local/lib/python3.6/pprint.py", line 53, in pprint
   printer.pprint(object)
   File "/usr/local/lib/python3.6/pprint.py", line 139, in pprint
   self._format(object, self._stream, 0, 0, {}, 0)
   File "/usr/local/lib/python3.6/pprint.py", line 176, in _format
   stream.write(rep)
   AttributeError: 'str' object has no attribute 'write'

2. So I modified it to use print('{}'.format(status),'{}'.format(data)) instead
   $ /usr/local/bin/python3 ./testing.py
   406 status=406,exception message=HTTP 406 Not Acceptable
   406 status=406,exception message=HTTP 406 Not Acceptable

3. messages.log below
   $ cat messages.log
   [2020-05-09 01:18:23,814] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/", timeouts=(1.0, None))
   [2020-05-09 01:18:24,151] DEBUG storeonce3.py:152 StoreOnce return status 200, delay 0.336 sec
   [2020-05-09 01:18:24,151] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/
   [2020-05-09 01:18:24,151] WARNING storeonce3.py:162 resp.content=b'NAME359526.35412483595263541248000.0201301004492800158225349632000158225.349632OK1HPE StoreOnce 5650OK1RunningSERIAL201301.00449284.1.2-1914.50Running6000.0/cluster/servicesets/111'
   [2020-05-09 01:18:24,151] WARNING storeonce3.py:163 resp.reason=OK
   [2020-05-09 01:18:24,151] DEBUG storeonce3.py:252 Authentification success
   [2020-05-09 01:18:24,154] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/", timeouts=(1.0, None))
   [2020-05-09 01:18:24,235] DEBUG storeonce3.py:152 StoreOnce return status 406, delay 0.81 sec
   [2020-05-09 01:18:24,235] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/?startTime=2020-05-07T06%3A00%3A00Z&endTime=2020-05-07T06%3A15%3A00Z&media=xml
   [2020-05-09 01:18:24,235] WARNING storeonce3.py:162 resp.content=b'status=406,exception message=HTTP 406 Not Acceptable'
   [2020-05-09 01:18:24,235] WARNING storeonce3.py:163 resp.reason=Not Acceptable
   [2020-05-09 01:19:04,906] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/", timeouts=(1.0, None))
   [2020-05-09 01:19:05,232] DEBUG storeonce3.py:152 StoreOnce return status 200, delay 0.325 sec
   [2020-05-09 01:19:05,232] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/
   [2020-05-09 01:19:05,232] WARNING storeonce3.py:162 resp.content=b'NAME359526.35412483595263541248000.0201301946109952158224408014848158224.408014848OK1HPE StoreOnce 5650OK1RunningSERIAL201301.9461099524.1.2-1914.50Running6000.0/cluster/servicesets/111'
   [2020-05-09 01:19:05,232] WARNING storeonce3.py:163 resp.reason=OK
   [2020-05-09 01:19:05,232] DEBUG storeonce3.py:252 Authentification success
   [2020-05-09 01:19:05,235] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/", timeouts=(1.0, None))
   [2020-05-09 01:19:05,317] DEBUG storeonce3.py:152 StoreOnce return status 406, delay 0.82 sec
   [2020-05-09 01:19:05,317] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/?startTime=2020-05-07T06%3A00%3A00Z&endTime=2020-05-07T06%3A15%3A00Z&media=xml
   [2020-05-09 01:19:05,317] WARNING storeonce3.py:162 resp.content=b'status=406,exception message=HTTP 406 Not Acceptable'
   [2020-05-09 01:19:05,317] WARNING storeonce3.py:163 resp.reason=Not Acceptable
   [2020-05-09 01:22:04,334] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/", timeouts=(1.0, None))
   [2020-05-09 01:22:04,677] DEBUG storeonce3.py:152 StoreOnce return status 200, delay 0.342 sec
   [2020-05-09 01:22:04,677] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/
   [2020-05-09 01:22:04,677] WARNING storeonce3.py:162 resp.content=b'NAME359526.35412483595263541248000.0201307195023360158219159101440158219.15910144OK1HPE StoreOnce 5650OK1RunningSERIAL201307.195023364.1.2-1914.50Running6000.0/cluster/servicesets/111'
   [2020-05-09 01:22:04,677] WARNING storeonce3.py:163 resp.reason=OK
   [2020-05-09 01:22:04,677] DEBUG storeonce3.py:252 Authentification success
   [2020-05-09 01:22:04,679] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/", timeouts=(1.0, None))
   [2020-05-09 01:22:04,760] DEBUG storeonce3.py:152 StoreOnce return status 406, delay 0.80 sec
   [2020-05-09 01:22:04,760] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/?startTime=2020-05-07T06%3A00%3A00Z&endTime=2020-05-07T06%3A15%3A00Z&media=xml
   [2020-05-09 01:22:04,760] WARNING storeonce3.py:162 resp.content=b'status=406,exception message=HTTP 406 Not Acceptable'
   [2020-05-09 01:22:04,760] WARNING storeonce3.py:163 resp.reason=Not Acceptable
   [2020-05-09 01:22:29,870] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/", timeouts=(1.0, None))
   [2020-05-09 01:22:30,176] DEBUG storeonce3.py:152 StoreOnce return status 200, delay 0.305 sec
   [2020-05-09 01:22:30,176] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/
   [2020-05-09 01:22:30,176] WARNING storeonce3.py:162 resp.content=b'NAME359526.35412483595263541248000.0201308454907904158217899216896158217.899216896OK1HPE StoreOnce 5650OK1RunningSERIAL201308.4549079044.1.2-1914.50Running6000.0/cluster/servicesets/111'
   [2020-05-09 01:22:30,176] WARNING storeonce3.py:163 resp.reason=OK
   [2020-05-09 01:22:30,177] DEBUG storeonce3.py:252 Authentification success
   [2020-05-09 01:22:30,179] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/", timeouts=(1.0, None))
   [2020-05-09 01:22:30,260] DEBUG storeonce3.py:152 StoreOnce return status 406, delay 0.80 sec
   [2020-05-09 01:22:30,260] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/?startTime=2020-05-07T06%3A00%3A00Z&endTime=2020-05-07T06%3A15%3A00Z&media=xml
   [2020-05-09 01:22:30,260] WARNING storeonce3.py:162 resp.content=b'status=406,exception message=HTTP 406 Not Acceptable'
   [2020-05-09 01:22:30,260] WARNING storeonce3.py:163 resp.reason=Not Acceptable
   [2020-05-09 01:22:30,261] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/", timeouts=(1.0, None))
   [2020-05-09 01:22:30,561] DEBUG storeonce3.py:152 StoreOnce return status 200, delay 0.299 sec
   [2020-05-09 01:22:30,561] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/
   [2020-05-09 01:22:30,561] WARNING storeonce3.py:162 resp.content=b'NAME359526.35412483595263541248000.0201308454907904158217899216896158217.899216896OK1HPE StoreOnce 5650OK1RunningSERIAL201308.4549079044.1.2-1914.50Running6000.0/cluster/servicesets/111'
   [2020-05-09 01:22:30,561] WARNING storeonce3.py:163 resp.reason=OK
   [2020-05-09 01:22:30,561] DEBUG storeonce3.py:252 Authentification success
   [2020-05-09 01:22:30,562] DEBUG storeonce3.py:136 GET("https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/", timeouts=(1.0, None))
   [2020-05-09 01:22:30,643] DEBUG storeonce3.py:152 StoreOnce return status 406, delay 0.80 sec
   [2020-05-09 01:22:30,643] WARNING storeonce3.py:161 resp.url=https://A.B.C.D:443/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/?startTime=2020-05-07T06%3A00%3A00Z&endTime=2020-05-07T06%3A15%3A00Z&media=xml
   [2020-05-09 01:22:30,643] WARNING storeonce3.py:162 resp.content=b'status=406,exception message=HTTP 406 Not Acceptable'
   [2020-05-09 01:22:30,643] WARNING storeonce3.py:163 resp.reason=Not Acceptable

from python-storage-clients.

Just for comparison here is the curl output. It returns json even though the request is for xml. I notice the G3 does honour the media parameter
curl -XGET --user LOGIN:PASSWORD -k "https://A.B.C.D/storeonceservices/cluster/node/1/resourceMonitoring/cpu/min/?startTime=2020-05-09T00:00:00Z&endTime=2020-05-09T00:05:00Z&media=xml"
{"cpuStats":[{"timestamp":"2020-05-09T00:00:00Z","user":10.0856,"nice":1.09772E-7,"sys":6.18989,"idle":76.7125,"hardIRQ":0.0,"softIRQ":0.0516948,"steal":0.0,"guest":0.0,"ioWait":6.96031,"cpuUsage":23.2875},{"timestamp":"2020-05-09T00:01:00Z","user":12.7132,"nice":8.44398E-9,"sys":6.74111,"idle":75.1604,"hardIRQ":0.0,"softIRQ":0.125011,"steal":0.0,"guest":0.0,"ioWait":5.26031,"cpuUsage":24.8396},{"timestamp":"2020-05-09T00:02:00Z","user":8.91802,"nice":6.49537E-10,"sys":5.40817,"idle":79.6417,"hardIRQ":0.0,"softIRQ":0.0478122,"steal":0.0,"guest":0.0,"ioWait":5.98427,"cpuUsage":20.3583},{"timestamp":"2020-05-09T00:03:00Z","user":10.5212,"nice":4.99644E-11,"sys":5.941,"idle":78.0314,"hardIRQ":0.0,"softIRQ":0.0506142,"steal":0.0,"guest":0.0,"ioWait":5.45579,"cpuUsage":21.9686},{"timestamp":"2020-05-09T00:04:00Z","user":8.31558,"nice":2.411E-4,"sys":5.33836,"idle":80.5304,"hardIRQ":0.0,"softIRQ":0.0425599,"steal":0.0,"guest":0.0,"ioWait":5.77284,"cpuUsage":19.4696},{"timestamp":"2020-05-09T00:05:00Z","user":10.2971,"nice":1.85462E-5,"sys":6.36377,"idle":78.4499,"hardIRQ":0.0,"softIRQ":0.0482906,"steal":0.0,"guest":0.0,"ioWait":4.84098,"cpuUsage":21.5501}]}

from python-storage-clients.

thanks for getting onto this so promptly

from python-storage-clients.

I've found the root of the problem. All library versions use request header "Accept: text/xml" for all StoreOnceG3 requests. It works well for all Gen3 devices. As you can see, old xml-based API was ported from Gen3 to Gen4 devices with undocumented modifications. Gen4 returns only JSON for "resourceMonitoring" endpoint, but client library ask for xml. It's reason why we get HTTP 406. Unfortunately, nobody cares about backward compatibility.

Of course, you should not use deprecated and undocumented enpoints. I have no plans to update StoreOnceG3 interface according with undocumented changes in Gen4. StoreOnce Gen4 has new JSON based API and we should use it. But, I don't need to change library interface to resolve your issue :)

All version from 0.9.4 has optional parameter "headers" for all request types (get(), post(), ...). Before sending the request, library merge user defined headers and two default headers:

• Accept: text/xml
• Content-Type: text/xml

Latest commit 782bdd7 in the branch v0.9.9 allows you to override default headers.

1. Download latest version:

git clone -b v0.9.9 https://github.com/HewlettPackard/python-storage-clients

2. Override 'Accept' header:

with StoreOnceG3('address', 'login', 'password') as so:  
    so.open()  
    status, data = so.get(url, params=params, headers={'Accept': '*/*'})  
    print(status, data)

from python-storage-clients.

thanks again for your prompt attention to this, I can confirm that v0.9.9 update has resolved my issue

from python-storage-clients.