OneView API unable to handle concurrent 'interconnect.get_statistics' calls #396
Description
Scenario/Intent
Retrieve interconnect statistics at 5min collection intervals (Traffic, error data etc. for all interconnect uplinks and downlinks). Each interconnect takes between 6-12 seconds to return its statistics which is too slow for 5 min monitoring, so I am using the Python 'map' function to apply the func 'interconnect.get_statistics(uri)' on an list of Interconnect URIs as to run them concurrently. The returned performance data is then taken and inserted into a Time Series DB for visualization with Grafana.
Environment Details
- OneView SDK Version: hpOneView==4.7.0
- OneView Appliance Version: 3.10.07-0310774
- OneView Client API Version: 300
- Python Version: 3.4.0
- Platform: Windows Server 2012 R2, 64 Bit
Steps to Reproduce
Attempt to query more than 5 interconnects with the 'interconnect.get_statistics(uri)' concurrently on the same OneView API/Appliance.
Expected Result
The OneView API to return the performance/statistics JSON response for each interconnect concurrently.
Actual Result
The OneView API returns the following traceback. I have not had any luck with a Thread Pool > 5, I see this same behavior across multiple appliances.
Traceback (most recent call last):
File ".\performance_oneview_interconnects.py", line 101, in interconnect_stats_uri
result = oneview_appliance.interconnects.get_statistics(uri)
File "C:\Python34\lib\site-packages\hpOneView\resources\networking\interconnects.py", line 91, in get_statistics
return self._client.get(uri)
File "C:\Python34\lib\site-packages\hpOneView\resources\resource.py", line 294, in get
return self._connection.get(uri)
File "C:\Python34\lib\site-packages\hpOneView\connection.py", line 338, in get
raise HPOneViewException(body)
hpOneView.exceptions.HPOneViewException: ('This request cannot be processed at this time.', {'errorCode': 'CRM_CANNOT_PROCESS_REQUEST', 'message': 'This request cannot be processed at this time.', 'errorSource': None, 'details': 'Sufficient resources are temporarily not available to process this request.', 'data': {}, 'nestedErrors': [], 'recommendedActions': ['Please retry this operation later.']})
- Line ('This request cannot be processed at this time.', {'errorCode': 'CRM_CANNOT_PROCESS_REQUEST', 'message': 'This request cannot be processed at this time.', 'errorSource': None, 'details': 'Sufficient resources are temporarily not available to process this request.', 'data': {}, 'nestedErrors': [], 'recommendedActions': ['Please retry this operation later.']})