Rate Limiter on NLCD 2021

[colintle]

What happened?

I am calling nlcd.bycoords() with a list of coordinates passed in. I keep getting connection errors below

Traceback (most recent call last):
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp\connector.py", line 1025, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\asyncio\base_events.py", line 1121, in create_connection
raise exceptions[0]
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\asyncio\base_events.py", line 1103, in create_connection
sock = await self._connect_sock(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\asyncio\base_events.py", line 1006, in _connect_sock
await self.sock_connect(sock, address)
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\asyncio\selector_events.py", line 651, in sock_connect
return await fut
^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\asyncio\selector_events.py", line 691, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 10060] Connect call failed ('152.61.141.32', 443)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "\?\C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Scripts\outage-map-script.py", line 33, in
sys.exit(load_entry_point('Outage-Map-API', 'console_scripts', 'outage-map')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\users\co986387\documents\outageprobabilityapi\outage_map\import_dss_cli.py", line 87, in import_dss
covers = getLandCover(coords)
^^^^^^^^^^^^^^^^^^^^
File "c:\users\co986387\documents\outageprobabilityapi\outage_map\util\NetworkFunctions.py", line 134, in getLandCover
land_usage_land_cover = gh.nlcd_bycoords(coords, years={"cover": [year]})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\pygeohydro\nlcd.py", line 266, in nlcd_bycoords
ds_list = [nlcd_wms.get_map(g, 30) for g in geoms]
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\pygeohydro\nlcd.py", line 147, in get_map
r_dict = self.wms.getmap_bybox(bbox, resolution, self.crs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\pygeoogc\pygeoogc.py", line 471, in getmap_bybox
rbinary = ar.retrieve_binary(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\async_retriever\async_retriever.py", line 638, in retrieve_binary
return retrieve(
^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\async_retriever\async_retriever.py", line 433, in retrieve
resp = [r for _, r in sorted(tlz.concat(results))]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\async_retriever\async_retriever.py", line 431, in
results = (loop.run_until_complete(session(url_kwds=c)) for c in chunked_reqs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\asyncio\base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\async_retriever\async_retriever.py", line 236, in async_session_with_cache
return await asyncio.gather(*tasks) # pyright: ignore[reportReturnType]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\async_retriever_utils.py", line 81, in retriever
async with session(url, **s_kwds) as response:
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp\client.py", line 1197, in aenter
self._resp = await self._coro
^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp_client_cache\session.py", line 80, in _request
new_response = await super()._request(method, str_or_url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp\client.py", line 581, in _request
conn = await self._connector.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp\connector.py", line 544, in connect
proto = await self._create_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp\connector.py", line 944, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp\connector.py", line 1257, in _create_direct_connection
raise last_exc
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp\connector.py", line 1226, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\co986387\AppData\Local\anaconda3\envs\outage-map-api\Lib\site-packages\aiohttp\connector.py", line 1033, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.mrlc.gov:443 ssl:default [Connect call failed ('152.61.141.32', 443)]
PS C:\Users\co986387\Documents\outageProbabilityAPI>

Is there a limit to the number of requests I can make?

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

pygeohydro==0.16.5

[cheginit]

This is a common issue when sending large web requests to web services in a short period of time, i.e., hammering the web service. HyRiver doesn't impose any rate limit, but on the server side, there might be a rate limiting mechanism in place. For large requests, I recommend batching your requests (say 100 points per batch), storing the results of each batch on disk, then concatenating the results, if needed. You can set a sleep time between batches, to avoid hammering the service.