Cannot instantiate S3Store without credentials

[astrojuanlu]

Here is a public S3 object that doesn't need credentials:

❯ aws s3 ls --no-sign-request s3://ai2-public-datasets/ropes/
2021-02-04 20:48:17    3395072 ropes-train-dev-v1.0.tar.gz

(source)

And yet, I'm not able to make obstore work. My last attempt, with obstore==0.3.0b8:

In [4]: import boto3

In [5]: from obstore.store import S3Store

In [6]: session = boto3.Session()

In [7]: store = S3Store.from_session(session, "ai2-public-datasets")
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], line 1
----> 1 store = S3Store.from_session(session, "ai2-public-datasets")

AttributeError: 'NoneType' object has no attribute 'get_frozen_credentials'

I think the reason is this:

In [9]: session.get_credentials() is None
Out[9]: True

I then tried some dummy credentials, but I'm still getting errors:

In [24]: session = boto3.Session(aws_access_key_id="abc", aws_secret_access_key="abc", aws_session_token="abc", region_name="us-east-1")

In [25]: store = S3Store.from_session(session, "ai2-public-datasets")

In [26]: obs.get(store, "ropes/ropes-train-dev-1.0.tar.gz")
---------------------------------------------------------------------------
GenericError                              Traceback (most recent call last)
Cell In[26], line 1
----> 1 obs.get(store, "ropes/ropes-train-dev-1.0.tar.gz")

GenericError: Generic S3 error: Received redirect without LOCATION, this normally indicates an incorrectly configured region

[kylebarron]

See here #88 (comment)

[astrojuanlu]

Thanks, that did the trick, partially. I still needed fake credentials:

In [23]: session = boto3.Session(aws_access_key_id="abc", aws_secret_access_key="abc", aws_session_token="abc")

In [24]: store = S3Store.from_session(session, "ai2-public-datasets", config={"AWS_SKIP_SIGNATURE": "True", "AWS_REGION": "us-west-2"})  # Works

In [25]: obs.get(store, "ropes/ropes-train-dev-v1.0.tar.gz")
Out[25]: <GetResult at 0x1129b9ed0>

In [26]: session = boto3.Session()

In [27]: store = S3Store.from_session(session, "ai2-public-datasets", config={"AWS_SKIP_SIGNATURE": "True", "AWS_REGION": "us-west-2"})
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[27], line 1
----> 1 store = S3Store.from_session(session, "ai2-public-datasets", config={"AWS_SKIP_SIGNATURE": "True", "AWS_REGION": "us-west-2"})

AttributeError: 'NoneType' object has no attribute 'get_frozen_credentials'

[astrojuanlu]

Feel free to close as duplicate of #88 anyway 🙏🏼

[kylebarron]

We should probably add an __init__ constructor, but you can use S3Store.from_url without needing to touch boto3 at all

[astrojuanlu]

I confirm:

In [1]: import obstore as obs

In [2]: from obstore.store import S3Store

In [3]: store = S3Store.from_url("s3://ai2-public-datasets/ropes/", config={"AWS_SKIP_SIGNATURE": "True", "AWS_REGION": "us-west-2"})

In [4]: obs.get(store, "ropes/ropes-train-dev-v1.0.tar.gz")
Out[4]: <GetResult at 0x7f897a89ee90>

[kylebarron]

Closing in favor of #88 and #104