Support object storage path style requests

[arthurzenika]

Describe the current behavior

We're trying to deploy peertube on kubernetes with the new s3 native support (yeah! congrats on that).

As described in coopgo/peertube-k8s#2 (we're trying to use the excellent work of @coopgo ) we need to configure the use_path_request_style for the S3 connexion.

Steps to reproduce:

1. deploy with https://github.com/coopgo/peertube-k8s/ or with configuration for a minio s3 object store
2. upload video
3. wait for ffmpeg to trancode
4. see errors like :

Job: 1 Type: move-to-object-storage Processed on Sep 24, 2021, 4:49:50 PM Finished on Sep 24, 2021, 4:49:50 PM
--
{   "videoUUID": "a4a94b48-8a72-4abf-b892-cf9f94e58b85",   "isNewVideo": true }
Error: getaddrinfo ENOTFOUND peertube.minio.example.org     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26)

Describe the expected behavior

As @adelcasse described :

And S3Client is declared

PeerTube/server/lib/object-storage/shared/client.ts

Line 21 in 1c93ad0

s3Client = new S3Client({

without the option to use "forcePathStyle" https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/s3clientconfig.html#forcepathstyle

Additional information

• PeerTube instance: n/a

  • URL: n/a
  • version: v3.4.0-rc.1-buster
  • NodeJS version: n/a
  • Ffmpeg version: n/a

• Browser name, version and platforms on which you could reproduce the bug: n/a

• Link to browser console log if relevant: n/a

• Link to server log if relevant (journalctl or /var/www/peertube/storage/logs/): n/a

[Chocobozzz]

Hello,

Unfortunately PeerTube does not support path request style:

• It's deprecated since 2019 on aws
• It would require additional work in peertube code to handle these new URLs format
• It would multiply our test suite by 2 (and add potential edge cases and bugs)

Minio does not support virtual host style requests?

[dhk2]

It appears to.

By default, MinIO supports path-style requests that are of the format http://mydomain.com/bucket/object. MINIO_DOMAIN environment variable is used to enable virtual-host-style requests. If the request Host header matches with (.+).mydomain.com then the matched pattern $1 is used as bucket and the path is used as object.

https://docs.min.io/docs/minio-server-configuration-guide.html

[faust64]

Rook is also affected: rook/rook#4780
And even though I'm not using Rook, I do use Rados Gateways - and MinIO, Most of the time, I would create a single Service object. All my applications would query their respective buckets, from that single FQDN or IP.
Some of the applications I use, that do support this configuration : NextCloud, Nexus, EtherPad, CodiMD, ...

Although AWS may have dropped path-based, the S3 API implementations goes beyond Amazon itself.
Some of those APIs did not catch up - and arguably, may not have to: being able to create a bucket without having to register DNS records is for sure easier..

[gsugambit]

Is this working for you these days @arthurzenika ? I'm trying to use my minio server config to push the images to google cloud storage. I don't see any errors in the peertube logs but it's definitely not pushing the images to GCS either.

[arthurzenika]

@gsugambit sorry I can't say if this has been fixed on the setups I used to have access to. As far as I can see from the history above, this is still not possible. Good luck for your setup.

[ddzobov]

This is actual for me too. Please add option to settings with this property.

We are using MiniO and we can not use virtual host style requests due to internal security reasons.

[b0rmann]

This is actual for me too. For MiniO.

[vincib]

I am using minio with peertube, and even if the documentation of minio regarding FQDN-style s3 is lacking ...

It's working perfectly ;)

as stated above : use MINIO_DOMAIN in the configuration file to tell which FQDN root will be used. Remember to get a wildcard certificate (free with letsencrypt) if you want to support multiple buckets, since each bucket will use its own subdomain.

(as a result, I don't need this feature)

[yunkuangao]

I use contabo object storage and I hope to add this functionality.

[ghost]

Same issue here.. Can't use peer tube with my S3 provider because they support only path style..

[tio-trom]

Same here...I was happy to move to an object storage only to come across this issue :D . Contabo uses S3 and yet it does not work. I hope the only path style support can be added to Peertube.

[padthaitofuhot]

Please enable path-style buckets in PeerTube? It would put it at feature parity with every other Fediverse app out there. Virtual-host buckets are significantly more complex to orchestrate than path-style buckets and places high burden on self-hosters. In my self-hosted setup, it is virtual-host buckets xor path-style buckets, and since the other seven services I have running are all using path-style buckets I'm unable to go live with the PeerTube instance I spent the last couple days deploying.

[32bitx64bit]

I don't get why this isn't a thing...
Vultr from what I can gather ONLY supports Path-style requests... At least I can't get a Vultr bucket to work.

So like... Can we please more to a actual standard... Instead of something only AWS does....