Remove aws sdk and set up minio in-place

blaginin · Jan 9, 2025 · 7d4050d · 7d4050d
1 parent 17e8486
commit 7d4050d
Show file tree

Hide file tree

Showing 8 changed files with 33 additions and 394 deletions.
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
@@ -211,16 +211,25 @@ jobs:
           rustup default stable
       - name: Setup Minio - S3-compatible storage
         working-directory: datafusion-cli
-        run:
-          echo "MINIO_CONTAINER=$(docker run -d -p 9000:9000 -e MINIO_ROOT_USER=TEST-DataFusionLogin -e MINIO_ROOT_PASSWORD=TEST-DataFusionPassword quay.io/minio/minio server /data)" >> $GITHUB_ENV
+        run: |
+          docker run -d --name minio-container \
+            -p 9000:9000 \
+            -e MINIO_ROOT_USER=TEST-DataFusionLogin -e MINIO_ROOT_PASSWORD=TEST-DataFusionPassword \
+            -v $(pwd)/../datafusion/core/tests/data:/source quay.io/minio/minio \
+            server /data
+          docker exec minio-container /bin/sh -c "\
+            mc ready local
+            mc alias set localminio http://localhost:9000 TEST-DataFusionLogin TEST-DataFusionPassword && \
+            mc mb localminio/data && \
+            mc cp -r /source/* localminio/data"
       - name: Run tests (excluding doctests, but with integration tests)
         working-directory: datafusion-cli
         run: cargo test --profile ci --lib --tests --bins --all-features
       - name: Verify Working Directory Clean
         run: git diff --exit-code
       - name: Minio Output
         if: ${{ !cancelled() }}
-        run: docker logs $MINIO_CONTAINER
+        run: docker logs minio-container
 
   linux-test-example:
     name: cargo examples (amd64)

diff --git a/datafusion-cli/CONTRIBUTING.md b/datafusion-cli/CONTRIBUTING.md
@@ -47,15 +47,26 @@ then provide the necessary configuration for that object store.
 
 To test the S3 integration against [Minio](https://github.com/minio/minio)
 
-First start up a container with Minio
+First start up a container with Minio and load test files.
 
-```
-$ docker run -d -p 9000:9000 -e MINIO_ROOT_USER=TEST-DataFusionLogin -e MINIO_ROOT_PASSWORD=TEST-DataFusionPassword quay.io/minio/minio server /data
+```shell
+docker run -d \
+  --name datafusion-test-minio \
+  -p 9000:9000 \
+  -e MINIO_ROOT_USER=TEST-DataFusionLogin \
+  -e MINIO_ROOT_PASSWORD=TEST-DataFusionPassword \
+  -v $(pwd)/../datafusion/core/tests/data:/source \
+  quay.io/minio/minio server /data
+
+docker exec datafusion-test-minio /bin/sh -c "\
+  mc alias set localminio http://localhost:9000 TEST-DataFusionLogin TEST-DataFusionPassword && \
+  mc mb localminio/data && \
+  mc cp -r /source/* localminio/data"
 ```
 
 Setup environment
 
-```
+```shell
 export TEST_STORAGE_INTEGRATION=1
 export AWS_ACCESS_KEY_ID=TEST-DataFusionLogin
 export AWS_SECRET_ACCESS_KEY=TEST-DataFusionPassword
@@ -67,6 +78,6 @@ Note that `AWS_ENDPOINT` is set without slash at the end.
 
 Run tests
 
-```
-$ cargo test
+```shell
+cargo test
 ```