Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

records: Add examples about library and application use #764

Merged
merged 2 commits into from
Dec 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,11 @@ updates:
schedule:
interval: "daily"

- directory: "/application/records"
package-ecosystem: "pip"
schedule:
interval: "daily"

# Frameworks.

- directory: "/framework/dbt/basic"
Expand All @@ -112,6 +117,11 @@ updates:
schedule:
interval: "daily"

- directory: "/framework/records"
package-ecosystem: "pip"
schedule:
interval: "daily"

- directory: "/framework/streamlit"
package-ecosystem: "pip"
schedule:
Expand Down
74 changes: 74 additions & 0 deletions .github/workflows/application-records.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: records (application)

on:
pull_request:
branches: ~
paths:
- '.github/workflows/application-records.yml'
- 'application/records/**'
- '/requirements.txt'
push:
branches: [ main ]
paths:
- '.github/workflows/application-records.yml'
- 'application/records/**'
- '/requirements.txt'

# Allow job to be triggered manually.
workflow_dispatch:

# Run job each night after CrateDB nightly has been published.
schedule:
- cron: '0 3 * * *'

# Cancel in-progress jobs when pushing to the same branch.
concurrency:
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.ref }}

jobs:
test:
name: "
Python: ${{ matrix.python-version }}
CrateDB: ${{ matrix.cratedb-version }}
on ${{ matrix.os }}"
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
python-version: [ '3.9', '3.13' ]
cratedb-version: [ 'nightly' ]

services:
cratedb:
image: crate/crate:${{ matrix.cratedb-version }}
ports:
- 4200:4200
- 5432:5432
env:
CRATE_HEAP_SIZE: 4g

steps:

- name: Acquire sources
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
architecture: x64
cache: 'pip'
cache-dependency-path: |
requirements.txt
application/records/requirements.txt
application/records/requirements-dev.txt

- name: Install utilities
run: |
pip install -r requirements.txt

- name: Validate application/records
run: |
ngr test --accept-no-venv application/records
74 changes: 74 additions & 0 deletions .github/workflows/framework-records.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
name: records (framework)

on:
pull_request:
branches: ~
paths:
- '.github/workflows/framework-records.yml'
- 'framework/records/**'
- '/requirements.txt'
push:
branches: [ main ]
paths:
- '.github/workflows/framework-records.yml'
- 'framework/records/**'
- '/requirements.txt'

# Allow job to be triggered manually.
workflow_dispatch:

# Run job each night after CrateDB nightly has been published.
schedule:
- cron: '0 3 * * *'

# Cancel in-progress jobs when pushing to the same branch.
concurrency:
cancel-in-progress: true
group: ${{ github.workflow }}-${{ github.ref }}

jobs:
test:
name: "
Python: ${{ matrix.python-version }}
CrateDB: ${{ matrix.cratedb-version }}
on ${{ matrix.os }}"
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
python-version: [ '3.9', '3.13' ]
cratedb-version: [ 'nightly' ]

services:
cratedb:
image: crate/crate:${{ matrix.cratedb-version }}
ports:
- 4200:4200
- 5432:5432
env:
CRATE_HEAP_SIZE: 4g

steps:

- name: Acquire sources
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
architecture: x64
cache: 'pip'
cache-dependency-path: |
requirements.txt
framework/records/requirements.txt
framework/records/requirements-dev.txt

- name: Install utilities
run: |
pip install -r requirements.txt

- name: Validate framework/records
run: |
ngr test --accept-no-venv framework/records
57 changes: 57 additions & 0 deletions application/records/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Verify the `records` program with CrateDB

Records: SQL for Humans™

## About

This folder includes software integration tests for verifying
that the [Records] Python program works well together with [CrateDB].

Records is a very simple, but powerful, library for making raw SQL
queries to most relational databases. It uses [SQLAlchemy].

Records is intended for report-style exports of database queries, and
has not yet been optimized for extremely large data dumps.

## What's Inside

- `example.sh`: A few examples that read CrateDB's `sys.summits` table
using the `records` program. A single example that inserts data into
a column using CrateDB's `OBJECT` column.

## Install

Set up sandbox and install packages.
```bash
pip install uv
uv venv .venv
source .venv/bin/activate
uv pip install -r requirements.txt
```

## Synopsis
Install packages.
```shell
pip install --upgrade records sqlalchemy-cratedb
```
Define database connection URL, suitable for CrateDB on localhost.
For CrateDB Cloud, use `crate://<username>:<password>@<host>`.
```shell
export DATABASE_URL="crate://"
```
Invoke query.
```shell
records "SELECT * FROM sys.summits WHERE region ILIKE :region" region="ortler%"
```

## Tests

Run integration tests.
```bash
sh test.sh
```


[CrateDB]: https://cratedb.com/database
[Records]: https://pypi.org/project/records/
[SQLAlchemy]: https://www.sqlalchemy.org/
34 changes: 34 additions & 0 deletions application/records/example.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env sh

# Using `records` with CrateDB: Basic usage.
#
# pip install --upgrade records sqlalchemy-cratedb
#
# A few basic operations using the `records` program with CrateDB.
#
# - https://pypi.org/project/records/

# Define database connection URL, suitable for CrateDB on localhost.
# For CrateDB Cloud, use `crate://<username>:<password>@<host>`.
export DATABASE_URL="crate://"

# Basic query, tabular output.
records "SELECT * FROM sys.summits ORDER BY height DESC LIMIT 3"

# Query with parameters.
records "SELECT * FROM sys.summits WHERE region ILIKE :region" region="ortler%"

# Export data.
# Supported formats: csv tsv json yaml html xls xlsx dbf latex ods
records "SELECT * FROM sys.summits ORDER BY height DESC LIMIT 3" csv
records "SELECT * FROM sys.summits ORDER BY height DESC LIMIT 3" json
records "SELECT * FROM sys.summits LIMIT 42" html > "${TMPDIR}/sys_summits.html"
records "SELECT * FROM sys.summits LIMIT 42" ods > "${TMPDIR}/sys_summits.ods"
records "SELECT * FROM sys.summits LIMIT 42" xlsx > "${TMPDIR}/sys_summits.xlsx"

# Insert data.
records "DROP TABLE IF EXISTS testdrive.example"
records "CREATE TABLE testdrive.example (data OBJECT(DYNAMIC))"
records "INSERT INTO testdrive.example (data) VALUES (:data)" data='{"temperature": 42.42, "humidity": 84.84}'
records "REFRESH TABLE testdrive.example"
records "SELECT * FROM testdrive.example"
23 changes: 23 additions & 0 deletions application/records/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[tool.pytest.ini_options]
minversion = "2.0"
addopts = """
-rfEX -p pytester --strict-markers --verbosity=3
--capture=no
"""
log_level = "DEBUG"
log_cli_level = "DEBUG"
testpaths = ["*.py"]
xfail_strict = true
markers = [
]


[tool.coverage.run]
branch = false
omit = [
"test*",
]

[tool.coverage.report]
fail_under = 0
show_missing = true
1 change: 1 addition & 0 deletions application/records/requirements-test.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pytest<9
3 changes: 3 additions & 0 deletions application/records/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
records<0.7
sqlalchemy-cratedb<0.41
tablib[ods]
11 changes: 11 additions & 0 deletions application/records/test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import shlex
import subprocess
import pytest


def run(command: str):
subprocess.check_call(shlex.split(command))


def test_example():
run("sh example.sh")
62 changes: 62 additions & 0 deletions framework/records/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Verify the `records` library with CrateDB

Records: SQL for Humans™

## About

This folder includes software integration tests for verifying
that the [Records] Python library works well together with [CrateDB].

Records is a very simple, but powerful, library for making raw SQL
queries to most relational databases. It uses [SQLAlchemy].

Records is intended for report-style exports of database queries, and
has not yet been optimized for extremely large data dumps.

## What's Inside

- `example_basic.py`: A few examples that read CrateDB's `sys.summits` table.
An example inquiring existing tables.

- `example_types.py`: An example that exercises all data types supported by
CrateDB.

## Install

Set up sandbox and install packages.
```bash
pip install uv
uv venv .venv
source .venv/bin/activate
uv pip install -r requirements.txt -r requirements-test.txt
```

## Synopsis
```shell
pip install --upgrade records sqlalchemy-cratedb
```
```python
from pprint import pprint
import records

# Define database connection URL, suitable for CrateDB on localhost.
# For CrateDB Cloud, use `crate://<username>:<password>@<host>`.
db = records.Database("crate://")

# Invoke query.
rows = db.query("SELECT * FROM sys.summits ORDER BY height DESC LIMIT 3")
data = rows.all()
pprint(data)
```

## Tests

Run integration tests.
```bash
pytest
```


[CrateDB]: https://cratedb.com/database
[Records]: https://pypi.org/project/records/
[SQLAlchemy]: https://www.sqlalchemy.org/
Loading
Loading