NESO Solar Consumer

This code can be used to download NESO solar forecasts and save them to a PostgreSQL database. It fetches solar generation estimates for embedded solar farms and processes the data for analysis.

Requirements

Docker
Docker Compose

Installation & Running

Clone the repository:

git clone https://github.com/openclimatefix/neso-solar-consumer.git
cd neso-solar-consumer

Copy the example environment file:

cp .env.example .env

Update the neso_solar_consumer/config.py file with your NESO API configuration:

RESOURCE_ID = "db6c038f-98af-4570-ab60-24d71ebd0ae5"
LIMIT = 100
MODEL_TAG = "neso-solar-forecast"

and .example.env with DATABASE_URL.

Start the application:

docker compose up -d

The above command will:

Start a PostgreSQL database container
Build and start the NESO Solar Consumer application
Configure all necessary networking between containers

To stop the application:

docker compose down

To view logs:

docker compose logs -f

Note: The PostgreSQL data is persisted in a Docker volume. To completely reset the database, use:
docker compose down -v

Documentation

The package provides three main functionalities:

Data Fetching: Retrieves solar forecast data from the NESO API
Data Formatting: Processes the data into standardized forecast objects
Data Storage: Saves the formatted forecasts to a PostgreSQL database

Key Components:

fetch_data.py: Handles API data retrieval
format_forecast.py: Converts raw data into forecast objects
save_forecast.py: Manages database operations
app.py: Orchestrates the entire pipeline

Development

Set up the development environment:

pip install ".[dev]"

Run tests:

pytest

Format code:

black .

Run linter:

ruff .

Running the Test Suite

The test suite includes unit tests and integration tests:

# Run all tests
pytest

# Run specific test file
pytest tests/test_fetch_data.py

# Run with coverage
pytest --cov=neso_solar_consumer

FAQ

Q: What format is the data stored in? A: The data is stored in PostgreSQL using SQLAlchemy models, with timestamps in UTC and power values in megawatts.

Q: How often should I run the consumer? A: This depends on your use case and the NESO API update frequency. The consumer can be scheduled using cron jobs or other scheduling tools.