Getting Started (Extended)
This guide provides a comprehensive introduction to setting up and using DemandCast. It covers installation, configuration, basic usage, and answers to frequently asked questions.
1. Installation and Setup
To install DemandCast, follow these steps:
1.1 Clone the repository
git clone https://github.com/open-energy-transition/demandcast.git
cd demandcast
1.2 Set up your environment
This project uses uv as a package manager to install the required dependencies and create an environment stored in .venv.
uv can be used within the provided Dockerfile or installed standalone (see installing uv).
The demandcast folder contains a pyproject.toml file that defines all the dependencies for the project.
To set up the environment, run:
cd demandcast
uv sync
To use the optional LSTM model, add its dependencies with uv sync --extra lstm.
Alternatively, you may use a package manager of your choice (e.g., conda) to install the dependencies listed in the pyproject.toml. If you choose this approach, please adjust the commands below to align with the conventions of your selected package manager.
The scripts run from the demandcast folder, with uv run <script>.py, and Jupyter notebooks (details) with:
uv run --with jupyter jupyter lab --allow-root
1.3 Configure environment variables
Some modules require API keys to access data from external services. These keys should be stored in a .env file in the demandcast/ directory. The .env file should not be included in the repository and should contain the following environment variables:
CDS_API_KEY=<your_key> # For data retrieval from Copernicus CDS
ENTSOE_API_KEY=<your_key> # For data retrieval from ENTSO-E
EIA_API_KEY=<your_key> # For data retrieval from EIA
EMBER_API_KEY=<your_key> # For the annual electricity demand per capita of Ember, with a free key
EPIAS_USERNAME=<your_username> # For data retrieval from EPIAS, with a free account
EPIAS_PASSWORD=<your_password> # For data retrieval from EPIAS
ISONE_USERNAME=<your_username> # For data retrieval from ISO New England, with a free ISO Express account
ISONE_PASSWORD=<your_password> # For data retrieval from ISO New England
ZENODO_API_KEY=<your_key> # For releases on Zenodo (publish.py)
SANDBOX_ZENODO_API_KEY=<your_key> # For test releases on the Zenodo Sandbox
Replace <your_key>, <your_username> and <your_password> with your actual API keys and account. You can obtain them by registering on the respective service websites.
Everything that the scripts write, from the downloaded data to the trained models, the results and the logs, goes in demandcast/data/. To keep it elsewhere, such as on a larger disk, set the environment variable DEMANDCAST_DATA_FOLDER to another folder.
2. Retrieving Data
Note: The retrieved data is not redistributed, since the licences of many data sources do not allow it, so this section retrieves it from the sources. If you have access to a bucket with a copy of the data folder, sync.py downloads it instead (see sharing and publishing).
In this example, we will retrieve data for four European countries (France, Germany, Italy, and Spain), train a model on data from 2020-2023, and then use it to forecast electricity demand for Spain in 2024. The following steps will guide you through the data retrieval process.
Each script reads its settings from config/<script>_config.yaml, or from the file given with --config, and checks them before it starts: a misspelt option, a scenario or climate model that the data does not have, or a missing setting stops it with an error that lists the valid values.
2.1 Create a Country List File
First, create a YAML file listing all countries. Create a file named demandcast/config/example_countries.yaml:
entities:
- country_name: France
country_code: FRA
- country_name: Germany
country_code: DEU
- country_name: Italy
country_code: ITA
- country_name: Spain
country_code: ESP
2.2 Retrieve Electricity Demand
Retrieve historical electricity demand data for all four countries. Edit demandcast/config/retrieve_config.yaml:
variable: electricity_demand
electricity_data_source: entsoe
file: config/example_countries.yaml
Then run:
cd demandcast
uv run retrieve.py
2.3 Retrieve Annual Electricity Demand per Capita
Retrieve annual electricity demand per capita for all four countries from 2020 to 2024. Edit demandcast/config/retrieve_config.yaml:
variable: annual_electricity_demand_per_capita
file: config/example_countries.yaml
start_year: 2020
end_year: 2024
Then run:
cd demandcast
uv run retrieve.py
2.4 Retrieve GDP PPP per Capita
Retrieve GDP PPP per capita for all four countries from 2020 to 2024. Edit demandcast/config/retrieve_config.yaml:
variable: gdp_ppp_per_capita
file: config/example_countries.yaml
start_year: 2020
end_year: 2024
Then run:
cd demandcast
uv run retrieve.py
2.5 Retrieve Population
Retrieve population data for all four countries from 2020 to 2024. Edit demandcast/config/retrieve_config.yaml:
variable: population
file: config/example_countries.yaml
start_year: 2020
end_year: 2024
Then run:
cd demandcast
uv run retrieve.py
2.6 Retrieve Temperature
Retrieve temperature data for all four countries from 2020 to 2024. Edit demandcast/config/retrieve_config.yaml:
variable: temperature
file: config/example_countries.yaml
start_year: 2020
end_year: 2024
Then run:
cd demandcast
uv run retrieve.py
3. Assembling Training Data
Note: You can skip this section if you want to use the trained model published on Zenodo, with its forecasts.
After retrieving the necessary data, assemble it into a unified dataset for model training. This step merges electricity demand, temperature, GDP PPP per capita, annual electricity demand per capita, and population data for all four countries from 2020 to 2023. The electricity demand is the combined one, which joins all the data sources of each country (see Data Retrieval).
Edit demandcast/config/assemble_config.yaml:
target_use: training
file: config/example_countries.yaml
start_year: 2020
end_year: 2023
Then run:
cd demandcast
uv run assemble.py
This will create a file in data/assembled/ with the training data.
4. Training the Model
Our approach involves using socioeconomic and weather parameters to predict hourly electricity demand. The model learns patterns from all four countries (France, Germany, Italy, and Spain): with reserve_testing_set: true, the latest year of each country, 2023, is kept for testing, and the model is trained on 2020-2022. With use_validation_set: true, 2022 would also be kept, for validation, and the model trained on 2020-2021. The model will later be used to forecast Spain's demand for 2024.
Edit demandcast/config/train_config.yaml to specify the assembled data file path:
reserve_testing_set: true
use_validation_set: false
data_path: data/assembled/assembled_data_for_training_YYYYMMDD_HHMMSS.parquet # Or leave empty for the latest data assembled for training
Then run:
cd demandcast
uv run train.py
The trained model will be saved in data/models/xgboost/ with a timestamp in the filename, with its manifest.
Before using the model, validate it on the year kept for testing, 2023. Edit demandcast/config/validate_config.yaml:
used_validation_set: false # as use_validation_set in the training
model_path: # empty for the latest model
data_path: # empty for the data of the training of the model
Then run:
cd demandcast
uv run validate.py
The MAPE of each country in the training and testing sets is saved in data/results/validation/ (see ML).
5. Assembling Forecasting Data
To forecast electricity demand for Spain in 2024, we need to assemble the input features (temperature, GDP PPP per capita, annual electricity demand per capita, and population) for Spain for that year. Create a file named demandcast/config/spain_only.yaml:
entities:
- country_name: Spain
country_code: ESP
Edit demandcast/config/assemble_config.yaml:
target_use: forecasting
file: config/spain_only.yaml
start_year: 2024
end_year: 2024
Then run:
cd demandcast
uv run assemble.py
This will create a file in data/assembled/ with the forecasting data for Spain.
6. Forecasting
Now use the trained model to forecast electricity demand for Spain. The forecasting script requires both the trained model and the assembled forecasting data. Because electricity demand is predicted in normalized form, the input data must include population to denormalize the predictions back to absolute values (MW).
Edit demandcast/config/forecast_config.yaml:
model_path: data/models/xgboost/xgboost_model_YYYYMMDD_HHMMSS.json # Use the actual filename
data_path: data/assembled/assembled_data_for_forecasting_YYYYMMDD_HHMMSS.parquet # Use the actual filename
Then run:
cd demandcast
uv run forecast.py
The forecasts will be saved in data/results/forecasts/ and will contain hourly electricity demand predictions for Spain for 2024.
7. Example Results
The figure below illustrates the complete workflow for Spain. The top panel shows the retrieved historical electricity demand data for 2023 and 2024. The bottom panels show the input features used for forecasting: annual electricity demand per capita, GDP PPP per capita, and temperature data for the same years.
In this example workflow: - The model was trained on data from all four countries (France, Germany, Italy, and Spain) for 2020-2022, with 2023 kept for testing, - The model uses the 2024 input features (temperature, GDP PPP per capita, annual electricity demand per capita, and population) to forecast Spain's hourly electricity demand for 2024, - This demonstrates how DemandCast can forecast future electricity demand for countries using a model trained on historical data from multiple regions.
