Ensemble Price Forecasting with Actuals

Overview

This tutorial demonstrates how to build an ensemble forecasting system with pyconvexity. We'll create a single-bus electricity market model with multiple forecast scenarios, then compare the outputs against stored "actual" (measured) values.

What you'll learn

• Creating ensemble scenarios for probabilistic forecasting
• Storing "actual" (measured/true) values separately from forecasts
• Working with wind capacity factor uncertainty
• Comparing forecast outputs against actual values
• Solving multiple scenarios programmatically

Key concept

In real-world forecasting, you often have:

1. Ensemble forecasts - Multiple scenarios with different assumptions (e.g., wind forecasts)
2. Actual values - The true/measured outcomes for comparison

pyconvexity's scenario system lets you store both, enabling forecast verification and accuracy analysis.

Prerequisites

pip install pyconvexity

Download Example Files

📥 Python Script (.py) · Database (.db) · Excel (.xlsx)

The Forecasting Problem

We model a simple UK electricity system over one week:

• 1 bus (Central England)
• 3 gas generators (different costs: £35, £50, £80/MWh)
• 1 wind farm (zero marginal cost, variable output)
• 1 load (daily varying demand)

The uncertainty comes from wind forecasts. We create 5 ensemble members with different wind capacity factor predictions, then store the "actual" wind and resulting prices for comparison.

Step 1: Set up the model

import pyconvexity as px
import numpy as np
from datetime import datetime, timedelta

# Configuration
PERIODS_PER_DAY = 24
NUM_DAYS = 7
TOTAL_PERIODS = PERIODS_PER_DAY * NUM_DAYS  # 168 hours
NUM_ENSEMBLE_MEMBERS = 5

# Generator specifications
GENERATORS = {
    "Gas Peaker": {"p_nom": 200, "marginal_cost": 80},
    "Gas CCGT": {"p_nom": 400, "marginal_cost": 50},
    "Gas Base": {"p_nom": 300, "marginal_cost": 35},
    "Wind Farm": {"p_nom": 500, "marginal_cost": 0},
}

# Create database
db_path = "forecast_truth.db"
px.create_database_with_schema(db_path)

Step 2: Create network and components

with px.database_context(db_path) as conn:
    # Create network metadata
    start_time = datetime(2024, 1, 1, 0, 0, 0)
    end_time = start_time + timedelta(hours=TOTAL_PERIODS - 1)
    
    network_req = px.CreateNetworkRequest(
        name="Forecast vs Truth Example",
        description="One-week ensemble price forecast with actual values",
        start_time=start_time.strftime("%Y-%m-%d %H:%M:%S"),
        end_time=end_time.strftime("%Y-%m-%d %H:%M:%S"),
        time_resolution="PT1H",
    )
    px.create_network(conn, network_req)
    
    # Create carriers
    carriers = {}
    for carrier_name in ["AC", "Gas", "Wind"]:
        carrier_id = px.create_carrier(conn, name=carrier_name)
        carriers[carrier_name] = carrier_id
    
    # Create bus
    bus_id = px.create_component(
        conn,
        component_type="BUS",
        name="Main Bus",
        carrier_id=carriers["AC"],
        latitude=52.5,
        longitude=-1.5,
    )
    
    # Create generators
    generator_ids = {}
    for gen_name, gen_info in GENERATORS.items():
        carrier = carriers["Wind"] if "Wind" in gen_name else carriers["Gas"]
        gen_id = px.create_component(
            conn,
            component_type="GENERATOR",
            name=gen_name,
            bus_id=bus_id,
            carrier_id=carrier,
        )
        px.set_static_attribute(conn, gen_id, "p_nom", px.StaticValue(float(gen_info["p_nom"])))
        px.set_static_attribute(conn, gen_id, "marginal_cost", px.StaticValue(float(gen_info["marginal_cost"])))
        generator_ids[gen_name] = gen_id
    
    # Create load with daily varying profile
    load_id = px.create_component(
        conn,
        component_type="LOAD",
        name="System Load",
        bus_id=bus_id,
        carrier_id=carriers["AC"],
    )
    
    # Generate realistic daily load profile
    load_profile = generate_daily_load_profile(NUM_DAYS, base_load=350)
    px.set_timeseries_attribute(conn, load_id, "p_set", load_profile)

Step 3: Create wind profile generator

def generate_wind_profile(num_periods: int, mean_capacity_factor: float = 0.35) -> list:
    """Generate realistic wind capacity factor profile with temporal correlation."""
    profile = []
    current_cf = mean_capacity_factor
    
    for _ in range(num_periods):
        # Mean reversion factor
        reversion = 0.1 * (mean_capacity_factor - current_cf)
        # Random walk step
        step = np.random.normal(0, 0.08)
        # Update and clip
        current_cf = np.clip(current_cf + reversion + step, 0.0, 1.0)
        profile.append(current_cf)
    
    return profile

This creates temporally correlated wind profiles that look realistic - wind doesn't jump randomly hour-to-hour but tends to persist.

Step 4: Create ensemble scenarios

    # Set base wind profile
    base_wind_cf = generate_wind_profile(TOTAL_PERIODS, mean_capacity_factor=0.35)
    px.set_timeseries_attribute(conn, generator_ids["Wind Farm"], "p_max_pu", base_wind_cf)
    
    # Create ensemble scenarios with different wind forecasts
    for i in range(1, NUM_ENSEMBLE_MEMBERS + 1):
        scenario_name = f"ensemble/{i:02d}"
        scenario_id = px.create_scenario(conn, name=scenario_name, description=f"Ensemble member {i}")
        
        # Generate perturbed wind profile for this ensemble member
        # Each member has a slightly different mean capacity factor
        ensemble_wind_cf = generate_wind_profile(
            TOTAL_PERIODS, 
            mean_capacity_factor=0.30 + 0.1 * np.random.random()
        )
        
        # Set the wind p_max_pu for this scenario
        px.set_timeseries_attribute(
            conn, generator_ids["Wind Farm"], "p_max_pu", ensemble_wind_cf, scenario_id=scenario_id
        )
        print(f"Created {scenario_name}: wind CF mean={np.mean(ensemble_wind_cf):.2f}")

Key insight: Each ensemble member overrides only the wind capacity factor. All other model data (generators, load, network) is shared from the base scenario.

Step 5: Store actual values

    # Generate "actual" wind and prices
    actual_wind_cf = generate_wind_profile(TOTAL_PERIODS, mean_capacity_factor=0.38)
    actual_prices = generate_marginal_prices(TOTAL_PERIODS, GENERATORS, load_profile, actual_wind_cf)
    
    # Store actual values using the special "Actual" scenario
    px.set_actual_timeseries_value(conn, bus_id, "marginal_price", actual_prices)
    px.set_actual_timeseries_value(conn, generator_ids["Wind Farm"], "p_max_pu", actual_wind_cf)
    
    print(f"Actual price range: ${min(actual_prices):.1f} - ${max(actual_prices):.1f}/MWh")
    print(f"Actual wind CF mean: {np.mean(actual_wind_cf):.2f}")
    
    conn.commit()

The "Actual" scenario is a special system scenario in pyconvexity designed for storing measured/true values. It's separate from forecast scenarios and can be used for verification.

Step 6: Solve all scenarios

# Solve base scenario
print("Solving base scenario...")
result = px.solve_network(db_path=db_path, solver_name="highs")
print(f"Base: success={result.get('success')}, objective={result.get('objective_value'):.2f}")

# Solve each ensemble scenario
with px.database_context(db_path) as conn:
    scenarios = px.list_scenarios(conn)

for scenario in scenarios:
    if scenario.name.startswith("ensemble/"):
        result = px.solve_network(
            db_path=db_path,
            solver_name="highs",
            scenario_id=scenario.id,
        )
        print(f"{scenario.name}: success={result.get('success')}, objective={result.get('objective_value'):.2f}")

Step 7: Export to Excel

from pyconvexity.io.excel_exporter import ExcelModelExporter

xlsx_path = db_path.replace('.db', '.xlsx')
exporter = ExcelModelExporter()
exporter.export_model_to_excel(db_path=db_path, output_path=xlsx_path)
print(f"Exported to: {xlsx_path}")

Understanding the Results

After solving, each ensemble member produces:

• Generator dispatch (p) - How much each generator produces each hour
• Bus marginal price - The system price at each hour

You can compare these against the stored "actual" values to:

• Measure forecast accuracy (RMSE, MAE, etc.)
• Analyze forecast bias
• Understand how wind uncertainty propagates to price uncertainty

Expected patterns:

• When wind is high, prices are low (wind displaces expensive gas)
• When wind is low, gas peakers set the price (~£80/MWh)
• Ensemble spread shows the range of possible outcomes