From Energy Data to Carbon-Aware Forecasts:Building EcoLens

A forecasting model cannot work directly with a messy stream of market observations.

Before it can predict tomorrow's electricity demand, the platform must establish what happened today — and ensure that the historical record is trustworthy.

EcoLens integrates energy data from sources such as the Australian Energy Market Operator (AEMO) and OpenElectricity. It is designed to work with multiple electricity regions, including:

• New South Wales — NSW1
• Queensland — QLD1
• Victoria — VIC1
• South Australia — SA1
• Tasmania — TAS1
• Western Australia's WEM/SWIS data, where available

Each region has its own generation mix, weather conditions, consumption habits, and market behavior. A forecasting approach that performs well in one region may not perform equally well in another.

The data pipeline therefore handles several responsibilities:

The platform combines analytical and operational technologies, including DuckDB, PostgreSQL-based data marts, Redis, and object storage. dbt transformations help organize data into reusable analytical tables, while scheduled ingestion processes keep the system updated.

For a non-technical reader, this is similar to preparing ingredients before cooking. Even the best recipe cannot produce a good meal if the ingredients are missing, mislabeled, or inconsistent.

For a technical reader, the principle is equally important:

A forecast is only as dependable as the data pipeline supporting it.

There is no universally best forecasting model.

A simple statistical method can outperform a sophisticated neural network when demand is highly regular. A deep-learning model may discover nonlinear relationships that a simple baseline cannot. A pretrained foundation model may generalize well to unfamiliar datasets, but still lack the domain-specific information needed for a production energy application.

Instead of assuming that one architecture must win, EcoLens evaluates four different approaches:

The Seasonal Naive model is intentionally straightforward.

It assumes that the future will resemble a corresponding period in the past. If electricity demand follows a strong weekly pattern, the forecast for Monday at 2:00 p.m. might begin with demand observed during a previous Monday at 2:00 p.m.

The seasonal period depends on the dataset's sampling frequency and the recurring pattern being modeled. It might represent one day, one week, or another operational cycle.

Its value is not sophistication. Its value is comparison.

If an LSTM produces lower forecast error than Seasonal Naive, that suggests the neural network is learning something beyond ordinary repetition. If it performs worse, the added complexity may not be justified.

Seasonal Naive can also serve as a potential fallback when a machine-learning model fails a health check or encounters invalid inputs.

However, it cannot naturally understand sudden structural changes, unusual weather, market disruptions, long-term trends, or complex interactions between external variables.

It is not intended to be the most intelligent model.

It is intended to establish a reliable reference point.

EcoLens's current production deep-learning model is an LSTM, or Long Short-Term Memory network.

LSTMs are designed for sequential data. They process observations in order and maintain internal representations of earlier time steps, making them suitable for electricity demand forecasting.

Demand may rise after people wake up, peak during working hours, change in the evening, or behave differently during holidays. Recent demand can also provide clues about what happens next.

The EcoLens LSTM uses a two-layer recurrent encoder:

The first layer processes the input sequence, while the second learns a higher-level representation of the temporal patterns.

After encoding, the model applies attention pooling. This allows it to assign different importance to different historical observations rather than relying only on the final recurrent state.

For example, a recent demand spike, a similar time from the previous day, or a recurring weekly pattern may matter differently depending on the forecast context.

Predicting uncertainty

EcoLens does not only predict one number. Its LSTM produces a central forecast alongside lower and upper uncertainty estimates.

Conceptually:

Positive spread transformations, such as softplus, help ensure that the predicted spreads remain non-negative.

The model is trained using quantile-oriented objectives, including pinball loss. Conformal calibration is then used to improve the reliability of the resulting prediction intervals.

The LSTM is currently selected for production because it offers a practical balance between forecasting capability, computational cost, implementation maturity, serving simplicity, and operational reliability.

That does not mean it is permanently the best model.

The Temporal Fusion Transformer (TFT) is a more expressive forecasting architecture that combines recurrent components, attention, variable selection, gated residual connections, and static enrichment.

It is designed to work with different types of information, including historical observations, known future inputs, and static context.

EcoLens includes a hand-rolled TFT implementation as an alternative to the LSTM. Its static enrichment pathway exists conceptually, although the current static context is represented by a zero vector rather than a rich learned regional embedding.

The TFT is trained and evaluated independently, but it is not currently the sole production-serving model. A model can train successfully and still require additional validation before deployment. Accuracy, latency, resource usage, failure behavior, and reproducibility all matter.

The fourth approach is TimesFM, a pretrained time-series foundation model developed by Google.

TimesFM is used primarily in a zero-shot or frozen-checkpoint configuration. It offers an opportunity to test whether broad pretrained temporal knowledge can transfer to electricity demand without extensive task-specific training.

However, its current setup primarily uses raw demand history rather than the full set of EcoLens's domain-specific features. It may not directly incorporate weather forecasts, regional metadata, market variables, or operational constraints.

TimesFM is therefore valuable as an experimental benchmark — not an automatic replacement for a specialized production model.

The underlying principle is the same for both architectures:

Once a forecasting model works, another question becomes important:

EcoLens's production DemandLSTM has 128 hidden units per layer. That configuration was selected empirically, but model size should not become a permanent assumption simply because it worked once.

A smaller model could be cheaper to store, load, and run. But reducing capacity can also damage accuracy. EcoLens therefore approaches pruning as an experiment that must be measured — not as an optimization that is automatically considered successful.

Why structured pruning matters

EcoLens's pruning work is scoped to the LSTM. The TFT's interconnected variable-selection, gated-residual, and attention structure makes correct structural compaction substantially more complicated. TimesFM is also out of scope because it is used as a frozen foundation-model checkpoint.

The implementation performs structured pruning, meaning it removes complete logical hidden units rather than merely setting individual weights to zero.

This distinction matters.

Zero-masking leaves the model physically the same size. The tensor still occupies the same storage, and dense hardware may not run faster simply because some values are zero.

EcoLens instead performs physical compaction: it rebuilds a smaller model and copies only the surviving weights.

How units are selected

The pruning process ranks hidden units using L2-norm importance scores. It considers both:

• How strongly a unit is computed.
• How strongly that unit feeds into other units.

This matters because a unit can be important as an input to other units even if its own outgoing computation appears relatively small.

The compaction process updates the affected recurrent weights, biases, attention layer, and forecasting heads. It also returns the original indices of the surviving units, making the decision auditable.

A basic correctness invariant is tested: when the keep fraction is 1.0 and nothing is pruned, the compacted model must reproduce the original weights exactly.

The pruning gate

Compaction alone is not enough. EcoLens runs a complete pipeline:

1. 1Load a registered LSTM version.
2. 2Compact it structurally.
3. 3Measure parameter count, artifact size, and CPU inference latency.
4. 4Fine-tune the compacted model to recover lost capacity.
5. 5Re-evaluate it through the same walk-forward harness.
6. 6Check whether the result passes both efficiency and accuracy requirements.

The efficiency gate requires the artifact to be at least 5% smaller and not more than 10% slower. A separate accuracy check requires recovered walk-forward MAPE regression to remain within a default 1% tolerance of the unpruned version.

Even when the gate passes, the recovered model is registered to MLflow with a prune_gate_passed tag but is not automatically promoted to Production.

If pruning fails, the result is recorded honestly:

That is a useful engineering result, not something to hide.

A lesson from recovery

The pruning experiment also exposed a practical issue: recovery fine-tuning needs enough time to relearn after a structural cut.

At a 50% keep fraction, three recovery epochs produced a severe relative walk-forward MAPE regression of +131.2%. Fifteen epochs recovered performance to better than the unpruned version, reaching approximately −21.1% relative regression. The current default was subsequently increased to 20 recovery epochs.

The lesson is broader than pruning:

A forecast interval labeled P10–P90 may sound like an 80% confidence range. But raw quantile-regression outputs do not automatically guarantee that the true value will fall inside that interval 80% of the time.

A model can have low pinball loss and still produce intervals that are too narrow.

EcoLens addresses this with Conformalized Quantile Regression (CQR).

How CQR works

CQR uses a held-out calibration split that the model did not train on.

For each calibration sample, it measures how far the actual value falls outside the raw prediction interval:

It then calculates a calibrated quantile of these nonconformity scores and uses that value to adjust future intervals:

In simple terms, if the model's original interval was too narrow on held-out data, the calibration layer widens it by an amount supported by that evidence.

EcoLens persists the calibration result as an MLflow artifact, allowing serving to load and apply it automatically.

Why order matters

EcoLens applies regional bias correction before conformal calibration.

This is deliberate. Conformal calibration adjusts interval width around the model's central forecast; it does not fix a forecast that is systematically shifted too high or too low.

Bias correction addresses the center first. Conformal calibration then measures how much uncertainty remains around that corrected center.

Coverage is computed and logged for both raw and calibrated intervals so the result can be inspected rather than assumed.

The important limitation

Conformal coverage guarantees rely on assumptions such as exchangeability. Chronological time-series data does not strictly satisfy this assumption when demand patterns shift over time.

That means CQR is not a magical guarantee under every deployment condition. If the data distribution changes substantially, its historical calibration may no longer describe current reality.

It also cannot fix a systematically mis-centered forecast. That requires better modeling or bias correction.

Static calibration is calculated at training time. But real-world demand volatility can change after the model is deployed.

EcoLens therefore includes an adaptive calibration layer inspired by Adaptive Conformal Inference.

For each model-and-region pair, the system maintains a scalar multiplier in Redis. When a served forecast is later reconciled against the actual observed demand, the multiplier is updated based on whether the outcome fell inside the predicted interval.

Conceptually:

A miss gradually pushes the scale upward, widening future intervals. A hit nudges it downward.

The scale is bounded between 0.5 and 3.0, preventing it from collapsing toward zero or expanding without limit. The update step is deliberately small, allowing the system to respond gradually rather than overreact to a short streak of unusual outcomes.

The adaptive layer is not intended to replace the model. It addresses a different problem:

• Adaptive calibration: The model is still appropriate, but its uncertainty needs adjustment.
• Forecast-quality circuit breaker: The model's behavior has deteriorated enough that the system should fall back to a simpler method.

This separation is important. Not every calibration issue requires replacing the entire forecasting model.

What the regional results revealed

Walk-forward evaluation on real warehouse data showed uneven coverage before calibration and tuning:

The target coverage was 80%.

These results show why a single global correction is insufficient. Bias varies in both direction and magnitude across regions.

Per-region bias correction helped substantially in some cases. NSW1 and TAS1, for example, moved to within approximately 6 MW of zero bias in the follow-up experiment. But other regions became worse, suggesting that a small calibration split — roughly 25–27 rows per region — was not always stable enough to estimate persistent regional bias.

That result is not a failure of the experiment. It is evidence that calibration must be evaluated carefully, especially when data is limited and behavior changes over time.

All these mechanisms depend on consistent evaluation.

EcoLens uses a shared walk-forward harness to compare models under conditions that resemble real forecasting. The process measures central forecast accuracy, quantile loss, interval coverage, interval width, regional stability, and performance across different horizons.

This same discipline extends to pruning and calibration.

A smaller model is not accepted merely because its parameter count decreases. It must also preserve accuracy and meet latency requirements.

A calibrated interval is not trusted merely because a correction formula was applied. Its empirical coverage must be measured.

And a model is not promoted to production merely because it exists in the codebase.

The operational system must also handle missing observations, delayed ingestion, invalid values, model-loading failures, distribution changes, resource constraints, API timeouts, and regional outages.

EcoLens includes components for scheduled ingestion, transformation, model training, registration, serving, caching, monitoring, logging, metrics, tracing, and alerting.

A future forecast-quality circuit breaker can reject outputs containing NaN values, impossible negative demand, implausible jumps, severe drift, excessively wide intervals, or failed health checks — and fall back to the Seasonal Naive baseline when necessary.

The goal is not to pretend that models never fail.

EcoLens is not built around the assumption that the newest or most complicated model must be the best one.

Its architecture combines a simple baseline, a production LSTM, a more advanced Temporal Fusion Transformer, and a pretrained foundation model. Each offers a different trade-off between simplicity, expressiveness, generalization, uncertainty estimation, and operational cost.

But the platform goes further than model selection.

It asks whether the production LSTM can be physically reduced in size without sacrificing meaningful accuracy. It uses structured pruning, recovery fine-tuning, and explicit efficiency gates to answer that question.

It asks whether uncertainty intervals deserve to be trusted. It uses conformalized quantile regression to calibrate them against held-out evidence, then uses adaptive calibration to respond to drift after deployment.

And it records when these mechanisms do not work as hoped.

That is the common thread.

Ultimately, a forecasting model should not enter production because it is fashionable, complex, or impressive on paper.

It should enter production because it has demonstrated that it can produce useful predictions, communicate uncertainty honestly, operate reliably, and improve the decisions the platform is designed to support.

That is the standard EcoLens is working toward: