Platform Architecture

How ecoLens works end-to-end — from external energy APIs to the dashboard in front of you.

End-to-end pipeline

From external energy APIs to the dashboard in front of you — decoupled, event-driven, fully auditable.

1. Ingestion
Every 5 min · 9 sources
Scheduled cron fetches external REST APIs
9 data sources (AEMO NEM/WEM, BoM, Open-Meteo, …)
Stages into DuckDB for fast local processing
Publishes new_data event to RabbitMQ
event
2. Warehousing
RabbitMQ + dbt + Postgres
new_data event triggers dbt flow
Raw data copied to raw.* schema (untouched)
dbt models build stg → int → mart layers
Curated tables land in PostgreSQL (NeonDB)
features
3. Predictive Modeling
LSTM · TFT · TimesFM
4-model ensemble, all producing P10/P50/P90
Online + incremental learning on streaming data
Conformal calibration self-corrects intervals
MLflow tracks experiments, versions, artifacts
API
4. Frontend
Next.js 14 · REST APIs
Dashboard BFF calls internal services via REST
Real-time KPIs, forecast charts, carbon insights
LazyMotion + GSAP for animated visualisations
This page is the result of that whole chain

What does the platform answer?CORE QUESTIONS

Q1
How much electricity will be needed over the next 24 hours?

Probabilistic demand forecast — P10 (conservative), P50 (expected), P90 (peak). Decomposed by NEM region (NSW1, QLD1, VIC1, SA1, TAS1) and WEM.

Q2
How clean will that electricity be, based on the expected generation mix?

Forecast contribution of coal, gas, wind, solar, hydro, and battery. Translated into carbon intensity (gCO₂e/kWh) and total emissions.

Latency budget (typical)

Source fetch (AEMO NEM)~480 ms
Anomaly detection (per record)< 5 ms
dbt warehouse build2-5 min
Feature engineering for ML< 30 s
Model inference (4 models)~50 ms
Conformal calibration< 10 ms
API response (cached)5 ms
API response (uncached)80-200 ms
Dashboard render< 100 ms
Total end-to-end≈ 5-7 min (ingest → user)

Why this architecture works

Decoupled, event-driven, auditable.

Decoupled stages

Ingestion never blocks on warehousing. If the warehouse is rebuilding, new data still keeps flowing. The event bus (RabbitMQ) absorbs the dependency.

Near-real-time

Forecasts are updated every 5 min. Anomalies are flagged within seconds of arrival. The dashboard cache is 30-60 s — fresh enough to act on.

Fully auditable

Raw data is stored exactly as received. dbt models are version-controlled in git. Every transformation has lineage back to the source.