How it works

The engineering behind Fantasy Streamer — a daily batch pipeline, and the decisions that shaped it.

What it does

Fantasy Streamer runs once a day. It pulls schedules and box scores from the MLB Stats API, lands them as raw JSON, normalizes them into Parquet, rolls them up into season-level pitching features, and runs a model over today's probable starters to project how many innings each is likely to throw. The result is written as a single static JSON file per date and served from a CDN — no application server and no database anywhere in the path.

The current model is a linear baseline, chosen so the pipeline around it could be built and operated first. Swapping it is a single artifact change.

Architecture

  1. MLB Stats API schedules, box scores
    raw JSON
  2. Bronze raw JSON, date-partitioned
    parsed
  3. Silver normalized Parquet
    aggregated
  4. Gold / Features season-partitioned Parquet
    inference
  5. Model projected innings per starter
    Parquet + JSON
  6. S3 origin
    cached at the edge
  7. CloudFront CDN distribution
    fetch
  8. Browser the predictions table

Prefect 3 orchestrates the Bronze, Silver, Gold and prediction stages as one scheduled deployment, pulling flow source from GitHub at run time.

Daily flow from the MLB Stats API through to the browser.

Design decisions

Static files, not an API

The write happens once a day and the read is the same bytes for every visitor, so a server and a database would be infrastructure to operate, pay for and scale with nothing to do between runs. Prediction JSON is uploaded with Cache-Control: max-age=3600, so a slate is served from the edge for an hour before CloudFront goes back to the bucket for it. Every key is generated from one template table that yields both the local path and the S3 key — the two layouts are identical by construction, which makes a local run a true dry run of the cloud one.

Why the pipeline is layered

Changing a feature definition is a re-run over data already on disk, not a re-fetch from the MLB Stats API, because every layer keeps what it was derived from. Bronze holds API responses verbatim, partitioned by date; silver is those responses normalized into Parquet on the same partitioning; gold rolls the daily rows up into season-level pitching features, one file per season. Derivation only runs one direction, so any layer can be rebuilt from the one beneath it.

Deployment is a merge

Prefect pulls flow source from the repository's main branch when a run starts rather than baking it into an image, so merging to main is the deployment step — no image build, no registry push. The container is a stock prefecthq/prefect image plus a pinned dependency list. One scheduled deployment runs ingestion, box score conversion, the feature update and prediction in sequence at 4:00 AM Central.

Stack

Python ≥3.11 with pandas, scikit-learn, pyarrow, boto3 and MLB-StatsAPI, managed by uv. Prefect 3 for orchestration, S3 for storage, CloudFront for delivery. The front end is three static files — no framework, no build step, and nothing fetched from a third party at page load.

About