# NFL Predictor Application Guide

## What The App Does

The application maintains an incremental NFL data store, trains separate candidates for every configured outcome, selects finalists with chronological validation, creates team and player prop distributions, derives projected scores, winner, spread, and game-total probabilities, and grades saved calls when real results arrive.

Team passing and rushing outcomes remain team totals. Player outcomes use a separate player-game table and are labeled with NFLverse/GSIS identity, position, team, roster source, snapshot, and inferred prop role. The player models cover passing, rushing and receiving yards; rushing plus receiving yards; pass attempts and completions; interceptions thrown; receptions; individual defensive sacks; and passing/rushing/receiving touchdowns. Reception probabilities run from `1+` through `12+`. Player passing touchdowns are passes thrown by a quarterback; receiving touchdowns are scores on caught passes; rushing touchdowns are scores on carries. Individual sacks are defensive `def_sacks`, including half-sacks, rather than sacks suffered by a quarterback.

The **Touchdown scorers** table ranks roster-verified players by their chance of scoring `1+` rushing/receiving touchdowns, alongside `2+` and `3+` probabilities and Monte Carlo uncertainty intervals. The dedicated `player_scoring_tds` model learns the combined rushing plus receiving count, not a sum of two probabilities. Touchdown passes thrown and return/defensive scores are excluded. `3+` remains available as a rare outcome rather than being hard-capped. **Update data + models** trains this new target; existing complete component outcomes can be expanded without refetching older seasons.

After an upgrade adds outcome fields, **Update data + models** automatically backfills cached seasons that have no completed observations of those fields. The task log says `Schema backfill` and lists seasons/fields; `source_data_manifest.json` retains this plan. Once the fields are present, completed seasons are reused normally. An outcome with too little chronological history is marked **Insufficient validation history** in Data & models and saved in its `training_status.json`. Other outcomes still complete, and the skipped target is reconsidered when source data changes. Missing truth is never replaced with fabricated zeros or a model trained without temporal validation.

## Recommended Workflow

1. Open **Data & models** and run **Update data + models** for the first setup. This fetches all requested seasons, builds team/player/matchup sources, resolves saved calls, conditionally trains models, refreshes future calls, and regenerates documentation.
2. Use **Fetch latest results** after games finish. It refreshes the active season and any unresolved prior-season rows, grades available calls, reports why anything remains pending, and only refits models when completed source rows changed.
3. Open **Predictions**, select an eligible scheduled game, and click **Run prediction**. Projection runs never retrain models.
   For the whole current-day slate, click **Run all today** instead. It submits one task per pre-kickoff game using the application timezone, in kickoff order. Games that are completed, already underway, or on another date are excluded.
4. Open the generated result from **Tasks** or **Saved runs**. Projected scores, total points and O/U, winner, spread, team props, and player props are presented together.
5. Use **Models** to compare finalists and inspect validation confidence, input variables, weighting, hyperparameters, coefficients, or feature importance.

## Data Sources

`team_game_stats.csv` contains one row per team/game, schedule context, final score, and team-level outcomes. Direct NFLverse weekly team stats are the primary source for completed passing/rushing yardage; player-stat aggregation fills fields missing from that feed. `player_game_stats.csv` contains historical yardage and pass/rush/receiving touchdown results plus future candidates verified against the latest applicable weekly roster and dated depth chart. Recent six-game passing, rushing, and receiving opportunity ranks only those verified players. A team/game without roster evidence receives no player props instead of a historical-name fallback.

`matchup_source.csv`, `train_data.csv`, and `test_data.csv` support the separate legacy win-probability workflow. The game prediction page primarily derives winner and spread probabilities from the two specialized `points_scored` distributions so those probabilities share the same game simulation as the projected score.

## Game Prediction Parallelism

**Run prediction** checks the schedule and saved-projection fingerprint before building histories. An unchanged projection skips player-history loading, feature generation and model scoring. Fingerprints include source files, requested targets/lines, finalist records, configuration and feedback calibration.

For an uncached game, independent team-stat and player-outcome history families run in a spawned process pool. Specialized team/player scoring jobs use that same pool afterwards. Historical calculations still use the existing leakage-safe builders; all configured feature families are assembled for the selected game, so cross-outcome predictors remain available. Participant order, roster eligibility, seeded probabilities and score/market assembly are preserved.

The automatic budget uses `models.parallel_jobs = "auto_minus_2"`, respects the UI's CPU affinity and bounds execution by the container-visible CPU bandwidth quota. On an unrestricted 24-logical-CPU flatserver this allows at most 22 processes, further limited by the number of independent jobs. `PREDICTION_PARALLEL_JOBS` can override the budget; use `1` for serial troubleshooting. Each job bounds model/native threads to one, including prediction from previously saved multi-threaded tree models. Saved model files are not changed.

The run log reports history/scoring phases, job START/DONE, worker PIDs and per-job wall/CPU seconds. The projection JSON's `execution` field records the CPU plan, worker PIDs, stage timings and observed `peak_parallel_jobs`. This observed overlap is distinct from the configured maximum and is not a claim of sustained utilization or a measured speedup. Source reads/hashing and final probability calibration/output assembly are serial, so CPU usage will fall during those stages or when only a few jobs remain.

Game inference remains a local background process pool on the app host in both local and distributed modes; the Celery pool distributes training jobs separately. Browser refresh does not stop either workflow. **Kill process** stops the local prediction process tree, including its spawned workers. Budget app-host inference together with any colocated Celery workers, and reduce `PREDICTION_PARALLEL_JOBS` if RAM pressure/swap or concurrent training hurts throughput. Rebuild the app image to deploy these changes; restarting the old image does not install the new execution path or its core `threadpoolctl` dependency.

## Today's Prediction Queue

**Run all today** creates a separate run ID and downloadable log for every eligible matchup. One game is **running** and the remaining games are **pending** in Tasks. Each game still uses its own multiprocessing budget internally. An unchanged saved prediction completes from cache without retraining or rescoring. Duplicate schedule rows are collapsed by game ID, and submitting another batch while a workflow is active is rejected.

Cancel any pending task individually without stopping the active prediction. **Kill process** kills the active process tree and cancels the rest of that batch. A failed game does not discard the other queued games. Eligibility is checked again when each task starts: a game whose kickoff or prediction date passed while waiting cannot produce a late pregame call.

Browser refreshes and closing the tab do not stop the queue. Restarting the application/container is different: abandoned pending batch tasks become **cancelled**, active batch tasks become **interrupted**, and nothing is automatically resumed. Submit the current pregame slate again after restart. Use one app/control-plane instance against a results directory.

## CPU Capacity And Cluster Views

Tasks displays visible logical CPUs, affinity, CPU quota, the CPU-minus-two recommendation, and Linux host/container CPU samples when available. A 24-thread host at about 8% total CPU utilization is roughly two busy logical CPUs, not evidence that each core needs multiple CPU-bound processes. Data downloads, cache hits, serial assembly and an empty or small queue can also leave CPUs idle. Windows previews report unavailable utilization rather than invented values.

Local target tournaments and legacy win candidates use process workers; inner estimator/native threads stay at one. Distributed workers have a separate **WORKER_CONCURRENCY** setting. Its default `auto_minus_2` policy resolves at worker startup using affinity and container quota, leaving two effective logical processors for the host/control plane. Explicit positive integers are also supported. Measure wall time, RAM/swap and queue depth; lower the slot count if concurrent feature tables create memory pressure. Colocated prediction and training pools share physical resources.

In distributed mode, Tasks is also the cluster frontend: online workers, distinct physical hosts, visible CPU totals, configured process slots and running jobs are shown together. Visible CPUs are capacity, not reserved cores. Each worker deployment includes a lightweight, read-only monitor, separate from its compute process, with connection/preflight state and planned concurrency. It remains accessible when worker preflight fails. See [Distributed Worker Deployment](distributed-worker-deployment.md) for private monitor ports and per-machine settings.

## Training And Validation

Every outcome has its own tournament. Walk-forward folds preserve time order. The configured final season is held out from candidate selection, scored after finalists are chosen, and then included in the deployment refit. Recent games receive more weight through exponential chronological decay. A changed completed-row signature triggers retraining; unchanged data reuses existing artifacts.

Player models are pooled models specialized by outcome, not a separately fitted model for every named player. The input row is player-specific: lagged passing yardage and opportunity features group by stable `player_id`, while team/opponent features supply context. A trade retains the player's own history; matching display names cannot mix different players. Passing-yard prediction rows now include prior-game count, last historical game/date and the player/usage/team/opponent feature snapshot under **Player inputs**. Residual uncertainty is pooled by target. A large individual-game miss alone cannot prove an identity mix-up; inspect this provenance and the prediction interval. These details appear after generating a new projection; older saved artifacts are not fabricated or silently rewritten.

## Spread And Winner Grading

Winner and spread probabilities share the simulated score distributions, but they are different events. A win requires `team score - opponent score > 0`. Cover requires `team score - opponent score + signed team handicap > 0`; equality is a push. PHI **+6** winning by two covers by eight; PHI **-6** winning by two fails to cover by four. A projected winning margin of 5.2 is a mean estimate, not a promise to win by six and not the criterion for grading a cover.

NFLverse's raw `spread_line` is a **home-favorite margin**, not the displayed home handicap: positive six means home **-6** and away **+6**. The corrected conversion is applied to new projections. The probabilities page distinguishes projected/actual raw margin from adjusted cover margin and retains the signed handicap and frozen model call.

Saved calls created before this correction are not silently rewritten after the result is known. **Fetch latest results** detects known legacy reversed-line rows, keeps the historical grade against the handicap the app actually displayed, shows both that original-call cover margin and the authoritative source-market cover margin, and excludes the row from calibration. A newly generated eligible prediction uses the corrected conversion. This keeps the audit trail honest without teaching the feedback engine from a malformed market definition.

## Feedback And Results

Automatic result grading compares the frozen yes/no decision for each marker with one completed source value. A marker such as `350+` is the event definition; the separate model-call field says `350+` or `Below 350` according to the saved decision probability. Manual feedback remains available as a fallback. Calibration is conservative and prior-shrunk, and cannot rewrite a historical decision after the outcome. Full model improvement comes from later source syncs adding completed games and triggering refits.

Pending calls include a machine-readable reason such as `game_not_found_in_team_source`, `final_score_not_available`, `player_source_not_available`, `player_not_found_in_result_source`, `target_stat_not_available`, or `completed_game_stat_missing_from_source`. The last reason means the final score exists but the refreshed source still lacks that requested stat.

## Tasks And Logging

Workflows run in background processes and survive browser refreshes. The Tasks page reports steps, process ID, elapsed and estimated time, live logs, log download, and process-tree termination. The same workflow lines are emitted to container standard output with run IDs so Coolify can display source fetches, row counts, cache decisions, model progress, result diagnostics, and failures.

## Generated Documentation

`results/documentation/model_reference.md` and `model_reference.json` are regenerated after model refreshes. They contain the variable dictionary, model-family definitions, and up to five finalist records per outcome with exact features and available coefficients or tree importance.