A synthetic-only model evaluation lab that compares a prevalence baseline with a calibrated challenger, exposes calibration and threshold tradeoffs, and keeps patient data outside the public system.
Synthetic Sepsis Risk Evaluation Lab
How do you make a risk-model demo useful to reviewers without turning synthetic metrics into a clinical claim?
Start with the cheapest honest baseline, freeze a holdout, calibrate one reproducible challenger, expose threshold consequences, and keep the public data and deployment boundary visibly synthetic.
On the fixed synthetic holdout, the calibrated Extra Trees challenger improves AUROC from 0.5000 to 0.9796 and Brier score from 0.0475 to 0.0228. The dashboard turns those model results into an inspectable threshold tradeoff without presenting them as clinical evidence.
Data Scientist
Built a platform-stable synthetic model evaluation that compares a calibrated Extra Trees challenger with a prevalence baseline, improving holdout AUROC from 0.5000 to 0.9796 and Brier score from 0.0475 to 0.0228 with versioned calibration and threshold evidence.A data-science hiring manager, model reviewer, or technical stakeholder assessing reproducibility, calibration, threshold communication, and public data boundaries.
Context and stakes
The decision context behind the system.
The challenger clearly beats the prevalence-only reference on this fixed synthetic fixture, while the product remains explicit about what that does not prove. At threshold 0.20 it marks 8.7 rows per 100 with 86.7% synthetic sensitivity and 50.0% positive predictive value. That tradeoff is useful for reviewing model behavior, not for selecting a clinical alert threshold.
Risk-model portfolios often lead with a strong score and leave the harder questions unanswered: compared with what, calibrated how, at what review burden, and on which data boundary? This project makes those questions the product. A reviewer can compare the calibrated challenger with a prevalence baseline, change an illustrative threshold, inspect the resulting workload, and trace every public claim to deterministic synthetic evidence.
Approach
A bounded approach shaped by the published constraints.
Start with the cheapest honest baseline, freeze a holdout, calibrate one reproducible challenger, expose threshold consequences, and keep the public data and deployment boundary visibly synthetic.
Design constraints
- Synthetic labels are deterministic functions of same-row features, so the task is easier than real-world prediction.
- Rows have no patient, site, subgroup, or temporal structure; the evaluation cannot establish patient-level leakage control or transportability.
- The evidence has no confidence intervals, external validation, subgroup analysis, drift study, prospective workflow, or intervention effect.
- Threshold counts treat rows independently and do not model repeat alerts, suppression, staffing, or clinical workflow.
Decision journey
The system moves through an inspectable sequence.
Each stage below comes from the published project manifest.
- Stage
Generate a fixed synthetic
Deterministic synthetic cohort and feature contract
- Stage
Freeze a seeded stratified
Seeded stratified training and holdout split
- Stage
Fit the prevalence baseline
Prevalence-only reference and calibrated Extra Trees challenger
- Stage
Record discrimination, calibration, leakage,
Versioned evaluation JSON with fixed calibration bins
- Stage
Serve a synthetic-only score
Flask health, timeline, and synthetic scoring API
- Stage
Let reviewers change the
Accessible threshold and timeline dashboard
- Stage
Build a source-linked container
GitHub Actions container release
- Stage
Run the public demo
Azure Container Apps scale-to-zero deployment
Interactive architecture
The published architecture, made interactive.
A fixed synthetic generator creates the public cohort, then a seeded stratified split freezes training and holdout rows before fitting. The baseline predicts training prevalence. A single-threaded Extra Trees classifier receives three-fold isotonic calibration inside the training partition, chosen after a fixed-seed XGBoost result proved platform-sensitive. The evaluation command emits aggregate discrimination, calibration, leakage, and threshold evidence as versioned JSON. Flask serves health, synthetic timeline, and strictly validated synthetic-score routes; GitHub Actions tests the release and publishes a source-linked AMD64 image; Azure Container Apps runs that image with zero minimum replicas. MIMIC files and every patient-level derivative remain in an ignored local boundary and never enter the public repository or image.
Drag to pan · use controls to zoom · select any stageRead the architecture as text
- Deterministic synthetic cohort andStage 1 · Python
Deterministic synthetic cohort and feature contract
- Seeded stratified training andStage 2 · scikit-learn
Seeded stratified training and holdout split
- Prevalence-only reference and calibratedStage 3 · Flask
Prevalence-only reference and calibrated Extra Trees challenger
- Versioned evaluation JSON withStage 4 · Docker
Versioned evaluation JSON with fixed calibration bins
- Flask health, timeline, andStage 5 · GitHub Actions
Flask health, timeline, and synthetic scoring API
- Accessible threshold and timelineStage 6 · Microsoft Azure
Accessible threshold and timeline dashboard
- GitHub Actions container releaseStage 7 · Python
GitHub Actions container release
- Azure Container Apps scale-to-zeroStage 8 · scikit-learn
Azure Container Apps scale-to-zero deployment
- Separate ignored local boundaryStage 9 · Flask
Separate ignored local boundary for MIMIC research files and artifacts
Pivotal decisions
Technology choices recorded with their trade-offs.
These choices are derived from the project’s published decision record.
Use Calibrated Extra Trees within the published boundary
A single-threaded tree ensemble preserves nonlinear synthetic ranking while producing byte-stable evidence across macOS and Linux.
- Constraints
- Scale to zero minimizes idle compute but can add a cold start for the first visitor.
- The public image uses deterministic synthetic inputs only; MIMIC files, patient-level derivatives, local models, MLflow runs, and SHAP exports remain excluded.
- The score endpoint is an educational synthetic surface, not a monitored clinical service or operational alerting system.
- Alternatives
- Calibrated XGBoost
- Criteria
- Fit for the stated scope · Inspectable evidence · Explicit operating boundary
- Selected
- Calibrated Extra Trees
- Why
- A single-threaded tree ensemble preserves nonlinear synthetic ranking while producing byte-stable evidence across macOS and Linux.
- Trade-off
- The deterministic challenger gives up some synthetic AUPRC but avoids platform-dependent public evidence.
- What supported it
- 0.9796 Synthetic holdout AUROC: Calibrated Extra Trees versus 0.5000 for the prevalence-only baseline on 300 fixed holdout rows.
- Revisit when
- Keep the current public surface small until a real research cohort, external validation plan, and clinical governance owner exist.
Use Prevalence-only baseline and frozen holdout within the published boundary
The cheapest credible reference and a pre-fit holdout make improvement and calibration claims interpretable.
- Constraints
- Scale to zero minimizes idle compute but can add a cold start for the first visitor.
- The public image uses deterministic synthetic inputs only; MIMIC files, patient-level derivatives, local models, MLflow runs, and SHAP exports remain excluded.
- The score endpoint is an educational synthetic surface, not a monitored clinical service or operational alerting system.
- Alternatives
- Report challenger metrics without a named baseline
- Criteria
- Fit for the stated scope · Inspectable evidence · Explicit operating boundary
- Selected
- Prevalence-only baseline and frozen holdout
- Why
- The cheapest credible reference and a pre-fit holdout make improvement and calibration claims interpretable.
- Trade-off
- One small synthetic holdout is easy to reproduce but cannot estimate uncertainty, transportability, or patient-level performance.
- What supported it
- 0.0228 Synthetic holdout Brier score: Calibrated challenger versus 0.0475 for the constant 5% prevalence estimate.
- Revisit when
- If repeated-row workflows are studied, add patient-level temporal evaluation, alert deduplication, cooldown rules, and workload simulation before interpreting alert burden.
Use Native HTML, CSS, and SVG dashboard within the published boundary
The threshold control, timeline, table, and disclosure states need accessible browser primitives more than a charting framework.
- Constraints
- Scale to zero minimizes idle compute but can add a cold start for the first visitor.
- The public image uses deterministic synthetic inputs only; MIMIC files, patient-level derivatives, local models, MLflow runs, and SHAP exports remain excluded.
- The score endpoint is an educational synthetic surface, not a monitored clinical service or operational alerting system.
- Alternatives
- Add a component or visualization dependency
- Criteria
- Fit for the stated scope · Inspectable evidence · Explicit operating boundary
- Selected
- Native HTML, CSS, and SVG dashboard
- Why
- The threshold control, timeline, table, and disclosure states need accessible browser primitives more than a charting framework.
- Trade-off
- The project owns a small amount of rendering code but keeps the public runtime and interaction model compact.
- What supported it
- 8.7 per 100 Rows marked at threshold 0.20: 26 of 300 holdout rows, with synthetic sensitivity 0.8667 and positive predictive value 0.5000.
- Revisit when
- If traffic exceeds portfolio review volume, add measured request limits, durable telemetry, cost alerts, and an explicit availability objective before increasing replicas.
Use Azure Container Apps with a public GHCR image within the published boundary
A scale-to-zero container provides a verifiable HTTPS demo without a paid registry, database, or always-on replica.
- Constraints
- Scale to zero minimizes idle compute but can add a cold start for the first visitor.
- The public image uses deterministic synthetic inputs only; MIMIC files, patient-level derivatives, local models, MLflow runs, and SHAP exports remain excluded.
- The score endpoint is an educational synthetic surface, not a monitored clinical service or operational alerting system.
- Alternatives
- Provision Azure Container Registry and an always-on service
- Criteria
- Fit for the stated scope · Inspectable evidence · Explicit operating boundary
- Selected
- Azure Container Apps with a public GHCR image
- Why
- A scale-to-zero container provides a verifiable HTTPS demo without a paid registry, database, or always-on replica.
- Trade-off
- Cold starts and minimal observation are accepted for a low-traffic portfolio surface.
- What supported it
- 0 px / 0 violations Required responsive and automated accessibility checks: Stakeholder and technical states at 1440 px and 390 px.
- Revisit when
- Do not connect the demo to clinical systems or care workflows without prospective validation, subgroup analysis, security review, and formal safety governance.
Results and evaluation
Measured results within the disclosed evaluation boundary.
- Method
- Seeded row holdout with calibration fitted inside training rows only.
- Meaning
- The challenger ranks positive synthetic rows well above the constant reference.
- Limitation
- Synthetic labels are deterministic functions of same-row features, so the task is easier than real-world prediction.
Limitations and next steps
Useful evidence, bounded claims.
What this does not prove
- Synthetic labels are deterministic functions of same-row features, so the task is easier than real-world prediction.
- Rows have no patient, site, subgroup, or temporal structure; the evaluation cannot establish patient-level leakage control or transportability.
- The evidence has no confidence intervals, external validation, subgroup analysis, drift study, prospective workflow, or intervention effect.
- Threshold counts treat rows independently and do not model repeat alerts, suppression, staffing, or clinical workflow.
- The anonymous scale-to-zero demo has no availability objective, authentication, durable telemetry, or clinical monitoring.
- No result establishes clinical validity, clinical utility, safety, or suitability for patient care.
What I would do next
- Keep the current public surface small until a real research cohort, external validation plan, and clinical governance owner exist.
- If repeated-row workflows are studied, add patient-level temporal evaluation, alert deduplication, cooldown rules, and workload simulation before interpreting alert burden.
- If traffic exceeds portfolio review volume, add measured request limits, durable telemetry, cost alerts, and an explicit availability objective before increasing replicas.
- Do not connect the demo to clinical systems or care workflows without prospective validation, subgroup analysis, security review, and formal safety governance.
Synthetic educational research demo only. It contains no patient data, does not establish clinical validity or utility, does not provide clinical guidance, and must not be used for patient care.
Explore the rest of the work