A retrospective validation workstation over 30,000 licensed academic credit records, built to compare a non-demographic model with simple references, quantify fixed review workloads, and inspect score-ranked sensitivity without making a lending decision.
Credit Risk Model Validation & Review-Capacity Lab
Does the research model clear simple references, and what does a fixed historical review workload contain?
It clears prevalence/random, repayment-delay, and logistic references on repeated paired development evidence, but ties calibrated Extra Trees because the PR-AUC advantage does not clear the fixed practical margin. A 10% simulated holdout queue contains 431 historical defaults among 600 rows, with uncertainty shown beside every measure.
The selected model clears prevalence/random, repayment-delay, and logistic references on repeated paired development evidence, but ties calibrated Extra Trees because the PR-AUC advantage does not clear the prespecified practical margin. At 10% holdout capacity, 600 historical rows contain 431 observed defaults with 71.8% precision and 3.25× lift, each reported with uncertainty.
Analytics / Model Validation
Built a 30,000-record credit-risk validation lab that kept a 6,000-row holdout frozen, compared the selected model with random, rule-based, logistic, and Extra Trees references across six paired development folds, and reported a tie when the practical margin was not met.A credit-risk analyst or model-risk reviewer assessing retrospective model readiness and review-capacity evidence.
Context and stakes
The decision context behind the system.
The selected model is useful for retrospective research but not uniquely superior to every credible alternative. Its frozen-holdout PR-AUC is 0.5764, and repeated paired evidence clears prevalence/random, repayment-delay, and logistic references. Against calibrated Extra Trees the PR-AUC gain is 0.0061 with a 95% interval of 0.0019-0.0109, so it does not clear the 0.010 margin and is reported as a tie. At 10% historical review capacity, 600 rows include 431 observed defaults with 71.8% precision and 3.25× lift. The product shows that workload and its uncertainty without treating rank sensitivity as an approval, denial, price, or consumer reason.
A model-risk reviewer can compare a non-demographic credit model with simple references, see where evidence is unresolved, test five historical review workloads with uncertainty, and inspect the exact rows behind the simulation without receiving a lending recommendation.
Approach
A bounded approach shaped by the published constraints.
It clears prevalence/random, repayment-delay, and logistic references on repeated paired development evidence, but ties calibrated Extra Trees because the PR-AUC advantage does not clear the fixed practical margin. A 10% simulated holdout queue contains 431 historical defaults among 600 rows, with uncertainty shown beside every measure.
Design constraints
- One historical academic population with one target horizon; no calendar-time or true out-of-time validation.
- No external, geographic, prospective, drift, operational, causal, loss, pricing, or lending-decision validation.
- Repeated folds remain correlated views of the same development population.
- Capacity intervals describe the frozen audit sample, not staffing needs or financial benefit.
Decision journey
The system moves through an inspectable sequence.
Each stage below comes from the published project manifest.
- Stage
Validate the integrity-pinned UCI
Local Python pipeline validates the licensed workbook, reproduces model evidence, and records source, split, command, tool, and revision lineage
- Stage
Lock selection on train
A version-4 static analyst artifact contains approved non-demographic fields, derived measures, retrospective out-of-fold scores, and deterministic ranks for all 30,000 rows
- Stage
Run paired repeated stratified
The React workstation computes cohort views client-side and renders model evidence from the immutable evaluation envelope using semantic HTML and native SVG
- Stage
Measure holdout ranking, calibration,
Cloudflare Pages serves the application and artifact; read-only Pages Functions expose health and current-release evidence
- Stage
Build and validate versioned
Neon stores one immutable aggregate release behind a least-privilege runtime credential; viewers have no write path
- Stage
Serve the read-only validation
Serve the read-only validation workstation with immutable release health and fail-closed availability
Interactive architecture
The published architecture, made interactive.
The central control is separation: model selection stays on train and validation data, the 6,000-row holdout is frozen under a private integrity record, and repeated stability tests run only on the 24,000 development rows. All candidates share the same folds, and a practical-margin rule turns small unresolved differences into ties. A versioned evaluation envelope then carries split identities, uncertainty, calibration warnings, cohort sample sizes, ablations, revision, command, and tool lineage into a 30,000-row non-demographic analyst artifact. The React workstation keeps holdout, development, artifact, and live-health evidence visually distinct, while Cloudflare and Neon provide a zero-dollar read-only serving path.
Drag to pan / use controls to zoom / select any stageRead the architecture as text
- Local Python pipeline validatesStage 1 / Python
Local Python pipeline validates the licensed workbook, reproduces model evidence, and records source, split, command, tool, and revision lineage
- A version-4 static analystStage 2 / pandas
A version-4 static analyst artifact contains approved non-demographic fields, derived measures, retrospective out-of-fold scores, and deterministic ranks for all 30,000 rows
- The React workstation computesStage 3 / scikit-learn
The React workstation computes cohort views client-side and renders model evidence from the immutable evaluation envelope using semantic HTML and native SVG
- Cloudflare Pages serves theStage 4 / PyArrow
Cloudflare Pages serves the application and artifact; read-only Pages Functions expose health and current-release evidence
- Neon stores one immutableStage 5 / TypeScript
Neon stores one immutable aggregate release behind a least-privilege runtime credential; viewers have no write path
Pivotal decisions
Technology choices recorded with their trade-offs.
These choices are derived from the project's published decision record.
Use Repeated paired stratified evaluation within the published boundary
Shared folds measure candidate differences without spending the frozen holdout on further selection.
- Constraints
- The source has one target horizon, so the fixed stratified holdout is not calendar-time or out-of-time validation.
- Repeated folds reduce dependence on one development split but remain correlated views of the same historical academic population.
- Capacity intervals describe one audit sample; they are not staffing, loss, revenue, or benefit forecasts.
- Alternatives
- Compare one score from one development split
- Criteria
- Fit for the stated scope / Inspectable evidence / Explicit operating boundary
- Selected
- Repeated paired stratified evaluation
- Why
- Shared folds measure candidate differences without spending the frozen holdout on further selection.
- Trade-off
- Six folds cost more compute and are still correlated views of one source population.
- What supported it
- tie Selected model versus Extra Trees: Mean PR-AUC delta 0.0061 with paired 95% interval 0.0019-0.0109, short of the prespecified 0.010 practical margin.
- Revisit when
- The 15 MB static analyst artifact is the current load constraint. At materially larger row counts, move filtered aggregates and paginated records behind a read-only API while retaining the same public schema and privacy tests.
Use Static 30,000-row analyst artifact within the published boundary
Full-record filtering and inspection work without a public per-record query service or viewer credentials.
- Constraints
- The source has one target horizon, so the fixed stratified holdout is not calendar-time or out-of-time validation.
- Repeated folds reduce dependence on one development split but remain correlated views of the same historical academic population.
- Capacity intervals describe one audit sample; they are not staffing, loss, revenue, or benefit forecasts.
- Alternatives
- Server-side record API
- Criteria
- Fit for the stated scope / Inspectable evidence / Explicit operating boundary
- Selected
- Static 30,000-row analyst artifact
- Why
- Full-record filtering and inspection work without a public per-record query service or viewer credentials.
- Trade-off
- The approximately 15 MB artifact dominates initial transfer and does not scale to much larger datasets.
- What supported it
- 431 of 600 Historical defaults inside the 10% simulated review set: Precision 0.7183, recall 0.3248, and lift 3.2479; each measure carries a 95% interval.
- Revisit when
- The evaluation is offline and batch by design. A recurring validation program would need new time-indexed data, scheduled runs, drift ownership, alert thresholds, and evidence retention before any service automation.
Use Native SVG and semantic HTML within the published boundary
The small set of analytical charts needs linked interaction and accessible tabular alternatives, not a broad charting framework.
- Constraints
- The source has one target horizon, so the fixed stratified holdout is not calendar-time or out-of-time validation.
- Repeated folds reduce dependence on one development split but remain correlated views of the same historical academic population.
- Capacity intervals describe one audit sample; they are not staffing, loss, revenue, or benefit forecasts.
- Alternatives
- A chart-library dependency
- Criteria
- Fit for the stated scope / Inspectable evidence / Explicit operating boundary
- Selected
- Native SVG and semantic HTML
- Why
- The small set of analytical charts needs linked interaction and accessible tabular alternatives, not a broad charting framework.
- Trade-off
- Chart primitives and accessibility behavior are maintained in the project.
- What supported it
- 0.0124 Frozen-holdout 10-bin calibration error: Calibration slope 1.0743 and intercept 0.0757; one high-score bin is sparse at n=18.
- Revisit when
- Neon stores one immutable aggregate release. Multiple concurrent releases would require an explicit release catalog and retention policy rather than overwriting the current snapshot.
Use Cloudflare Pages and Neon Free within the published boundary
A static workstation plus one read-only aggregate release fits a zero-dollar public demonstration.
- Constraints
- The source has one target horizon, so the fixed stratified holdout is not calendar-time or out-of-time validation.
- Repeated folds reduce dependence on one development split but remain correlated views of the same historical academic population.
- Capacity intervals describe one audit sample; they are not staffing, loss, revenue, or benefit forecasts.
- Alternatives
- A managed application server and database tier
- Criteria
- Fit for the stated scope / Inspectable evidence / Explicit operating boundary
- Selected
- Cloudflare Pages and Neon Free
- Why
- A static workstation plus one read-only aggregate release fits a zero-dollar public demonstration.
- Trade-off
- There is no uptime guarantee, scheduled monitoring, write workflow, or multi-release catalog.
- What supported it
- 30,000 rows Validated public research artifact: All licensed rows are available for non-demographic analysis; protected fields and local fairness evidence are excluded from public records.
- Revisit when
- No operational credit workflow should be added until external, out-of-time, prospective, legal, and fair-lending validation supports that change in scope.
Results and evaluation
Measured results within the disclosed evaluation boundary.
- Method
- Two repeats of three shared stratified development folds.
- Meaning
- The evidence does not support a practically resolved winner between the two strongest candidates.
- Limitation
- One historical academic population with one target horizon; no calendar-time or true out-of-time validation.
Limitations and next steps
Useful evidence, bounded claims.
What this does not prove
- One historical academic population with one target horizon; no calendar-time or true out-of-time validation.
- No external, geographic, prospective, drift, operational, causal, loss, pricing, or lending-decision validation.
- Repeated folds remain correlated views of the same development population.
- Capacity intervals describe the frozen audit sample, not staffing needs or financial benefit.
- Cross-cohort PR-AUC values reflect different prevalence levels and are not directly comparable without context.
- Ablations are model-reliance evidence only and cannot be used as consumer explanations or adverse-action reasons.
- The free-tier public demonstration has no uptime guarantee or scheduled monitoring.
- Automated accessibility checks do not replace manual keyboard, zoom, screen-reader, and contrast review.
What I would do next
- The 15 MB static analyst artifact is the current load constraint. At materially larger row counts, move filtered aggregates and paginated records behind a read-only API while retaining the same public schema and privacy tests.
- The evaluation is offline and batch by design. A recurring validation program would need new time-indexed data, scheduled runs, drift ownership, alert thresholds, and evidence retention before any service automation.
- Neon stores one immutable aggregate release. Multiple concurrent releases would require an explicit release catalog and retention policy rather than overwriting the current snapshot.
- No operational credit workflow should be added until external, out-of-time, prospective, legal, and fair-lending validation supports that change in scope.
This is a public, retrospective academic research demonstration using the UCI Default of Credit Card Clients dataset under CC BY 4.0. It is not validated for lending use and does not make an approval, denial, eligibility, pricing, adverse-action, or lending recommendation.
Explore the rest of the work