FINANCIAL SERVICES / CONSUMER CREDIT
AnalyticsData Engineering
Credit Risk Model Validation & Review-Capacity Lab
The selected model clears prevalence/random, repayment-delay, and logistic references on repeated paired development evidence, but ties calibrated Extra Trees because the PR-AUC advantage does not clear the prespecified practical margin. At 10% holdout capacity, 600 historical rows contain 431 observed defaults with 71.8% precision and 3.25× lift, each reported with uncertainty.
STAKEHOLDER VIEW
What this project is for.
- Problem
- A model-risk reviewer can compare a non-demographic credit model with simple references, see where evidence is unresolved, test five historical review workloads with uncertainty, and inspect the exact rows behind the simulation without receiving a lending recommendation.
- Intended user
- A credit-risk analyst or model-risk reviewer assessing retrospective model readiness and review-capacity evidence.
- Decision supported
- Assess retrospective model readiness and compare fixed simulated review workloads without making an individual credit decision.
- Outcome
- The selected model is useful for retrospective research but not uniquely superior to every credible alternative. Its frozen-holdout PR-AUC is 0.5764, and repeated paired evidence clears prevalence/random, repayment-delay, and logistic references. Against calibrated Extra Trees the PR-AUC gain is 0.0061 with a 95% interval of 0.0019–0.0109, so it does not clear the 0.010 margin and is reported as a tie. At 10% historical review capacity, 600 rows include 431 observed defaults with 71.8% precision and 3.25× lift. The product shows that workload and its uncertainty without treating placement as an approval, denial, price, or consumer reason.
- What to try
- Confirm the evaluation timestamp, frozen-holdout checksum, source checksum, and readiness verdict
- Important limitation
- One historical academic population with one target horizon; no calendar-time or true out-of-time validation.
Reviewing whether a 10% simulated queue is supported
- 01Confirm the evaluation timestamp, frozen-holdout checksum, source checksum, and readiness verdict
- 02Compare the selected model with prevalence/random, repayment-delay, logistic, and Extra Trees references
- 03Read the paired interval and practical-margin verdict instead of choosing the highest point estimate
- 04Select 10% capacity and inspect queue size, captured historical defaults, precision, recall, lift, and their intervals
- 05Check calibration, sparse bins, non-demographic cohort sample sizes, and feature-group reliance before opening a record
- 06Inspect deterministic rank and inside/outside simulated placement alongside the research-only disclaimer
Technical reviewArchitecture, evidence, controls, deployment, and trade-offs
ARCHITECTURE
One source, one rendered system view.
The central control is separation: model selection stays on train and validation data, the 6,000-row holdout is frozen by ID checksum, and repeated stability tests run only on the 24,000 development rows. All candidates share the same folds, and a practical-margin rule turns small unresolved differences into ties. A versioned evaluation envelope then carries split identities, uncertainty, calibration warnings, cohort sample sizes, ablations, hashes, revision, command, and tool lineage into a 30,000-row non-demographic analyst artifact. The React workstation keeps holdout, development, artifact, and live-health evidence visually distinct, while Cloudflare and Neon provide a zero-dollar read-only serving path.
End-to-end flow
- 01Validate the checksum-pinned UCI workbook and exclude demographic fields from model inputs
- 02Lock selection on train and validation data while freezing the 6,000-row holdout by ID checksum
- 03Run paired repeated stratified development evaluation against three simple references and Extra Trees
- 04Measure holdout ranking, calibration, five review workloads, non-demographic cohort robustness, and feature-group reliance
- 05Build and validate versioned aggregate and 30,000-row non-demographic analyst artifacts
- 06Serve the read-only validation workstation with immutable release health and fail-closed availability
Technology stack
Pythonpandasscikit-learnPyArrowTypeScriptReactCloudflare Pages and WorkersNeon Postgres
Technology decisions
| Decision | Why | Alternative | Trade-off |
|---|---|---|---|
| Repeated paired stratified evaluation | Shared folds measure candidate differences without spending the frozen holdout on further selection. | Compare one score from one development split | Six folds cost more compute and are still correlated views of one source population. |
| Static 30,000-row analyst artifact | Full-record filtering and inspection work without a public per-record query service or viewer credentials. | Server-side record API | The approximately 15 MB artifact dominates initial transfer and does not scale to much larger datasets. |
| Native SVG and semantic HTML | The small set of analytical charts needs linked interaction and accessible tabular alternatives, not a broad charting framework. | A chart-library dependency | Chart primitives and accessibility behavior are maintained in the project. |
| Cloudflare Pages and Neon Free | A static workstation plus one read-only aggregate release fits a zero-dollar public demonstration. | A managed application server and database tier | There is no uptime guarantee, scheduled monitoring, write workflow, or multi-release catalog. |
Evaluation and evidence
| Metric | Plain-language meaning | Score / value | Dataset / scenario | Threshold / baseline | Interpretation | Evidence |
|---|---|---|---|---|---|---|
| Selected model versus Extra Trees | The evidence does not support a practically resolved winner between the two strongest candidates. | tie | 24,000 development rows | No threshold or baseline recorded. | Repeated folds remain correlated views of one historical population | evaluation.development |
| Historical defaults inside the 10% simulated review set | What one fixed historical workload contains, not what a lender should do. | 431 of 600 | Frozen 6,000-row holdout at 10% simulated review capacity | No threshold or baseline recorded. | Historical audit-sample estimate, not staffing or business benefit | review.capacity |
| Frozen-holdout 10-bin calibration error | Probability quality is strong overall but the thin high-score region is not presented as settled. | 0.0124 | Retrospective academic ranking and probability evidence | No threshold or baseline recorded. | Fixed historical split, not calendar-time or out-of-time validation | evaluation.holdout, validation.calibration |
| Validated public research artifact | The portfolio and record views reconcile to the complete approved source population. | 30,000 rows | UCI Default of Credit Card Clients | No threshold or baseline recorded. | One historical academic population with one target horizon; no calendar-time or true out-of-time validation. | data.source, privacy.boundary |
These are Frozen 6,000-row stratified holdout plus two repeats of three paired stratified development folds results, not a production service-level objective. Unknown values are shown as “Not recorded”; units and special characters retain their source meaning.
Technical terms and value conventions
| Term | Plain-language use | How this project uses it |
|---|---|---|
| Evaluation score | A measured result from a defined test or benchmark, not a general promise. | Read it with the dataset, scenario, and limitation shown in the evaluation table. |
| Unknown / null | A value that was not observed or is intentionally not claimed. | Rendered as “Not recorded” rather than silently converted to zero or omitted. |
| Special characters | Symbols such as %, currency, Unicode, and escaped input retain their source meaning. | Units and encoding are explained where they affect the decision. |
Data boundary
| Classification | Source | Permitted use | Excluded data |
|---|---|---|---|
| public | Yeh, I. (2009), Default of Credit Card Clients, UCI Machine Learning Repository, DOI 10.24432/C55S3H; extracted workbook SHA-256 30c6be3abd8dcfd3e6096c828bad8c2f011238620f5369220bd60cfc82700933 | Retrospective research, reproducible evaluation, model validation demonstration, and portfolio presentation | sex; education; marriage; age; credentials; model binaries; approval or denial outputs; pricing or adverse-action reasons |
This is a public, retrospective academic research demonstration using the UCI Default of Credit Card Clients dataset under CC BY 4.0. It is not validated for lending use and does not make an approval, denial, eligibility, pricing, adverse-action, or lending recommendation.
Security and privacy controls
| Control | Implementation | Evidence | Limitation |
|---|---|---|---|
| Protected-field exclusion | Public artifact validators reject demographic fields, local fairness evidence, credentials, model binaries, and decision outputs. | privacy.boundary | The source remains a public academic dataset; this control does not establish operational fair-lending compliance. |
| Read-only release path | Viewers receive static records and read-only release/health responses through a least-privilege Neon role. | deployment.live | Free-tier availability has no service-level guarantee or scheduled monitor. |
| Immutable candidate verification | The clean local gate and GitHub Actions must pass for the same full Git revision before a separately approved deployment. | delivery.quality | Private-repository branch protection is unavailable on the current plan, so this remains a compensating manual control. |
Deployment and cost boundary
| Provider | Runtime | State | Exposure | Verified | Production claim |
|---|---|---|---|---|---|
| Cloudflare Pages and Workers with Neon Postgres | Static React application plus read-only Pages Functions; immutable aggregate evidence in Neon Free | live | anonymous | 2026-08-10T16:15:12Z | No |
| Component | Boundary | Implication |
|---|---|---|
| Cloudflare Pages and Workers | Existing Free plan, no custom domain or paid capacity | The static application and small read-only Functions stay at zero incremental cost without an uptime guarantee. |
| Neon Postgres | Existing Free project with one immutable aggregate release | The database is evidence storage, not a per-record serving tier or operational system. |
| Governed analyst artifact | Approximately 15 MB transferred as one static file | Full-record client analysis avoids per-record infrastructure but dominates first load. |
Known limitations
- One historical academic population with one target horizon; no calendar-time or true out-of-time validation.
- No external, geographic, prospective, drift, operational, causal, loss, pricing, or lending-decision validation.
- Repeated folds remain correlated views of the same development population.
- Capacity intervals describe the frozen audit sample, not staffing needs or financial benefit.
- Cross-cohort PR-AUC values reflect different prevalence levels and are not directly comparable without context.
- Ablations are model-reliance evidence only and cannot be used as consumer explanations or adverse-action reasons.
- The free-tier public demonstration has no uptime guarantee or scheduled monitoring.
- Automated accessibility checks do not replace manual keyboard, zoom, screen-reader, and contrast review.
Scalability roadmap
- The 15 MB static analyst artifact is the current load constraint. At materially larger row counts, move filtered aggregates and paginated records behind a read-only API while retaining the same public schema and privacy tests.
- The evaluation is offline and batch by design. A recurring validation program would need new time-indexed data, scheduled runs, drift ownership, alert thresholds, and evidence retention before any service automation.
- Neon stores one immutable aggregate release. Multiple concurrent releases would require an explicit release catalog and retention policy rather than overwriting the current snapshot.
- No operational credit workflow should be added until external, out-of-time, prospective, legal, and fair-lending validation supports that change in scope.