A fraud strategy workbench built from the full 284,807-row MLG-ULB benchmark, with chronological evaluation, calibrated scoring, and a capacity-bounded review queue.
Financial Payments Fraud Decision Workbench
How much known fraud can a fixed review team capture, and where does extra capacity stop paying back?
At one review per 1,000 transactions, 56 reviews capture 51 of 75 observed frauds at 91.1% precision and 68.0% recall. Reaching 80% needs 120 reviews; reaching 90% needs 2,430, exposing the saturation zone.
A fraud operations lead can compare threshold and staffing choices, see captured and missed fraud, inspect anonymized cases, and identify where extra review capacity stops paying back.
Data Scientist
Selected a calibrated logistic fraud model over XGBoost after the challenger missed a predeclared 0.02 calibration PR-AUC lift gate, then measured 0.7744 PR-AUC on an untouched chronological holdout.A fraud operations lead comparing score thresholds and review capacity, supported by an analyst inspecting the bounded queue.
Context and stakes
The decision context behind the system.
At the current review budget, 56 reviews capture 51 of 75 observed frauds with five non-fraud reviews. The next useful step is not simply more model complexity: moving to 80% observed recall needs 120 reviews, while 90% needs 2,430. The product makes that workload curve visible before a strategy owner changes policy.
A full-data payments-fraud strategy product that turns rare-event model evidence into an explicit staffing decision instead of stopping at a score.
Approach
A bounded approach shaped by the published constraints.
At one review per 1,000 transactions, 56 reviews capture 51 of 75 observed frauds at 91.1% precision and 68.0% recall. Reaching 80% needs 120 reviews; reaching 90% needs 2,430, exposing the saturation zone.
Design constraints
- The benchmark is static, anonymized, and spans roughly two elapsed days without calendar timestamps.
- Only 57 fraud rows occur in calibration and 75 in test, so sampling uncertainty is material.
- No card, customer, merchant, channel, geography, analyst disposition, intervention outcome, or live feedback exists.
- Source amount is not realized fraud loss or loss avoided.
Decision journey
The system moves through an inspectable sequence.
Each stage below comes from the published project manifest.
- Stage
Validate all source rows
Private full benchmark validated into stable transaction lineage
- Stage
Split train, calibration, and
Chronological train, calibration, and untouched test partitions
- Stage
Compare baseline and challenger
Class-weighted logistic baseline and XGBoost challenger
- Stage
Calibrate the selected model
Calibration gate selecting the simpler model before test evaluation
- Stage
Convert threshold and review
Immutable model, evaluation, and 56,962-row scored holdout artifact run
- Stage
Serve policy consequences, scenarios,
Dash and Flask decision workbench on bounded Cloud Run
Interactive architecture
The published architecture, made interactive.
The pipeline validates every source row, preserves chronology, selects complexity on calibration evidence, evaluates once on an untouched period, and writes one immutable run. The workbench converts that run into threshold and capacity consequences, a deterministic queue, a capacity frontier, and anonymized record evidence.
Drag to pan / use controls to zoom / select any stageRead the architecture as text
- Private full benchmark validatedStage 1 / Python
Private full benchmark validated into stable transaction lineage
- Chronological train, calibration, andStage 2 / pandas
Chronological train, calibration, and untouched test partitions
- Class-weighted logistic baseline andStage 3 / scikit-learn
Class-weighted logistic baseline and XGBoost challenger
- Calibration gate selecting theStage 4 / XGBoost
Calibration gate selecting the simpler model before test evaluation
- Immutable model, evaluation, andStage 5 / Dash
Immutable model, evaluation, and 56,962-row scored holdout artifact run
- Dash and Flask decisionStage 6 / Flask
Dash and Flask decision workbench on bounded Cloud Run
Pivotal decisions
Technology choices recorded with their trade-offs.
These choices are derived from the project's published decision record.
Use Class-weighted logistic regression with Platt calibration within the published boundary
The challenger missed the predeclared calibration lift needed to justify extra complexity, while the baseline produced calibrated probabilities for threshold analysis.
- Constraints
- The challenger scored higher on test but missed the predeclared calibration lift, so the logistic baseline shipped
- The default queue captures 68.0% of observed fraud with five non-fraud reviews, while 90% recall needs 2,430 reviews and 2.8% precision
- Scale-to-zero and one maximum instance minimize cost but do not provide a production availability objective
- Alternatives
- Promote XGBoost because its held-out test PR-AUC was higher
- Criteria
- Fit for the stated scope / Inspectable evidence / Explicit operating boundary
- Selected
- Class-weighted logistic regression with Platt calibration
- Why
- The challenger missed the predeclared calibration lift needed to justify extra complexity, while the baseline produced calibrated probabilities for threshold analysis.
- Trade-off
- The simpler model gives up diagnostic test lift to preserve a selection rule fixed before test evaluation.
- What supported it
- 68.0% Fraud recall at one review per 1,000: 51 of 75 observed frauds captured in a 56-row queue
- Revisit when
- Add authenticated analyst access before using protected operational data
Use One immutable artifact run within the published boundary
Model, evaluation, partition report, and scored holdout remain tied to one validated run ID, so every dashboard number resolves to the same evidence.
- Constraints
- The challenger scored higher on test but missed the predeclared calibration lift, so the logistic baseline shipped
- The default queue captures 68.0% of observed fraud with five non-fraud reviews, while 90% recall needs 2,430 reviews and 2.8% precision
- Scale-to-zero and one maximum instance minimize cost but do not provide a production availability objective
- Alternatives
- Recompute policies from multiple stores or retrain during serving
- Criteria
- Fit for the stated scope / Inspectable evidence / Explicit operating boundary
- Selected
- One immutable artifact run
- Why
- Model, evaluation, partition report, and scored holdout remain tied to one validated run ID, so every dashboard number resolves to the same evidence.
- Trade-off
- Training stays owner-operated and freshness is manual until a real label source and cadence exist.
- What supported it
- 91.1% Retrospective queue precision: Five of 56 reviewed rows are observed non-fraud
- Revisit when
- Add durable distributed request controls before permitting multiple instances
Use Dash and Flask in one process within the published boundary
One read-only process serves policy controls, APIs, charts, queue filtering, and drill-down without introducing infrastructure unsupported by the data.
- Constraints
- The challenger scored higher on test but missed the predeclared calibration lift, so the logistic baseline shipped
- The default queue captures 68.0% of observed fraud with five non-fraud reviews, while 90% recall needs 2,430 reviews and 2.8% precision
- Scale-to-zero and one maximum instance minimize cost but do not provide a production availability objective
- Alternatives
- Kafka, Spark, Redis, SQL, CDC, and separate API and frontend services
- Criteria
- Fit for the stated scope / Inspectable evidence / Explicit operating boundary
- Selected
- Dash and Flask in one process
- Why
- One read-only process serves policy controls, APIs, charts, queue filtering, and drill-down without introducing infrastructure unsupported by the data.
- Trade-off
- The compact topology does not simulate streaming or multi-service production behavior.
- What supported it
- 0.7744 Held-out PR-AUC: 95% bootstrap interval from 0.6704 to 0.8501 on 56,962 test rows
- Revisit when
- Add scheduled retraining and model-freshness monitoring only after a live label source exists
Use Scale-to-zero Cloud Run with private Cloud Storage artifacts within the published boundary
The complete real artifact run can remain private while the decision surface stays publicly reachable at bounded cost.
- Constraints
- The challenger scored higher on test but missed the predeclared calibration lift, so the logistic baseline shipped
- The default queue captures 68.0% of observed fraud with five non-fraud reviews, while 90% recall needs 2,430 reviews and 2.8% precision
- Scale-to-zero and one maximum instance minimize cost but do not provide a production availability objective
- Alternatives
- Always-on compute, a public artifact bucket, or a generated substitute dataset
- Criteria
- Fit for the stated scope / Inspectable evidence / Explicit operating boundary
- Selected
- Scale-to-zero Cloud Run with private Cloud Storage artifacts
- Why
- The complete real artifact run can remain private while the decision surface stays publicly reachable at bounded cost.
- Trade-off
- Cold starts remain possible, and the one-process request limiter cannot support a multi-instance production service.
- What supported it
- 120 Minimum reviews for 80% observed recall: 64 more reviews than the default queue for nine additional fraud captures
- Revisit when
- Add customer, merchant, channel, analyst-cost, and intervention outcomes before making operating or fairness claims
Results and evaluation
Measured results within the disclosed evaluation boundary.
- Method
- Calibration-selected threshold and deterministic chronological holdout ranking
- Meaning
- The default queue captures over two-thirds of observed fraud while reviewing 0.098% of holdout rows.
- Limitation
- The benchmark is static, anonymized, and spans roughly two elapsed days without calendar timestamps.
Limitations and next steps
Useful evidence, bounded claims.
What this does not prove
- The benchmark is static, anonymized, and spans roughly two elapsed days without calendar timestamps.
- Only 57 fraud rows occur in calibration and 75 in test, so sampling uncertainty is material.
- No card, customer, merchant, channel, geography, analyst disposition, intervention outcome, or live feedback exists.
- Source amount is not realized fraud loss or loss avoided.
- The public service is a bounded demonstration and does not establish production availability, latency, throughput, or security.
What I would do next
- Add authenticated analyst access before using protected operational data
- Add durable distributed request controls before permitting multiple instances
- Add scheduled retraining and model-freshness monitoring only after a live label source exists
- Add customer, merchant, channel, analyst-cost, and intervention outcomes before making operating or fairness claims
The source benchmark is private and anonymized. The public service exposes scored holdout outputs without cardholder, account, merchant, or customer identity. Results are retrospective and do not establish production performance, causal explanation, loss avoided, fairness, or authorization to approve or decline a payment.
Explore the rest of the work