page-08.html / ML Engineering & MLOps
Page 09 GitHub

Page 09 · ML Engineering & MLOps

Machine Learning
Infrastructure & Models

Three purpose-built models operationalize the finance intelligence layer — anomaly detection, cash forecasting, and invoice classification — unified by a single MLOps spine: continuous retraining, multi-signal concept-drift monitoring, and HITL override rates as one of several early-warning signals of model health[cite: 12].

Models3 production[cite: 12]
PlatformVertex AI[cite: 12]
RetrainingWeekly · GreenOps[cite: 12]
ComplianceEU AI Act Art. 13[cite: 12]

System Architecture

End-to-End ML Pipeline

Data flows from source systems through the Feature Store to three independent model inference paths, each gated by HITL queues before downstream automation[cite: 12].

SOURCE SYSTEMS SAP · NetSuite · TMS EXTERNAL FX Rates · Calendar IC HISTORY 36mo transactions DATA GOV M-08 Schema valid. Lineage tag PII masking TFDV skew check on feature serve FEATURE STORE Vertex AI Online serving Batch export Time-travel IC ANOMALY IsoForest→XGBoost score > 0.72 → HITL-IC-01 CASH FORECAST LightGBM + Prophet 7-day PI · 90% MAPE >8% alert INV. EXCEPTION XGBoost multiclass APPROVED > 0.92 → auto-approve HITL QUEUE HITL-IC-01 HITL-CF-02 HITL-INV-03 OVERRIDE RATE ONE OF SEVERAL DRIFT SIGNALS → BigQuery ae_finance .model_drift → PSI + Retrain AUTOMATION IC Posting Agents TREASURY Liquidity Dashboard AP WORKFLOW Auto-approve/queue VERTEX AI PIPELINES — WEEKLY RETRAINING CYCLE GreenOps · ±6h carbon window · Model Registry · Version pinning

← scroll to view full diagram →

Data flows from SAP/TMS/External sources through M-08 Data Governance and the Vertex AI Feature Store into three parallel model paths (IC Anomaly, Cash Forecast, Invoice Classifier). Each model is gated by a HITL review queue before automation. Override rates and PSI signals loop back to trigger retraining[cite: 12].

Model Specifications

Three Production Models

Each model card documents architecture, features, thresholds, performance metrics, explainability method, and drift signals — stored alongside the model artifact in the Registry[cite: 12].

Model 01 · Anomaly Detection[cite: 12]

IC Anomaly Detector

Two-Stage: IsoForest → XGBoost[cite: 12]
Two-stage architecture: IsolationForest (unsupervised) generates an outlier score for every IC transaction pair[cite: 12]. Pairs scoring above a pre-filter threshold (0.50) are passed as candidates to a supervised XGBoost classifier, trained on verified mismatch labels from prior audit cycles[cite: 12]. The final anomaly_score is the XGBoost probability output — IsoForest acts as a computational gate, not a voting member[cite: 12]. This distinction preserves label-free coverage while leveraging supervision where labels exist[cite: 12].

Inference Flow

IC TRANSACTION Pair ingestion ISOLATION FOREST XGBOOST FINAL SCORE TREE EXPLAINER THRESHOLD HITL-IC-01

Task

Score each intercompany transaction pair for mismatch probability, routing high-confidence anomalies to the HITL-IC-01 queue before any automated posting[cite: 12].

Training Data

36 months of IC transaction history (ClaraVis Group + anonymised industry comparables)[cite: 12]. Positive labels: verified mismatch cases from prior audit cycles[cite: 12].

Input Features

entity_pair_id amount_vs_agreement_rate posting_timing_delta prior_mismatch_count currency_pair period_in_cycle

Performance Metrics

Precision @ 0.720.91[cite: 12]
Recall @ 0.720.87[cite: 12]
AUC-PR (held-out)0.94 · 6-month holdout[cite: 12]
Anomaly thresholdanomaly_score > 0.72 → HITL-IC-01[cite: 12]
Drift signalsPSI on features · HITL override rate (7-day) · prediction score dist[cite: 12].
ExplainabilityTreeExplainer · deterministic SHAP · EU AI Act Art. 13[cite: 12]

Model 02 · Time Series Forecasting[cite: 12]

Cash Forecast Model

LightGBM + Prophet Ensemble[cite: 12]

Ensemble Architecture & Forecast Flow

STRUCTURAL FEATURES CALENDAR FEATURES LIGHTGBM PROPHET WEIGHTED ENSEMBLE PI WIDTH CHECK CONFIDENCE TREASURY DASHBOARD

Task

Produce a 7-day rolling cash position forecast per legal entity and currency, with calibrated 90% prediction intervals to support liquidity management decisions[cite: 12].

Ensemble Design

LightGBM captures structural signals (scheduled payments, IC sweeps, FX moves)[cite: 12]. Prophet contributes calendar seasonality[cite: 12]. Outputs are blended with rolling weight optimisation[cite: 12].

Input Features

historical_balance_7d scheduled_payments IC_sweep_history FX_rate_7d_ma payroll_calendar invoice_due_dates

Performance Metrics

Achieved MAPE (7-day)4.3% · 3-month backtest[cite: 12]
PI coverage (90%)91.2% empirical coverage[cite: 12]
HITL trigger — PI widthPI width > €1.5M → HITL-CF-02[cite: 12]
HITL trigger — confidencePoint estimate confidence < 0.75[cite: 12]
Drift signalsforecast_vs_actual MAPE >8% · PSI on balance features[cite: 12]

Model 03 · Multiclass Classification[cite: 12]

Invoice Exception Classifier

XGBoost Multiclass[cite: 12]

Classification Flow & Output Classes

INVOICE XGBOOST MULTICLASS APPROVED PO_MISMATCH PRICE_VARIANCE MISSING_CC DUPLICATE P(APPROVED) > 0.92 TREE EXPLAINER HITL-INV-03

Task

Classify each incoming invoice into one of five exception categories or APPROVED[cite: 12]. High-confidence APPROVED predictions bypass the AP queue; all others receive SHAP explanations for the reviewer[cite: 12].

Output Classes

APPROVED PO_MISMATCH PRICE_VARIANCE MISSING_CC DUPLICATE

Input Features

po_match_score unit_price_variance_pct cost_center_present supplier_invoice_hash supplier_prior_exception_rate amount_band

Performance Metrics

Macro F1 (5-class)0.89 · 90-day holdout[cite: 12]
APPROVED precision0.97 @ 0.92 threshold[cite: 12]
Auto-approve thresholdP(APPROVED) > 0.92 → skip queue[cite: 12]
ExplainabilitySHAP for every non-APPROVED class · Art. 13[cite: 12]
Drift signalsException rate trend (7d MA) · PSI · APPROVED FP rate via HITL[cite: 12]

Shared Infrastructure

MLOps Pipeline

A single operational spine supports all three models — from weekly retraining on GreenOps schedules to per-model drift alerting in BigQuery[cite: 12].

VERTEX AI PIPELINES — WEEKLY RETRAINING CYCLE TRIGGER Weekly schedule GreenOps ±6h carbon window FEATURE EXPORT TRAINING JOBS EVALUATION MODEL REGISTRY DEPLOY CONCEPT DRIFT MONITORING BIGQUERY DRIFT METRICS CLOUD MONITORING RETRAIN TRIGGER

Vertex AI Pipelines

All three models retrain on a weekly cadence via Vertex AI managed pipelines[cite: 12]. Job scheduling is GreenOps-aware, with a ±6-hour flex window to target low-carbon compute availability[cite: 12].

Weekly cadenceGreenOps ±6hParallel jobsFlexible batch

Model Registry & Canary Deploy

Every trained artifact is version-pinned in the Vertex Model Registry with a data hash and pipeline run ID[cite: 12]. Deployment uses a canary pattern: 10% traffic for 48 hours, then full promotion — or automatic rollback to the pinned champion if the override rate exceeds 1.5× baseline[cite: 12].

Version pinningCanary 10%→100%Auto-rollbackModel cards

Multi-Signal Drift Monitoring

Drift is detected via three concurrent signals: Population Stability Index (PSI) on input features (BigQuery scheduled query — triggers retraining if PSI > 0.2 on any key feature), prediction confidence distribution shift (Cloud Monitoring), and HITL override rate as an early behavioral proxy[cite: 12]. Rising override rate alone validates PSI before triggering retraining[cite: 12].

PSI on featuresBigQueryCloud MonitoringOverride rate proxy

Training-Serving Skew Detection

Training-serving skew is validated post-deployment by comparing offline training feature statistics (computed at pipeline compile time and stored in GCS) against live Feature Store serving statistics via TFDV schema checks[cite: 12]. Schema violations block the promotion step and trigger an alert[cite: 12].

TFDV schema checksPost-deploy gateSkew alert

Feature Governance

Feature Lineage & Validation

All features are validated by the Data Governance process (M-08) before any write to the Feature Store[cite: 12]. Lineage tags link each feature to its source system and governance approval record[cite: 12].

SOURCE EXTRACT M-08 VALIDATION FEATURE STORE WRITE TRAINING / SERVING AUDIT LOG

← scroll to view full diagram →

Feature Source System Model(s) Governance Store Status
entity_pair_idSAP / Intercompany ledgerIC AnomalyM-08 · approved✓ VALIDATED
amount_vs_agreement_rateIC Agreement RegisterIC AnomalyM-08 · approved✓ VALIDATED
historical_balance_7dTreasury TMSCash ForecastM-08 · approved✓ VALIDATED
FX_rate_7d_maECB / Bloomberg feedCash ForecastM-08 · approved✓ VALIDATED
IC_sweep_historyTreasury TMSCash ForecastM-08 · approved✓ VALIDATED
po_match_scoreProcurement · ERP PO tableInvoice ClassifierM-08 · approved✓ VALIDATED
supplier_invoice_hashAP module · invoice metadataInvoice ClassifierM-08 · approved✓ VALIDATED
supplier_prior_exception_rate AP history · raw exception log 30-day lagged window from raw AP audit log — not from model output table, to prevent feedback loop[cite: 12]. Invoice Classifier M-08 · approved ✓ VALIDATED
payroll_calendarHR system · payroll scheduleCash ForecastM-08 · approved✓ VALIDATED
prior_mismatch_countHITL override log · BigQueryIC AnomalyM-08 · approved✓ VALIDATED

Training-Serving Skew Monitoring

Training feature statistics are computed at pipeline compile time and stored as TFDV schema patches in GCS alongside each model version[cite: 12]. Post-deployment, these offline statistics are compared against live Feature Store serving statistics on each inference batch[cite: 12]. Schema violations — indicating distributional divergence between training and serving environments — block the canary promotion step and raise a Cloud Monitoring alert for investigation before any further traffic shift[cite: 12].

EU AI Act Article 13 compliance: SHAP TreeExplainer is applied deterministically to every IC Anomaly score above the 0.72 threshold and to every non-APPROVED Invoice Classifier output[cite: 12]. Explanation payloads are stored in the HITL audit trail alongside the original model inputs, prediction confidence, and reviewer decision — satisfying transparency and logging obligations for high-risk automated financial decisions[cite: 12].

Art. 13 Obligation Architectural Component Status
Transparency of operationSHAP TreeExplainer payloads stored in HITL audit trail (BigQuery)✓ MET
Human oversight capabilityHITL queues HITL-IC-01 / HITL-CF-02 / HITL-INV-03 for all flagged decisions✓ MET
Logging of operationsAll model inputs, outputs, confidence scores, and reviewer decisions logged to ae_finance.hitl_audit✓ MET
Intended purpose disclosureSystem cards stored in Vertex Model Registry per acronym, documenting scope, known limitations, and deployment constraints✓ MET
Instructions for use (deployers)Threshold configuration, HITL escalation paths, and override handling documented in runbook stored alongside each Model Registry entry✓ MET