What this model is, and what it is not

Labels are synthetic. The generator’s causal structure is documented in data/GENERATOR.md and its code is in data/generate.py. Absolute metric values therefore reflect the generator, not the Indian market. What the metrics do establish is that the pipeline — leak-free time-based splitting, out-of-fold target encoding, probability calibration, and cost-optimal thresholding — is sound and reproducible against a known ground truth. Point it at a merchant’s real order log and the pipeline runs unchanged.

1. 01The dominant cause is unobservable. Each customer has a latent behavioural archetype — reliable 78%, flaky 15%, serial returner 6%, ring member 1% — which sets most of their RTO hazard and is never written to disk. The model sees only noisy consequences of it: return history, whether a phone was verified, how carefully an address was typed.
2. 02The label is stochastic. It is drawn Bernoulli(sigmoid(hazard)), and then 3% of labels are flipped. Two orders with byte-identical features get different outcomes, and there is no probability in the file for a model to regress onto.
3. 03Nothing separates the classes on its own. The strongest single feature reaches 0.690 ROC-AUC alone. The generator refuses to write a dataset where any feature exceeds 0.85.
4. 04The test period is deliberately harder. Cash-on-delivery share and return-ring activity rise from month 10, so the refusal rate climbs across the three splits. A shuffled split would have made them identical and the evaluation would measure nothing.
5. 05There is a ceiling, and it is published. An oracle handed the true hazard — including the latent archetype and the latent pincode difficulty — scores ROC-AUC 0.810 and is still wrong on 15.1% of orders. Any reported AUC near 1.0 on this data would be evidence of a leak, not of skill.

1. 01Order-time features only. The contract is 25 columns, all knowable when the customer taps Place order. Delivery attempts, courier status, NDR codes and refund records are not excluded from the feature matrix — they are never generated at all, so there is no post-shipment column available to include by accident.
2. 02Out-of-fold target encoding. The per-pincode refusal rate is computed inside five stratified folds on the training block only, so no row contributes to its own encoding. Smoothing is Bayesian toward each fold’s own prior: a pincode needs roughly 50 orders before its own rate outweighs the global one, which stops three-order pincodes from inventing signal. At serving time the encoder is the one fitted on the full training block and frozen; an unseen pincode falls back to the prior.
3. 03One label-to-feature path, and it is named. A customer’s prior refusal count and return rate are built from their earlier outcomes. That is legitimate — a merchant genuinely knows this phone number has refused parcels before — but it is the one place a realised label touches a feature, so it lives in its own function with strict-inequality accumulation: a row enters its customer’s running totals only after its own feature values are written. On a first order both are null, not zero. A new customer is not a customer with a perfect record.
4. 04The order id can never enter the matrix. Order ids are monotone in time, so with the drift in the last three months an id would leak the base-rate shift directly. Features are selected by an explicit contract list rather than by dropping known-bad columns, so this cannot happen by omission.
5. 05One feature implementation, shared. ml/features.py is imported by training, evaluation and the API. There is no second copy of the feature logic inside the service, which is the usual way a demo scores differently from the notebook.

1. 01The labels are synthetic, so the absolute numbers describe a generator we wrote. A PR-AUC of 0.64 is a fact about a data-generating process, not about Indian e-commerce. Nobody should quote any figure in this project as real-world performance, and no merchant should size a budget from the rupee saving above. What the numbers establish is that the pipeline is leak-free and reproducible against a ground truth we control.
2. 02The shipped model is not measurably better than a calibrated logistic regression. Their PR-AUC and Brier scores are within noise of each other in the table above. The generator’s hazard is a logistic function of a linear combination of the drivers, so the linear model is nearly correctly specified and gradient boosting has almost nothing left to find. Shipping LightGBM is a bet that real order data is more non-linear than this generator — a defensible bet, but these metrics do not support it. On this evidence a merchant could ship the logistic regression and lose nothing.
3. 03Three of the economics constants are invented, and the whole cost model rests on them. The abandon rate and the contribution margin came from the brief, not from a measurement, and the two nudge response rates are ours. The optimal threshold is a function of the ratio between the two error costs, so if a merchant’s real abandon rate is 0.15 rather than 0.35, every threshold on this site is wrong for them. The economics page exists so that can be checked rather than assumed, but a slider is not a measurement.
4. 04There is no feedback loop, and in production there would be. Blocking cash on delivery changes customer behaviour; the orders the model refuses never generate an outcome to learn from; the model’s own decisions poison its next training set. The generator holds each customer’s archetype fixed for twelve months and models no response to being scored. A model deployed under this design would decay in a way nothing here measures.
5. 05Serving trusts the caller to compute the velocity features the way training defined them. Orders from this phone in seven days, orders into this pincode in 24 hours, and distinct addresses for this phone all arrive in the request. Training computed them in a forward pass over strictly earlier orders, with the address count inclusive of the current order. A merchant who counts differently gets train/serve skew that neither the API nor the metrics can detect. Sharing the feature module protects the transformation; it cannot protect the inputs.
6. 06No fairness analysis has been done, and the model has an obvious axis for one. Pincode is among the strongest signals available to it, and in India geography correlates with income and community. This model will demand prepayment more often in tier-3 pincodes — that is what the pincode encoding is. Whether that distributional effect is acceptable, and to whom, is not a question the metrics answer and not one this project asked.
7. 07The pincode encoding is frozen at training time. It is fitted on the training months and never updates. A pincode whose delivery difficulty changes — a new courier, a new warehouse — carries a stale rate until someone retrains. New pincodes get the global prior, which is the safest wrong answer rather than a right one.
8. 08The test set is one draw of two months. One generator seed, two months, twenty-four thousand orders. The bootstrap intervals resample rows, not time periods, so they describe sampling noise within those two months and say nothing about how much the answer would move in the next two. Two months is not a stability claim.
9. 09The rules-baseline comparison is generous in one direction and harsh in another. Generous: average precision over a binary flag is a floor, as the baselines section says. Harsh: the rules baseline was not tuned at all — its three constants came from the brief, and a merchant would have tuned them against their own data. A tuned rules engine would close some of the gap.
10. 10The decision log does not survive a restart. The API runs on a free plan with an ephemeral filesystem, so the log shows recent activity, not history. It is a demo affordance, not an audit trail. A real deployment would put that table somewhere that persists.

• Any decision about a person rather than an order: creditworthiness, employment, insurance, or a permanent block on a customer.
• Deployment as-is on real orders. This model learned a synthetic generator's structure and must be retrained on the merchant's own order log before its scores mean anything. The pipeline transfers; the artifacts do not.
• Any use where a refusal cannot be appealed. The recommended action is to require prepayment, which leaves the customer a way to complete the order. It is not a cancellation and should not be implemented as one.
• Sharing scores between merchants, or building a cross-merchant blocklist. The features are one merchant's view of one customer relationship.

This system scores orders a merchant has already received, in order to decide whether to extend COD credit. It contains no capability to generate fraudulent orders, no adversarial or evasion tooling, no probing of external payment systems, and no third-party data scraping. All data is synthetic and contains no real personal information. The explanation endpoint returns feature contributions for the merchant’s own decision audit; it does not expose decision boundaries in a form usable to construct evasive orders. Model artifacts and thresholds are server-side only and are never shipped to the browser.

cd api
uv sync
uv run python -m ml.pipeline    # generate → train → evaluate