Customer loss is rarely a sudden event. It develops over time - fewer purchases, smaller order values, longer payment cycles, less frequent interaction.
By the time the loss becomes visible, it is often already irreversible. When the company finally recognises that the customer has left, there may be nothing left to retain.
The phenomenon is known as customer attrition, or churn in technical terms: the loss of existing customers, as opposed to the acquisition of new ones.
Most commercial organisations understand the issue and can usually quantify it. They know what proportion of their customer base they lose each year.
What they know far less clearly is which customers are likely to leave.
That is the only question that makes intervention possible, because retention capacity is limited. A company cannot treat every customer in the same way. It must decide where to act - and act early enough to make a difference.
What a model can detect that a rule cannot
To anticipate customer loss, companies usually begin with rules: raise an alert when a customer has not ordered for six months, or monitor accounts that fall below a given threshold.
These rules have a structural weakness: they examine one signal at a time. Churn risk rarely sits in one isolated variable. It emerges from a combination of factors.
In the case presented below, a 55-year-old customer holding a single product and showing no transaction activity has a fundamentally different risk profile from a 35-year-old customer holding several products - even when both have the same account balance.
It is not age alone, product holding alone or inactivity alone that predicts churn. It is the interaction between them.
This is precisely what a machine-learning algorithm can capture, while a manual rule generally sees only averages.
The model learns from the history of customers who have already left and identifies which combinations of signals tended to precede departure - including combinations that no one would necessarily have thought to formalise manually.
The output is not a verdict. It is a probability assigned to each active customer: the likelihood that the customer will leave within the coming months.
That is where the real work begins.
Value does not come from the score
The value of a churn model does not depend solely on its predictive accuracy.
It depends on four additional factors:
• the cost of an alert;
• the economic value of retaining a customer;
• the success rate of the retention action;
• the organisation’s ability to respond.
None of these parameters appears in the technical metrics commonly used to assess a model - ranking ability, detection rate or precision.
Yet these are the parameters that determine whether the model deserves investment.
The analysis below follows the complete chain, from data to business case, including the underlying figures.
The data
The demonstration is based on a widely used public banking dataset: 10,000 retail customers, an observed churn rate of 20.4%, and ten behavioural, contractual and financial variables.
The example is banking-related because robust public datasets are available in that sector. The underlying logic applies to any recurring customer portfolio - industrial companies, distribution businesses or service providers.
Real customer data should not be published.
A public dataset offers the opposite advantage of a confidential case study: every figure in this article can be recalculated and every analytical step reproduced.
The methodology: three datasets, not two
The data was divided into three separate groups:
• 6,000 customers to train the models;
• 2,000 customers to select the model and calibrate the alert threshold;
• 2,000 customers to measure final performance.
This separation is an important marker of analytical discipline, and it comes at the cost of using more data.
A common approach uses only two datasets: training and testing. The risk is immediate. The same test set may then be used to select the model, adjust the threshold and report final performance.
That is the equivalent of sitting an examination with access to the answer sheet.
The final test set in this analysis was used only once, at the very end: to measure how the selected model would perform on customers it had never seen, using a threshold determined without reference to them.
Three models, one economic criterion
Three model families were compared:
• logistic regression, as an interpretable statistical baseline;
• random forest;
• gradient boosting.
These are the terms that technical teams and suppliers will use. Their internal mechanics matter less than the decision principle that follows.
The linear model is not included for formality. When a simple model performs as well as a more complex one, the simpler solution should prevail.
The alert threshold - the probability above which a customer is flagged - is not merely a technical setting.
It is an economic decision.
It depends on the relationship between the value of a lost customer and the cost of a retention action.
Under the assumptions used in this case:
• value of a lost customer: €5,000;
• cost of a personalised retention action: €1,000;
• expected success rate of the action: 40%.
Each threshold produces a different expected economic cost.
That cost, rather than statistical performance alone, was used to select the model.
Once a threshold has been fixed, two measures describe the operating behaviour of the model.
The first is recall, or detection rate: what proportion of actual churners does the model identify?
The second is precision: what proportion of flagged customers actually leave?
These measures move in opposite directions. The broader the search, the more churners the model detects - but the more customers it also contacts unnecessarily.
| Model | Ranking (AUC) | Treshold | Recall | Precision | Customers contacted | Expected Costs |
|---|---|---|---|---|---|---|
| Logistic Regression | 0.834 | 0.80 | 75.2% | 44.7% | 685 | €2.108.000 |
| Random Forest | 0.858 | 0.80 | 54.8% | 67% | 333 | €1.922.000 |
| Gradient Boosting | 0.859 | 0.67 | 57.2% | 70% | 333 | €1.902.000 |
The logistic regression model identifies 75.2% of all churners - the highest detection rate of the three, and exactly the type of number a supplier would place prominently on a presentation slide.
It is nevertheless the worst choice.
More importantly, it costs more than doing nothing.
Doing nothing on this dataset produces an expected cost of €2,035,000 - 407 lost customers at €5,000 each.
The logistic regression model produces an expected cost of €2,108,000.
It destroys €73,000 of value.
The explanation sits in one column: precision of only 44.7%.
To identify 306 genuine churners, the model flags 685 customers. More than half of the retention contacts are wasted, at a cost of €1,000 each.
The gradient boosting model identifies fewer churners, but flags roughly half as many customers and achieves 70% precision. It concentrates the commercial effort where it has the highest probability of creating value.
This is the central point of the analysis.
Once intervention carries a cost, detection rate becomes a vanity metric.
What matters is how many customers are contacted, how many of them were genuinely at risk, and whether the cost of the intervention is justified by the value preserved.
One qualification is important.
The difference between random forest and gradient boosting is limited - one thousandth of AUC and €20,000 of expected cost.
A difference of this magnitude does not prove that one model family is inherently superior to the other. It justifies the choice in this case, but it should not be generalised.
What the selected model actually does
On the final test set - never used for training, model tuning or threshold calibration - the gradient boosting model achieves an AUC of 0.862.
In practical terms, this means the following:
Take one customer who will leave and one customer who will stay. In approximately 86% of cases, the model assigns the higher risk score to the customer who leaves.
A random model would score 50%. A perfect model would score 100%.
AUC therefore measures the model’s ability to rank customers by relative churn risk - and nothing more.
It does not indicate how many customers should be contacted, or whether the intervention will be economically attractive.
Among the 2,000 customers in the final test set, 407 actually churned.
The model flags 333 customers, or approximately 17% of the portfolio.
Of those 333:
• 229 are genuine churners;
• 104 are false alerts.
Nearly 69% of flagged customers genuinely leave.
The model fails to identify 178 churners.
Those 178 missed churners are not necessarily a technical defect to be corrected.
They are the deliberate cost of a selective strategy.
At €1,000 per intervention, attempting to capture every possible churner would cost more than the loss avoided. That is precisely the trap into which the logistic regression model falls.
The purpose of the model is not to detect every departure.
It is to allocate a limited retention budget where it is most likely to produce value.
Under the same assumptions:
• 333 retention actions cost €333,000;
• 229 genuine churners are contacted;
• a 40% success rate retains approximately 92 customers;
• avoided losses amount to approximately €458,000.
The expected net gain is therefore €125,000, against a baseline churn cost of €2,035,000.
That represents a reduction of approximately 6.1%.
The result is modest.
That is also what makes it credible.
A churn model does not eliminate customer attrition. It improves the targeting of a commercial intervention that already exists.
The break-even point
The intervention becomes profitable once the success rate of retention actions exceeds approximately 29%.
The model assumes a 40% success rate.
The resulting margin of safety is therefore eleven percentage points - and it depends on a parameter that very few organisations measure reliably.
The implication should be stated clearly:
If retention actions succeed only one time in four, this model destroys value.
Not because the model predicts poorly - its AUC would remain unchanged - but because the economics of the intervention no longer work.
Technical performance and profitability are independent variables.
The model creates no value on its own
The model identifies customers. It does not retain them.
Between a risk score and a euro of preserved value sits a commercial action.
That action determines the success rate on which the entire business case depends.
Its relevance matters: does the company have a credible proposition for a 55-year-old, inactive, single-product customer?
Its timing matters: an alert processed six weeks late may concern a customer who has already left.
Its quality of execution matters. And there must be a business owner accountable for the outcome, not merely for delivery of the project.
An accurate score passed to a team with no capacity, no appropriate offer and no clear accountability creates nothing - other than another dashboard.
What management must establish before launching
Three numbers are required, and none is technical.
The real economic value of a retained customer
Not annual revenue, but expected margin over the customer’s remaining lifetime.
The full cost of a retention action
Commercial time, potential discounting and the opportunity cost of the team involved.
The actual success rate of retention actions
This is the only one of the three that requires historical measurement - and the one most organisations do not possess.
These three figures determine the alert threshold, the operating behaviour of the model and ultimately its profitability.
They cannot be guessed or transferred from one company to another.
They must be measured on the organisation’s own data.
Change them, and the same model can move from value-creating to value-destroying.
Two additional conditions must also be addressed.
What proportion of alerts can the commercial organisation realistically process?
A model that flags 17% of the portfolio assumes that sufficient capacity exists. If that capacity must be doubled, the cost of intervention changes, and the threshold must change with it.
And:
Who owns the commercial outcome once the score has been produced?
Without a clear answer, even the strongest model remains a file.
What Nobilys does with this analysis
Nobilys does not develop models to demonstrate technical performance.
We help management teams determine whether a model can improve a decision, under what economic conditions, and with what level of execution risk.
This demonstration has one purpose: to show what a decision-grade analysis looks like.
That analysis should take place before investment - using the company’s actual data, not the assumptions of a generic case study.
Christian Cirino- Nobilys Group
Dataset: “Churn Modelling” - 10,000 banking customers, ten variables, publicly available on Kaggle. Models, thresholds and business case are reproducible from the figures published above.