A scorecard can post a strong Gini on its development sample and still disappoint in production, which is why credit scorecard validation matters. Often the cause is one of three things: it ranks borrowers well but predicts the wrong default rates, the applicant population has drifted since it was built, or it was only ever tested on the data used to build it.
Credit scorecard validation is how you catch these problems before the portfolio does. In short, you test discrimination (KS, Gini, AUC), calibration, rank ordering and stability (PSI, CSI) on data the model has not seen. Because portfolios change, you also repeat the tests as the book ages.
This guide explains each metric from first principles, so you can read every number with confidence. It also works through all of them in Python on synthetic data.
Credit scorecard validation is the process of testing whether a scorecard does its job. Specifically, it checks that the scorecard ranks borrowers by risk, predicts default rates close to what actually happens, and stays stable as the applicant population changes. It happens at development, in independent review and through ongoing monitoring.
Validation is not a single test run once. Instead, it is a set of checks that answer different questions at different points in a scorecard’s life:
How often each stage runs depends on the bank’s model risk policy, although the materiality of the scorecard also matters. For the governance and regulatory side, see our guide to credit risk validation methods. This article, however, focuses on the statistical tests specific to scorecards.
Discrimination is how well a scorecard separates future defaulters from non-defaulters, whereas calibration is how closely its predicted default rates match observed ones. KS, Gini and AUC, however, measure discrimination only. They depend on the order of scores, so a scorecard can rank perfectly and still predict the wrong default rates.
Here is why. Take any scorecard and double every predicted PD. The order of borrowers does not change, so AUC, Gini and KS stay exactly the same. However, the predicted default rates are now twice as high.
As a result, a scorecard can have an excellent Gini and still be badly wrong about levels. For example, a score band might predict a 2% default rate while 4% of its borrowers actually default. The ranking may be flawless and the Gini unchanged, yet anything built on that 2% is understated.
Calibration matters in three places:
Calibration, on the other hand, is checked by comparing the average predicted default rate with the observed rate in each score band. KS, Gini and AUC answer the ranking question, so credit scorecard validation should always report calibration alongside them. Our scorecard course covers model validation and performance metrics end to end.
The ROC curve plots the share of defaulters a scorecard catches (true positive rate) against the share of non-defaulters it wrongly flags (false positive rate) at every possible cut-off. AUC is the area under that curve. Gini = 2 × AUC − 1 rescales AUC, so 0 means no ranking power and 1 means perfect ranking. AUC and Gini are the most common discrimination measures in credit scorecard validation.
Treat “bad” (default) as the positive class, and flag every account below a chosen score cut-off as a predicted bad. At that cut-off:
For example, with 100 bads and 900 goods in a sample, a cut-off that flags 60 bads and 200 goods gives a true positive rate of 60% and a false positive rate of 22.2%. Moving the cut-off from the lowest score to the highest traces the curve.
A scorecard with no ranking power follows the diagonal, while a better one bows toward the top-left corner. AUC also has a plain-language meaning: it is the probability that a randomly chosen bad account scores as riskier than a randomly chosen good one, with ties counted as half. Therefore, an AUC of 0.75 means the bad account is ranked riskier in 75% of such pairs.
The relationship is exact: Gini = 2 × AUC − 1. As a result, an AUC of 0.50 gives a Gini of 0, an AUC of 0.70 gives 0.40, an AUC of 0.75 gives 0.50, and an AUC of 1.00 gives 1.
Three practical points apply:
The KS statistic is the maximum vertical gap between the cumulative distribution of bad accounts and the cumulative distribution of good accounts when both are ordered by score. Therefore, a larger KS means the scorecard separates the two groups better. It runs from 0 to 1, or from 0 to 100 when expressed as a percentage. KS is the other core discrimination measure in credit scorecard validation.
The example below is illustrative, so none of its figures are real portfolio data: it uses 10,000 accounts, 500 bads (a 5.0% overall bad rate) and 9,500 goods, split into deciles of 1,000 accounts each. The chart shows the bad rate in each decile on the left and the cumulative curves on the right.
By decile 4, the scorecard has captured 80.4% of bads but only 37.9% of goods, so the largest gap is 42.5 points and KS ≈ 42.5%. By common retail rules of thumb, that is solid, although benchmarks differ by product and market.
The decile where the gap peaks also tells you something practical. Here, the riskiest 40% of accounts contain about 80% of the bads but only about 38% of the goods. That is a useful reference point for setting a cut-off. However, a business cut-off must also weigh approval rate, profitability and risk appetite, so the KS peak is not automatically the right place to cut.
A decile table also tests rank ordering: the bad rate should fall at every step from the riskiest decile to the safest. In the chart above it does, falling from 17.0% in decile 1 to 0.7% in decile 10. If the bad rate rises in a later decile, the scorecard is mis-ranking part of the population, even when KS looks healthy.
Lift shows how concentrated the risk is. For example, decile 1 has a 17.0% bad rate against 5.0% overall, which is a lift of 3.4×. Lift is simply a decile’s bad rate divided by the overall bad rate.
One limitation remains: KS looks at a single point on the curve, whereas Gini uses the whole curve. As a result, two scorecards can share a KS and still differ in shape, which is why the metrics are reported together.
In credit scorecard validation, KS, Gini and AUC test ranking power on outcomes, so they tell you how well a scorecard separates goods from bads. PSI and CSI, on the other hand, test stability by showing whether the population being scored has changed. None of them tests calibration, so report predicted against observed default rates alongside them.
| Metric | What it measures | Use it when | Main limitation |
| KS | Largest gap between cumulative bad and good distributions | You want a quick read of separation or a region for a cut-off | Looks at one point only; depends on binning; ignores calibration |
| AUC | Chance a random bad scores riskier than a random good | You compare models on one number across all cut-offs | Averages over cut-offs you may never use; ignores calibration |
| Gini | AUC rescaled so 0 is random and 1 is perfect | You report ranking power in the usual industry shorthand | Same limits as AUC; scale varies (0–1 or 0–100) |
| PSI | Shift in a distribution, usually the score, between two samples | You monitor whether the scored population has moved; needs no outcomes | Does not show whether performance worsened; sensitive to binning |
| CSI | The same shift measure for each input variable | You need to find which variable drove a PSI move | Same as PSI; many variables to track |
The Population Stability Index (PSI) compares the score distribution today with the distribution at development, while the Characteristic Stability Index (CSI) applies the same formula to each input variable. Generally, a PSI below 0.10 is read as stable, 0.10–0.25 as a shift worth investigating, and above 0.25 as a significant shift. Stability checks complete credit scorecard validation by showing whether the scored population still resembles the development population.
PSI = Σ (Actual%ᵢ − Expected%ᵢ) × ln(Actual%ᵢ / Expected%ᵢ)
Here, Expected% is the share of the development population in bin i, whereas Actual% is the share of the current population in the same bin. To compute it:
The 0.10 and 0.25 cut-offs are conventions associated with Siddiqi’s work on credit scorecards, as summarised in a survey of R packages for credit scoring. They are rules of thumb rather than statistical tests.
Four points are worth keeping in mind:
Split-sample validation tests the scorecard on a random holdout from the development window, while out-of-time validation tests it on a different period, which is closer to real use. Bootstrapping, in addition, resamples the data many times to put a confidence interval around Gini or KS. Strong credit scorecard validation therefore uses more than one.
Fit the scorecard on one part of the data and test it on the rest, for example with a 70/30 split. This catches overfitting. However, both parts come from the same period, so a holdout cannot show whether the scorecard survives a change in the economy or the applicant mix.
An out-of-time sample contains observations from a period outside the development window, so it is the most realistic stability test. Because the bad definition needs time to mature, the accounts must have been observed long enough, and the right window depends on the product, the default definition and data volume.
Bootstrapping draws many samples with replacement and recomputes Gini or KS each time, so you can report a range instead of a single value. It is especially useful when bads are few. Moreover, it shows whether a gap between two samples, or between two scorecards, is real or just noise.
For this reason, judge results against intervals rather than single values, since a Gini that moves by a point or two may well be within sampling noise.
This example of credit scorecard validation in Python builds a simple logistic scorecard on synthetic applicants and then runs the tests above on a holdout sample and an out-of-time sample. Because the data is simulated, no real bank data is used. Moreover, the out-of-time sample deliberately includes a drifted population, so you can see what each metric catches and what it misses.
The full script simulates 40,000 development applicants and 15,000 out-of-time applicants. Specifically, the out-of-time group has lower average income, higher utilisation and a higher base default rate. Although it is simple, a logistic regression on three inputs (income, utilisation and past delinquencies) is fitted on 70% of the development data and scored on the rest.
Gini comes from AUC and KS comes from the ROC curve, while the decile table and PSI are built by hand.
import numpy as np
import pandas as pd
from sklearn.metrics import roc_auc_score, roc_curve
def gini(y, p):
return 2 * metrics.roc_auc_score(y, p) – 1
def ks(y, p):
fpr, tpr, _ = metrics.roc_curve(y, p)
return np.max(tpr – fpr)
def decile_table(y, p):
df = pd.DataFrame({“y”: y.values, “p”: p.values})
df[“decile”] = pd.qcut(df[“p”].rank(method=”first”, ascending=False),
10, labels=range(1, 11))
t = df.groupby(“decile”, observed=True).agg(
accounts=(“y”, “size”), bads=(“y”, “sum”), avg_pd=(“p”, “mean”))
goods = t[“accounts”] – t[“bads”]
rate = t[“bads”] / t[“accounts”]
cum_bad = t[“bads”].cumsum() / t[“bads”].sum()
cum_good = goods.cumsum() / goods.sum()
return t.assign(goods=goods, bad_rate=rate, cum_bad=cum_bad,
cum_good=cum_good, gap=cum_bad – cum_good)
def psi(expected, actual, bins=10):
edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1)))
edges[0], edges[-1] = -np.inf, np.inf
e = np.clip(np.histogram(expected, edges)[0] / len(expected), 1e-4, None)
a = np.clip(np.histogram(actual, edges)[0] / len(actual), 1e-4, None)
return float(np.sum((a – e) * np.log(a / e)))
# train, test, oot and FEATURES come from Step 1 (see the full script)
for name, d in [(“Test”, test), (“OOT”, oot)]:
print(name, “Gini”, round(gini(d[“default”], d[“pd_hat”]), 3),
“KS”, round(ks(d[“default”], d[“pd_hat”]), 3))
print(decile_table(test[“default”], test[“pd_hat”]).round(3))
print(“Score PSI:”, round(psi(train[“pd_hat”], oot[“pd_hat”]), 3))
for f in FEATURES:
print(“CSI”, f, round(psi(train[f], oot[f]), 3))
Running the full script gives the results below, and they are specific to this simulation.
The pattern matches the one described earlier: ranking is intact, but levels are off. A Gini-only review would therefore have passed this scorecard, whereas the calibration check would have flagged it. That points to recalibration, which the next section covers.
If you want to practise this workflow with instructor feedback, the Advanced Certificate in Credit Scorecard Modeling covers scorecard validation hands-on in Python.
Recalibrate when the scorecard still ranks risk well but its predicted default rates have drifted, because you only need to adjust the mapping from score to PD. Redevelop when ranking power has fallen, the data or definitions have changed, or the population has moved enough that the original variables and bins no longer fit.
Credit scorecard validation ends with this decision, and what counts as “materially” below is set by your model risk policy, usually through tolerance bands and escalation triggers, so there is no universal threshold. Moreover, recalibration cannot repair weak ranking. Any recalibrated or redeveloped scorecard also goes back through validation and governance approval.
Most validation failures come down to six avoidable mistakes:
Credit scorecard validation comes down to four questions. First, does the scorecard rank risk (KS, Gini, AUC, decile table)? Second, are its predicted default rates right (calibration)? Third, is the population it scores stable (PSI, CSI)? Finally, does it hold up on data it has not seen, including out-of-time data? Answer all four, repeat them as the book ages, and let the answers decide whether to keep, recalibrate or redevelop.
For the governance side, read our guide to credit risk validation methods. Likewise, if you are weighing scorecards against rating models for a portfolio, see credit rating model vs credit scoring.
To learn credit scorecard validation under instructor guidance, explore the Advanced Certificate in Credit Scorecard Modeling. The course is live and instructor-led, and it also uses hands-on work in Python. In addition, it covers KS, Gini, ROC curves, lift charts, performance monitoring and when to redevelop versus recalibrate.
Gini = 2 × AUC − 1. To get it, score each account, treat default as the positive class, compute the area under the ROC curve (AUC), and rescale it. In Python, 2 * roc_auc_score(y, pd_hat) – 1 does it. However, check whether your report uses a 0-1 or 0-100 scale before comparing results.
There is no universal cut-off, but a KS above about 30% is commonly treated as acceptable for retail scorecards, and 40% or more as strong. Benchmarks vary by product, data and market, so compare against your own history. An unusually high KS, moreover, should prompt a check for target leakage.
Application scorecards commonly show a Gini of roughly 0.40 to 0.60, which is equivalent to an AUC of 0.70 to 0.80. Behavioural scorecards often score higher because they use repayment history. Treat these as rules of thumb, and be wary of very high values, since they often signal leakage.
By convention, a PSI below 0.10 indicates a stable population, 0.10 to 0.25 suggests a shift worth investigating, and above 0.25 signals a significant shift. These are rules of thumb rather than statistical tests, and results depend on how the score is binned. Therefore, read PSI alongside performance, never alone.
Yes. Gini measures ranking only, so a scorecard can order borrowers correctly while predicting default rates that are too low or too high. It can also look strong on development data but weaken out of time. For this reason, credit scorecard validation should check calibration by score band and use a later sample.
Redevelop when ranking power has fallen materially, rank ordering breaks, inputs or the default definition have changed, or the population has shifted so far that the original variables no longer fit. Recalibrate when ranking holds but predicted default rates are consistently off. Your model risk policy, however, sets the tolerances.
A small script shows the mechanics. Production ECL work needs PD, LGD and EAD development, scenario design and validation done properly. If you want structured, instructor-led training on this, see DexLab’s Advanced Certificate in IFRS 9 Modeling
.
Credit and Market Risk, Credit Risk, credit risk analysis, Credit Risk Analytics And Modeling, credit risk analytics training, Credit risk certification India, Credit risk modeling course India, Credit Risk Modelling, Credit Risk Modelling Using SAS, Credit scorecard modeling, Expected credit loss ECL, IFRS 9 modeling course India, Logistic regression scorecard, PD LGD EAD modeling, Python credit risk modeling
Comments are closed here.