Binning or Feature Selection: Which One Won Us an Insurance Hackathon?

Binning or Feature Selection: Which One Won Us an Insurance Hackathon? | Towards Data Science

Machine Learning

Binning or Feature Selection: Which One Won Us an Insurance Hackathon?

For a year, we believed that supervised binning carried our winning entry to an insurance competition. After four seeds per arm and one shared feature set, the credit belongs elsewhere.

Mario Defrance

October 30, 2026

18 min read

Prefer us on Google

Share

Image by the author, from the ENS Challenge Data #161 data (Etalab Licence Ouverte 2.0)

Zacharie Buisson co-built the winning solution with me. Every number below is measured, and every notebook is stored with the outputs that produced it.

Disclosure: I am the author and maintainer of AutoCarver, an open-source (MIT) library. Read the library sections with that in mind. Every number can be checked from the repository and datasets linked below. All images are by the author, generated from the challenge data (Etalab Licence Ouverte 2.0).

···

The finish line, first

In 2025, we took first place in an insurance hackathon. A year later, we audited that result, and the audit reversed the belief underneath it. Binning did beat not binning. However, our own library's bins were indistinguishable from a rival's. The only gain that was not a close call was speed: 126 minutes of carving went down to 4.3, on the same features.

Tschuprow's T with the target, on held-out data: 0.0240 across the sixteen raw levels, 0.0358 across the two buckets, a 1.49× rise.

The figure shows one real feature from the challenge data. The policies are anonymised French farms, priced against fire. Floor area comes in sixteen bands, and claim frequency climbs roughly tenfold across them. Supervised binning cuts these bands in two, at 1000 m². Finer options existed, since ten of the bands clear the 2% min_freq floor. However, the carver's association measure ranked this cut above all of them ( tools/make_hero_chart.py --explain ). The lower panel explains the choice. Every band from 4500 m² to 7000 m² covers under 2% of the portfolio, so the most dramatic rates rest on a handful of policies. Without supervised binning, a pipeline would keep all sixteen bands, or merge them by hand. We carved 400 features this way, and we believed it was half of why we won.

This approach took us to first place in the Crédit Agricole Assurances Data Science Academy hackathon [5], out of more than 500 participants. The hackathon was ranked on the private leaderboard of the public ENS Challenge Data #161 [1], as it stood when the hackathon closed in spring 2025. The challenge is still open, so the live leaderboard has moved on since. For this reason, no number in this article is a leaderboard score. The challenge also served as the running example in the Data Science course we taught at CY Tech.

We re-ran the pipeline with the current version of the library to check two things: whether it was faster, and whether it was more accurate. Speed was easy to measure. Accuracy took three experiments. Every year-on-year accuracy gain turned out smaller than the noise in our own tuning, and the edge we had credited to carving came from which features the selector kept. To show this, we ran four seeds per arm, measured every margin against its own spread, and held one feature set constant to separate two steps we had always run together.

1. The problem

For a farm, a single fire can destroy barns, machinery and stock, and this risk is growing. The IPCC assesses that fire hazard conditions rose across Europe from 1980 to 2019, with substantive increases in Western and Central Europe ( high confidence ). It also assesses that wildfire risk can rise in every European region between 1.5 °C and 3 °C of warming ( medium to high confidence ) [6]. Insurers see the same trend. Swiss Re puts 2025 insured natural-catastrophe losses at USD 107 bn, and calls wildfire the fastest-growing line, at near 12% a year [7].

In France, Crédit Agricole Assurances covers this risk for farms through Pacifica, its property-and-casualty carrier, which writes the Contrat Multirisque Agricole . Challenge #161, called AssurPrime, asks participants to price the fire cover of this contract from anonymised policy data. Fire is a large share of the contract's claims [1]. Our work uses a single snapshot of this data and no temporal analysis, so none of the climate trends above is a finding of our model.

The challenge scores CHARGE , the total claim cost of a policy. Insurers do not predict such a cost in one step. As actuaries have done for decades, they factor it:

E

[

cost

]

=

number of claims

⏟

frequency

×

amount per claim

severity

\mathbb{E}[\text{cost}] = \underbrace{\mathbb{E}[\text{number of claims}]}_{\text{frequency}} \times \underbrace{\mathbb{E}[\text{amount per claim}]}_{\text{severity}}

​

CHARGE is exactly this product, scaled by the policy's years of exposure, and the challenge data provides both factors. Two things make the problem harder than it looks. First, claims are rare. Almost every policy has zero claims, so a model that predicts zero for everyone looks accurate but is useless. Second, the two targets behave very differently. Frequency is a small count, while severity is a heavy-tailed amount, and a single model for both would blur each of them. We therefore built two models.

2. The winning recipe (2025)

Our 2025 solution used neither a giant model nor an ensemble of ensembles. It rested on three decisions. First, we factored the problem into frequency × severity, as actuaries do. Second, we binned every feature against the target with AutoCarver , the open-source library named in the disclosure [2]. Third, we weighted the rare signal. The audit in this article only tests the second decision.

2.1 Carving the features first

Besides the anonymised policy features, we joined a wildfire-exposure profile for each geographic zone, built from BDIFF, France's national forest-fire database [3]: burnt area, fire counts by cause, number of fire stations and wind zone (23,462 fires recorded over 2016–2023).

Raw insurance features are messy: high-cardinality categoricals, ordinal levels with tiny populations, skewed numerics, and NaN s that carry meaning. Given such features, a tree model spends its depth rediscovering structure that could have been handed to it.

Supervised binning provides that structure. For each feature, AutoCarver searches for the admissible grouping with the strongest statistical association with the target. A grouping is admissible under three constraints: a minimum bin frequency, a maximum number of bins per feature, and validation on a held-out dev set. The last constraint matters most, since a grouping that only works on the training data is rejected. In practice, the feature types are declared once and everything is carved in a single fit :

python

# AutoCarver 7.0.5, as written in 2025. The current API appears later.
from AutoCarver import Features, MulticlassCarver
features = Features(
categoricals=["property_type", "region", ...],
quantitatives=["insured_value", "surface", ...],
ordinals={"quality_band": ["low", "mid", "high", "premium"]},
)
carver = MulticlassCarver(features=features, min_freq=0.02, max_n_mod=5)
train_carved = carver.fit_transform(train, train["n_claims"], X_dev=dev, y_dev=dev["n_claims"])
carver.summary  # every bucket: frequency, target rate, association

Note that this snippet still runs today, but it does something else. In 7.0.5, MulticlassCarver carved one-vs-rest, so every raw feature came back as several columns. That behaviour is now called OneVsRestCarver .

The obvious alternative is optbinning [4]. Where AutoCarver searches groupings heuristically, optbinning states the merge of CART prebins as a CP/MILP problem and solves it exactly, within a time limit. We chose differently for three reasons, which concern our setting rather than the library itself. First, we wanted every grouping tested on a held-out sample, and fit(x, y) has no notion of one. Second, our target is ordered, and MulticlassOptimalBinning does not treat it as such. Third, its multiclass path handles numerical features only, while most of our 435 features are categorical or declared ordinals. At the time, these were only reasons. We measure their consequences further below.

With binning, ordinals were merged in their natural order, and one pipeline handled all feature types. Missing values are never silently imputed: with dropna=True , AutoCarver carves NaN as its own bucket. We chose dropna=False instead, and let XGBoost route them natively. After carving, ClassificationSelector pre-selected features by association and pruned redundant ones in one pass.

2.2 Frequency: a weighted multiclass model

Almost no policy has more than two claims, so we treated frequency as a multiclass problem over 0 / 1 / 2+ . Inverse-frequency class weights stopped the model from collapsing onto the majority class. The law of total expectation then turns the class probabilities back into an expected frequency:

Y

P

(

0

)

∣

+

1

2

\mathbb{E}[Y] = P(0)\,\mathbb{E}[Y \mid 0] + P(1)\,\mathbb{E}[Y \mid 1] + P(2{+})\,\mathbb{E}[Y \mid 2{+}]

Every log loss in this article is the class-weighted one, with those same inverse-frequency weights, since this is what the search minimised. For reference, a constant base-rate model scores 0.041 unweighted and 4.47 weighted. A uniform guess of ⅓ per class scores 1.099. Our models score around 0.92.

2.3 Severity and tuning

The severity model only sees policies with claims. The trick we credit most is to weight each observation by the frequency model's absolute error. The severity model then concentrates on the policies where the first stage was wrong. This works as a cheap, boosting-like correction across the two stages. We measured its effect during the competition, on the public leaderboard, but that comparison is not reproduced in this article. Both models are XGBoost, tuned with Optuna. One idea from the tuning is worth reusing: the number of pruned features was itself a hyper-parameter.

3. What a year of releases actually bought

We won with AutoCarver 7.0.5 . A year later, we re-ran both pipelines on the current version, with the same data, the same train/dev split and the same XGBoost search budget. Every step is seeded, so the 2026 numbers below can be reproduced exactly (see Setup).

The Setup section lists what else changed between the two versions. The main point is that only the 2026 side is seeded, so the speed comparisons are clean, but the metric comparisons are not.

3.1 Multiprocessing: the slow step got fast

Carving the 680 MB of training data was the slowest step of our 2025 loop.

# AutoCarver 7.7.3
from AutoCarver import OrdinalCarver
from AutoCarver.discretizers import ProcessingConfig
carver = OrdinalCarver(features=features, min_freq=0.02, max_n_mod=5, config=ProcessingConfig(n_jobs=6))

The table gives the carving wall-clock time on the full dataset (383,610 rows, ~435 qualitative features, one machine, runs serialised):

Pipeline

Carver

7.0.5 (1 process)

today (6 workers)

speed-up

Frequency, qualitative

MulticlassCarver → OrdinalCarver

2842 s (47.4 min)

120 s (2.0 min)

23.7×

Frequency, quantitative

MulticlassCarver

383 s (6.4 min)

not carved — left to the selector

—

Frequency, total

3226 s (53.8 min)

26.9×

ContinuousCarver

4725 s (78.7 min)

140 s (2.3 min)

33.8×

Version 7.0.5 ran in a single process, while today's carve ran on 6 workers, so perfect scaling alone gives 6×. The remaining factor, ~4× on the frequency carve and ~5.6× on severity, comes from the dynamic-programming top-k search that replaced brute-force enumeration. The 2026 severity carve also ran at a lower min_freq (0.02 against 0.03), so 33.8× is conservative. Unlike the accuracy results later on, these gains are not close calls. The same carve has come back at 120.0, 126.8 and 129.4 s on this machine, a spread of 7.8%, nowhere near the 24× gap.

3.2 An ordinal target

Our claim-count target is not only multiclass. It is ordered ( 0 < 1 < 2+ ), and the 2025 MulticlassCarver ignored that order. OrdinalCarver optimises the bins against an ordinal target using Kendall's tau-c. We scored the carvings on a common yardstick, tau-c on the dev set. OrdinalCarver matches the association per feature of the 2025 one-vs-rest geometry, with 2.07× fewer columns and 2.05× fewer buckets, in a carve 3.8× faster. We also compared it with the unordered multiclass carving, which differs only in whether the order is used. The two return identical bucketings on 210 of 364 features, and the ordinal one wins 111 to 43 on the rest (two-sided sign test, p ≈ 4×10⁻⁸). The direction is reliable, but the size is small: +0.000107 in mean tau-c. Declaring ordinal features is therefore worth the effort. However, none of these carvings was carried through selection and tuning, so we claim no effect on the final metric.

3.3 Did the new version win by more?

The table below compares the two pipelines end to end.

Metric

2025 (7.0.5)

2026 (current)

verdict

Frequency — dev log loss

0.9194

0.9199

indistinguishable (seed spread 18× the gap)

Severity — dev RMSE

6613.8

6469.7

indistinguishable (the 2.18% gap is 1.01× the spread)

CHARGE — dev RMSE (the challenge metric)

6639.9

6482.8

indistinguishable (the 2.37% gap is 1.01× the spread)

The frequency row of this table is not a result, and it took a dedicated experiment to find this out. The two frequency numbers differ by 0.00055. The 2026 number comes from a seeded search that reproduces to sixteen significant figures, so it is tempting to read the gap as real. To check it, we re-ran the identical feature sets under four TPE seeds, changing nothing else:

Seed

Frequency log loss

Severity RMSE

CHARGE RMSE

42 (headline)

0.9246

6612.9

6638.1

7

0.9207

6591.5

6615.0

2026

0.9145

6608.9

6633.2

Across the four seeds, the frequency log loss spreads over 0.0101. This is eighteen times the gap we were about to interpret. Three seeds land above the 2025 baseline and one lands below it, so the sign of the comparison depends on the seed. Fixing a seed makes a run reproducible, but it does not make it representative. The two are easy to confuse, since the same seed returns the same number every time. With a single seeded run, the margin we act on is one draw from a distribution whose spread was never measured. In our case, we nearly claimed a year of progress that was not there.

Severity varies even more. Across the same seeds, dev RMSE spans 143.2, which is 2.16% of the 2025 figure. CHARGE spans 155.2 (2.34%). The 2.18% and 2.37% margins are therefore only 1.01× their own noise. In addition, seed 42, used for the headline table, is the best of the four on both metrics. The other three land within 0.4% of the 2025 baseline. Averaged over the four seeds, the margins fall to 0.65% and 0.72%. Both comparisons are draws, and our headline run is the luckiest of the four.

To sum up, nothing added by a year of releases improved accuracy here. This leaves an older and larger question, which we took for granted a year ago and never tested: did the binning help at all?

3.4 Three ways to bin

Earlier, we gave reasons for carving and for not using optbinning. To test them, we ran three arms of the same pipeline end to end, on both models, with four seeds each. Only the binning step changes between arms. The third arm carves nothing. Its ordinals are rank-encoded from their declared orderings, its categoricals are integer-coded on the training data, and its numericals are left untouched. This is the baseline a practitioner would build without a binning library ( tools/ablation_matrix.py ).

One of our reasons for not using optbinning, that its multiclass path only handles numerical features, turned out to be a hard limit. Given our three-class target (0, 1 and 2+ claims) and our categorical features, optbinning's BinningProcess fails outright:

markdown

ValueError: MulticlassOptimalBinning does not support categorical variables.

Since 435 of the 555 frequency columns are categorical or declared ordinals, this affects most of the dataset. The optbinning frequency arm therefore bins each feature against a binary target instead: whether a policy has at least one claim ( y > 0 ). This is optbinning's best-supported path for mixed feature types. Neither binner touches the numericals, which the pipeline leaves to the selector.

In the left panel, each arm carves, selects its own 100 features, then tunes. Every pair separates with no overlap: AutoCarver beats no carving by 2.45× the pooled seed spread, no carving beats optbinning by 2.56×, and AutoCarver beats optbinning by 3.50×. In the right panel, all three arms model the same 100 features, so only the binning differs. AutoCarver scores the same in both panels, since the shared list is its own selection. On this shared set, we find two things.

First, binning earned its place. Both binners beat the unbinned arm by 1.59× the pooled spread, with no overlap, and each wins on all four seeds individually. We had believed this for a year without testing it, and it holds.

Second, the two libraries are indistinguishable as binners. On identical features, the gap between them is 0.0063 of log loss, which is 0.45× their pooled spread, and their ranges overlap. Equalising the selection moves the 3.50× pipeline win by 0.05 of log loss and reverses its sign. What separated the libraries here was therefore not how they cut the features, but which features their selectors kept. This shared setting also favours AutoCarver. The 100 features are its own selection, and sixteen of them are ordinal bands, whose levels have a natural order. AutoCarver and the unbinned arm use that order, while optbinning treats these bands as unordered categories. Even with this advantage, AutoCarver did not win.

We gave optbinning two more chances on its own selection, and neither changes the picture. Rank-encoding the 238 ordinal bands, so that it can use the order as its documentation prescribes, helps by 0.48× the spread (0.9660 → 0.9571). Capping it at three bins, to match AutoCarver's realised width, makes it worse (0.9707).

Severity needs a different metric to show any difference. On the same 200 features, dev RMSE puts AutoCarver at 6570.7, no carving at 6580.7 and optbinning at 6612.8. No pair is more than 0.89× its pooled spread apart. A constant predictor, the training mean, scores 6617.6, within 47 of every one of them.

Top-decile lift shows more. It is the mean claim among the 10% of policies a model ranks highest, divided by the overall mean, so a model with no signal scores 1.00. On this metric, optbinning clears the unbinned arm by 2.41× the pooled spread. AutoCarver is also above the unbinned arm, but only by 0.74×, which is inside the noise. The two binners draw at 0.44×. The ordering is the same as for frequency, and RMSE could not see it.

RMSE is blind here because it rewards overfitting on this target. On the shared features, the binned models score about the same on train and dev. One unbinned model, however, reached a train RMSE three times lower than its dev RMSE (1994 against 6530). This memorising model posted the best dev RMSE of its arm, while its lift stayed low, at 2.37.

The mechanism shows in the tree depths the tuner chose. Without binning, the search chose depth 2 on three seeds and depth 9 on the fourth. With binning, it stayed at depth 1 on seven of eight seeds. Deep trees fit the amounts and lose the ordering. Binning therefore acted as a regulariser. On a heavy-tailed target, this trade suits anyone who needs to know which policies are expensive, rather than how expensive they are. It is a tendency, and it does not always hold: the one binned model that went to depth 4 saw its lift fall to 2.17.

This comparison has three limits. First, it covers one dataset and one downstream pipeline. Second, the arms only select the same features when forced to, which is why we report both comparisons. Third, every severity arm consumes the same frequency hand-off, so the severity comparison isolates the severity binner and nothing else.

4. Takeaways

• Factor the problem. Each half gets a model suited to its target. We compared this with a single model during the competition, on the public leaderboard, but that comparison is not reproduced here.

• Bin the features, then check which step paid off. Binning beat the unbinned arm by 1.59× the seed spread, but the gap between the pipelines was seven times the gap between the binners. Our pipeline won on feature selection.

• Check that the metric can see the difference. Dev RMSE could not separate any severity arm from predicting the mean, and it preferred a model that overfit threefold. Top-decile lift separated optbinning from no carving at 2.4× the seed spread.

• Compare with the simplest baseline first. For a year, 6613.8 dev RMSE looked like a working severity model. The training mean scores 6617.6.

• Measure the seed spread before reading a margin. Four extra runs per model turned two apparent wins into draws. It is the check we would run first next time.

Everything behind this article

• The code, with every notebook stored with the outputs that produced these numbers: github.com/mdefrance/caa-challenge

• The frequency and severity notebooks on Kaggle, to run without installing anything, at a reduced tuning budget and against a mirror of the data

• The saved 2025 models , to re-score the 2025 baseline

• AutoCarver: docs · source

Setup: exactly what was measured

The 2025-against-2026 comparison comes from four pipeline runs on one machine, on 2026-08-11. The seed sweeps and the binning comparison ran in September, on the same machine and with the same budgets, from the scripts in tools/ .

CPU / RAM

Intel Coffee Lake, 12 logical cores · 32 GB

GPU

GTX 1650 Max-Q (XGBoost runs device="cuda" )

3.11.7

2025 arm

AutoCarver 7.0.5, scikit-learn 1.9.0, numpy 2.0.2, single-process

2026 arm

AutoCarver 7.7.3, scikit-learn 1.8.0, numpy 2.4.6, n_jobs=6 , Optuna seeded ( TPESampler(seed=42) )

XGBoost / Optuna

3.2.0 / 4.9.0; 300 trials (frequency), 400 (severity), identical between eras

Data

ENS Challenge Data #161 [1] plus a BDIFF fire-history extract [3], both Etalab Licence Ouverte 2.0; the exact extract is mirrored on Kaggle

Figures

All four are ours, generated from that data by tools/make_hero_chart.py and tools/make_result_charts.py ; no third-party images

Beyond the carver, the 2026 pipeline differs in its selection measures, its severity min_freq (0.02 against 0.03) and a narrower candidate set. None of this affects the wall-clock comparison. The 400 tuning trials were scored on the dev set, so every dev score is optimistic rather than held out. Severity and CHARGE RMSE are unweighted. No leaderboard figure is quoted in this article.

The data is not in the repository: the challenge files come from the ENS platform or the Kaggle mirror, which also holds the exact BDIFF extract. With them, every 2026 run and every comparison in section 3.4 can be re-run from the repository. On our machine, with a CUDA GPU, the seeded runs reproduce exactly. On a CPU, XGBoost runs the same pipeline but does not match the last digits. The 2025 search was unseeded, so re-running the 2025 notebooks ( requirements-705.txt ) does not give the same numbers. Instead, the saved 2025 models are published as a Kaggle dataset , and tools/replay_2025_charge.py re-scores them without re-fitting anything. It reproduces the 2025 severity figure to 3×10⁻⁶ and gives the 2025 CHARGE baseline. The 2025 frequency figure comes from the stored output of the 2025 notebook.

References

[1] Crédit Agricole Assurances, AssurPrime : Saurez-vous prédire la prime d'assurance ? (2025), ENS Challenge Data #161

[2] M. Defrance, AutoCarver (2026), PyPI / GitHub

[3] Institut national de l'information géographique et forestière, Base de Données sur les Incendies de Forêts en France (BDIFF) (2025), Ministère de l'Agriculture et de la Souveraineté Alimentaire

[4] G. Navas-Palencia, OptBinning: The Python Optimal Binning library (2025), v0.21.0, GitHub

[5] M. Couillaud, Retour sur le hackathon de la Data Science Academy (2025), LinkedIn

[6] IPCC (GIEC), Climate Change 2022: Impacts, Adaptation and Vulnerability , ch. 13 “Europe” (2022), Sixth Assessment Report, Working Group II

[7] Swiss Re Institute, sigma 1/2026: Natural catastrophes in 2025 — the persistent rise of wildfire and storm risk (2026), sigma research

With thanks to Crédit Agricole Assurances and ENS Challenge Data for an excellent public competition, to the CY Tech students who worked through it with us, and to Zacharie Buisson for building the winning solution with me.