Just cracked the model selection puzzle for my churn dataset

I’ve been wrestling with a churn prediction project for the past month, and the moment finally hit when I found the right combination of preprocessing and algorithm tuning that boosted my AUC from 0.71 to 0.78. I started with a raw dataset of 12,000 rows, each row containing 37 features—customer demographics, usage patterns, and a handful of categorical variables that were heavily imbalanced.

First, I applied a systematic feature engineering routine: one‑hot encoded the categorical variables, created interaction terms for age and subscription length, and used a log transformation for the monthly spend column to reduce skewness. I then split the data into a 70/30 train/test set, preserving the churn rate via stratified sampling.

Next, I ran a quick grid search over several classifiers: logistic regression with L1 and L2 penalties, random forest, gradient boosting, and a simple neural network. I kept the hyperparameters modest at first to avoid overfitting. Surprisingly, the gradient boosting model with 200 trees and a learning rate of 0.05 outperformed the others by a clear margin.

To fine‑tune, I employed Bayesian optimization instead of a brute‑force grid, which cut the search space by 80% and still found a sweet spot: max_depth 5, subsample 0.8, and a minimum child weight of 3. After retraining on the full training set and evaluating on the held‑out test, the AUC rose to 0.78, and the precision‑recall curve looked clean.

The big win is that the model now captures a non‑linear relationship between monthly spend and churn likelihood that the linear baseline missed. I’ve also added a SHAP explanation layer, which shows that high monthly spend combined with low engagement spikes churn risk—something that could inform targeted retention campaigns.

I’m posting here to get feedback on the feature set and the choice of gradient boosting. Are there other techniques you’d recommend for a similar churn scenario? Also, any thoughts on the interpretability aspect—does the SHAP breakdown look convincing to stakeholders?

Thanks in advance for the insights!