Predictive models for lapse and midterm cancellations for individual health insurance: a machine learning approach

This project aimed to develop and evaluate machine learning models for predicting policy lapse and mid-term cancellation in health insurance. Customer retention is a key challenge for insurers, as policy cancellations can reduce profitability and affect long-term business sustainability. The main objective was therefore to identify policyholders at higher risk of cancellation and provide a framework that supports proactive retention strategies. The dataset consisted of historical health insurance policy information and underwent an extensive preparation process, including data cleaning, feature engineering, handling of class imbalance, and prevention of data leakage through appropriate data partitioning. Several machine learning algorithms were tested, including Logistic Regression, Decision Trees, Random Forest, XGBoost, LightGBM, and Multi-Layer Perceptrons. Hyperparameter tuning was performed using Optuna combined with stratified 5-fold cross-validation to ensure robust model selection. Model performance was assessed using discrimination, calibration, and overall predictive accuracy metrics. In addition, probability calibration techniques were applied to improve the quality of predicted probabilities, making the models more suitable for business decision-making. Explainability methods, particularly SHAP values, were used to interpret model predictions and identify the factors most associated with policy cancellations. The results showed that ensemble-based machine learning models outperformed traditional approaches in predictive performance. Furthermore, probability calibration significantly improved the reliability of risk estimates. Overall, the study demonstrates that advanced machine learning techniques can provide insurers with valuable insights into customer behaviour and support more effective retention strategies, helping reduce policy cancellations and improve portfolio management.