Layered holdout validation for predicting material degradation in nuclear applications

As machine learning is increasingly applied to materials science, a critical question emerges: how do you trust a model on conditions it has never seen before?
Traditional validation methods randomly split data into training and test sets, assuming test conditions will resemble training conditions. For nuclear structural materials, where mechanical and thermal loading conditions evolve over decades, that assumption is inaccurate. The risks affect daily operation, maintenance schedules, and long-term asset health.

Samuel Eka, a PhD researcher at the University of Manchester, developed a layered holdout validation framework to address this gap. Using openly sourced data from Argonne National Laboratory, Samuel first established a baseline using random train-test splits (Level 0) across 12 regression algorithms. Most achieved prediction accuracy above 90%. He then applied the layered holdout framework. Entire categories of data were deliberately withheld from training at three levels:

• Level 1: All specimens of a particular alloy grade
• Level 2: A subset of specimens extracted from a specific structural component
• Level 3: Specimens extracted in a specific orientation from a specific component and material grade (a subset of a subset of a subset)

For each scenario, models were trained on the remaining data and evaluated on these unseen holdout sets. This simulates the real engineering challenge: predicting behaviour for a new material or previously unseen parameter-state.

The question is not whether your model works on the data you have, the question is whether it works on the conditions you haven't tested yet. Standard validation gives you confidence in the past. Layered holdout gives you confidence in the unknown.

Samuel Eka

Under Level 1 holdout, performance dropped significantly compared to baseline. Surprisingly, at Level 2 and Level 3, prediction accuracy improved, sometimes exceeding the random-split baseline. This suggests that training on diverse but relevant data, even with deliberate gaps, can improve generalisation.

Ductility properties (elongation and reduction in area) were much harder to predict. Understandably, many governing features affecting ductility were not recorded in the source data. This highlights the importance of robust data collection strategies for through-life structural integrity assessments.

This work recently passed peer-review and was presented at the Pressure Vessels and Piping Conference (PVP2026), organised by ASME, in July 2026 in California, USA. Through STEM outreach, Samuel has also practised explaining these validation concepts to non-specialist audiences, breaking down how AI reliability testing differs from standard model testing and why that difference matters for nuclear safety.

Equipment Used

• Layered holdout validation framework (custom Python implementation)
• Standard ML libraries (scikit-learn)
• Regression Algorithms: Random Forest, Decision Trees, Extra Trees, Gradient Boosting, Hist-Gradient Boosting, Support Vector Regression, Bagging, AdaBoost, Bayesian Ridge, ElasticNet, Ridge, Lasso
• Dataset of nuclear-grade steel thermal ageing experiments (Argonne National Laboratory, Materials Engineering Associates)
• Bootstrapping for uncertainty quantification
• High-performance computing access via CDT partnership

Biography

Samuel Eka received his undergraduate degree in Mechanical Engineering (1st Class) from the University of Hertfordshire and a Masters in Offshore/Subsea Engineering (Distinction equivalent) from Cranfield University. Samuel has worked in several capacities in the engineering and IT space. He has managed several construction delivery packages in the rail and water sectors and has also functioned as a data scientist, providing useful insights for IT clients. He is currently pursuing a PhD with the Advanced Metallic Systems CDT at the University of Manchester, where he is utilising skills obtained from both engineering and data science domains. His research focuses on datacentric engineering of materials and structures, developing the entire data journey (collection, storage, extraction, and analysis) for through-life structural integrity assessments of safety-critical nuclear applications.