Hypervolume rankings decouple from unbiased predictive performance in multi-objective wavelength selection for NIR spectroscopy of Wolffia globosa
Article Sidebar
Main Article Content
Abstract
Multi-objective metaheuristics are widely used to select informative wavelengths in near-infrared (NIR) spectroscopy, and their quality is conventionally assessed with Pareto-based indicators such as hypervolume. Such indicators assume deterministic objectives, yet one objective here, the cross-validated prediction error, is an estimator carrying non-negligible variance, and the consequences of treating it as fixed remain largely unquantified. We benchmark four binary multi-objective metaheuristics (NSGA-II, MO-BPSO, MO-BDE, MO-bGWO) and the CARS selector on a 50-sample, 474-wavelength NIR protein calibration problem in Wolffia globosa, a small-sample regime where bias is most severe, under matched budgets, a shared selection operator and an identical fitness function, with the cardinality prior controlled separately. Under conventional flat cross-validation the algorithms separate decisively (Friedman chi-square = 52.75, p = 2.1e-11) and hypervolume ranks NSGA-II first. Under nested cross-validation the same comparison is not significant (chi-square = 5.85, p = 0.119) and the rank order changes. Three controlled experiments identify the mechanism. A budget sweep shows optimistic bias rising monotonically with search effort (21.9% to 40.7% from 270 to 3030 evaluations) while honest error does not improve. A label-permutation null shows the strongest searcher recovering apparent signal from destroyed labels (RMSECV 7.96 against a no-skill value of 8.69), whereas weaker searchers do not. A landscape analysis finds 386 solutions within 10% of the optimum with median pairwise Jaccard 0.056, including disjoint pairs of equivalent fitness. This degeneracy is an engineering asset rather than a defect, since many channel sets reach the same accuracy and the filter set of a low-cost instrument can be chosen on cost or manufacturability grounds. Hypervolume computed on estimated objectives therefore separates algorithms along a dimension uncorrelated with unbiased performance, search intensity converts into selection bias, and the low selection stability routinely reported reflects structural non-identifiability rather than algorithmic deficiency. A minimum reporting protocol is proposed.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
Zou X, Zhao J, Povey MJW, Holmes M, Mao H. Variables selection methods in near-infrared spectroscopy. Analytica Chimica Acta. 2010;667(1-2):14-32. https://doi.org/10.1016/j.aca.2010.03.048.
Yun YH, Li HD, Deng BC, Cao DS. An overview of variable selection methods in multivariate analysis of near-infrared spectra. Trends in Analytical Chemistry. 2019;113:102-15. https://doi.org/10.1016/j.trac.2019.01.018.
Zitzler E, Thiele L. Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Computation. 1999;3(4):257-71. https://doi.org/10.1109/4235.797969.
Ambroise C, McLachlan GJ. Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences. 2002;99(10):6562-6. https://doi.org/10.1073/pnas.102102699.
Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics. 2006;7:91. https://doi.org/10.1186/1471-2105-7-91.
Cawley GC, Talbot NLC. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research. 2010;11:2079-107.
Kuncheva LI. A stability index for feature selection. In: Proceedings of the 25th IASTED International Multi-Conference: Artificial Intelligence and Applications; 2007 Feb 12-14; Innsbruck, Austria. Anaheim: ACTA Press; 2007. p. 390-5.
Sorensen K. Metaheuristics-the metaphor exposed. International Transactions in Operational Research. 2015;22(1):3-18. https://doi.org/10.1111/itor.12001.
Barnes RJ, Dhanoa MS, Lister SJ. Standard normal variate transformation and de-trending of near-infrared diffuse reflectance spectra. Applied Spectroscopy. 1989;43(5):772-7. https://doi.org/10.1366/0003702894202201.
Wold S, Sjostrom M, Eriksson L. PLS-regression: a basic tool of chemometrics. Chemometrics and Intelligent Laboratory Systems. 2001;58(2):109-30. https://doi.org/10.1016/S0169-7439(01)00155-1.
Deb K, Pratap A, Agarwal S, Meyarivan T. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation. 2002;6(2):182-97. https://doi.org/10.1109/4235.996017.
Kennedy J, Eberhart R. Particle swarm optimization. In: Proceedings of the IEEE International Conference on Neural Networks; 1995 Nov 27-Dec 1; Perth, Australia. Piscataway: IEEE; 1995. p. 1942-8. https://doi.org/10.1109/ICNN.1995.488968.
Kennedy J, Eberhart RC. A discrete binary version of the particle swarm algorithm. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics; 1997 Oct 12-15; Orlando, USA. Piscataway: IEEE; 1997. p. 4104-8. https://doi.org/10.1109/ICSMC.1997.637339.
Storn R, Price K. Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization. 1997;11(4):341-59. https://doi.org/10.1023/A:1008202821328.
Mirjalili S, Mirjalili SM, Lewis A. Grey Wolf Optimizer. Advances in Engineering Software. 2014;69:46-61. https://doi.org/10.1016/j.advengsoft.2013.12.007.
Emary E, Zawbaa HM, Hassanien AE. Binary grey wolf optimization approaches for feature selection. Neurocomputing. 2016;172:371-81. https://doi.org/10.1016/j.neucom.2015.06.083.
Li H, Liang Y, Xu Q, Cao D. Key wavelengths screening using competitive adaptive reweighted sampling method for multivariate calibration. Analytica Chimica Acta. 2009;648(1):77-84. https://doi.org/10.1016/j.aca.2009.06.046.
Drucker H, Burges CJC, Kaufman L, Smola A, Vapnik V. Support vector regression machines. In: Advances in Neural Information Processing Systems 9. Cambridge: MIT Press; 1997. p. 155-61.
Derrac J, Garcia S, Molina D, Herrera F. A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation. 2011;1(1):3-18. https://doi.org/10.1016/j.swevo.2011.02.002.
Ein-Dor L, Kela I, Getz G, Givol D, Domany E. Outcome signature genes in breast cancer: is there a unique set? Bioinformatics. 2005;21(2):171-8. https://doi.org/10.1093/bioinformatics/bth469.