Efficient software development effort estimation approaches for improving scalability in the training phase

Abstract

Effective software effort estimation is essential for project management, but faces scalability challenges with large datasets. While clustering can address this complexity, standard methods often rely on random initial centers, leading to inconsistent and less precise results. This randomness frequently overlooks critical contextual factors, such as industry or domain-specific characteristics, which can impair cluster quality and the accuracy of effort predictions. To overcome these issues, this study introduces the Contextual Initial Cluster Centroids (CICC), a novel methodology designed to optimize initial centroid selection. Unlike approaches that depend on randomness or are computationally intensive, CICC uses parallel processing of Jaccard similarity and a refined neighbor-finding technique, K-Reciprocal Nearest Neighbors (KRNN), to identify the most relevant and similar projects as initial centers. This deterministic approach ensures clusters are built around meaningful, context-rich representatives, reducing computations and improving scalability. Experiments on public software project datasets show that CICC significantly outperforms existing techniques. It achieves higher cluster quality, measured by metrics like the Global Silhouette Index, and provides more accurate effort estimates, indicated by lower MAE and higher PRED values. By establishing a more robust and efficient foundation for effort estimation, CICC offers considerable potential to optimize project planning and resource allocation in large-scale software development.