Fuzzy masks for correlation matrix pruning

Abstract

Among statistical methods used in data analysis processes, correlation analysis holds one of the most significant places. Using correlation, analysts measure prediction potential between values of a pair of attributes of a dataset which can be summarized into a correlation matrix for any multidimensional dataset. Such matrices are usually sizable, and hard to read. Pruning of the correlation matrix for the precise selection of attribute pairs of a considered dataset which bear strong prediction potential is conventionally conducted via correlation matrix masks. Since these masks are commonly designed as crisp borders for the acceptability of values in the matrix, there is a strong lack of nuance in this approach. The work presented in the scope of this study focuses on the design and implementation of fuzzy masks for correlation matrices. This objective is divided into three main tasks - firstly, the visualization of correlation coefficient value frequency and its relationship to fuzzy membership function is designed and implemented, then visualisation of basic regression analysis and correlation context of attribute pairs in the fuzzy area of a studied dataset is designed and implemented, and lastly, the proposed approach is evaluated via case studies on three benchmarking datasets. The results obtained with the fuzzy approach to correlation matrix masks show a qualitative improvement in the matrix pruning task and a more appropriate identification of the relevant parts of the dataset compared to the conventional, crips approach.