Unveiling the Hidden Gems: Exploring Unexpected Rare Pattern Mining in Data

Pattern mining is the task of finding statistically relevant patterns in data that can provide valuable insights and knowledge. However, most existing pattern mining methods use a single threshold to determine the frequency of the patterns, which may not reflect the diversity and specificity of the data items. This may lead to two problems: (1) if the threshold is too low, it may generate too many patterns, many of which are redundant or uninteresting; (2) if the threshold is too high, it may miss some patterns, especially the rare ones that occur infrequently but have high significance or utility.

The rare pattern problem is a challenging and important issue in pattern mining, as rare patterns may represent unknown or hidden knowledge that can inform and inspire various domains and applications, such as medical diagnosis, fraud detection, or anomaly detection. Several studies have attempted to address this problem by mining frequent patterns, including rare ones, using different minimum item support thresholds (MIS) for each item. This approach can generate a complete set of frequent patterns without losing any significant ones. However, this approach is also very costly and inefficient, as it may still produce many redundant or useless patterns that consume a lot of time and memory.

The primary objective of this project is to enhance an efficient and effective method for mining rare patterns, without generating the complete set of frequent patterns. The method is based on frequent closed itemset mining, which is a technique that can reduce the number of patterns by eliminating those that are included in other patterns with the same frequency. The method also aims to avoid generating a large number of rules, and instead, to discover only those rules that are rare and generate more actionable insights. Therefore, the method can mine only the most interesting patterns, which are those that are rare, closed, and have high utility or significance. The method can be applied to various data sets and domains, such as health data, where rare patterns may represent rare diseases, hidden connections, or complex interactions. The project aims to evaluate the performance and quality of the method, and to compare it with other existing methods for rare pattern mining. The project also aims to demonstrate the usefulness and impact of the method, and to show how it can discover novel and intriguing patterns that can drive meaningful change.