Similarity sets: a new concept of sets to seamlessly handle similarity in database management systems

Abstract

Besides integers and short character strings, current applications require storing photos, videos, genetic sequences, multidimensional time series, large texts and several other types of “complex” data elements. Comparing them by equality is a fundamental concept for the Relational Model, the data model currently employed in most Database Management Systems. However, evaluating the equality of complex elements is usually senseless, since exact match almost never happens. Instead, it is much more significant to evaluate their similarity. Consequently, the set theoretical concept that a set cannot include the same element twice also be comes senseless for data “sets” that must be handled by similarity, suggesting that a new, equivalent definition must take place for them. Targeting this problem, this paper introduces the new concept of “similarity sets”, or SimSets for short. Specifically, our main contributions are: (i) we highlight the central properties of SimSets; (ii) we develop the basic algorithm required to create SimSets from metric datasets, ensuring that they always meet the required properties; (iii) we formally define unary and binary operations over SimSets; and (iv) we develop an efficient algorithm to perform these operations. To validate our proposals, the SimSet concept and the tools developed for them were employed over both real world and synthetic datasets. We report results on the adequacy, stability, speed and scalability of our algorithms with regard to increasing data sizes.