| dc.description.abstract | In this work, we present effective algorithms to automatically annotate and parse clothes from social media data, such as Facebook and Instagram. Clothing annotation can be informally stated as recognizing, as accurately as possible, each garment item that appears in a photo.Clothing parsing, in turn, locates and annotate each garment item in a photo. These tasks play important roles in several areas, including surveillance, action recognition, person search, recommender systems and e-commerce. They also pose interesting challenges for existing vision and recognition algorithms, such as distinguishing between similar but conceptually different types of clothes or identifying a pattern of a specific item, since it can have different colors, shapes, textures and appearance. Initially, the clothing annotation problem was analyzed considering statistical methods of machine learning. For this purpose, we perform an extensive evaluation of the visual feature extraction techniques, including global and local descriptors. Then, we formulate the annotation task as a multi-label and multi-modal classification problem (i) both image and textual content (i.e., tags related to the image) are available for learning classifiers, (ii) the classifiers must predict a set of labels (i.e., a set of garment items), and (iii) the decision on which labels to assign to the query photo comes from instances (or {\em bag} of instances) that are used to build a function, which separates labels that should be assigned to the query photo, from those that should not be assigned. Using this configuration, we propose two approaches: (i) the pointwise one, called MMCA, which uses a single image as input to the classifiers, and (ii) a multi-instance classification, called M3CA, also known as pairwise approach, that uses pair of images as input to the classifiers. We compare both approaches in order to define the best one for the problem. For both of them, we propose a classification algorithm that employs association rules in order to build a recognition model that combines textual and visual information. We also adopt an entropy-minimization strategy in order to find the best set of labels that should be assigned to the query photo. We conduct a systematic evaluation of the proposed algorithms using everyday photos collected from two major fashion-related social media, namely \url{pose.com} and \url{chictopia.com}. Our results show that the proposed approaches provide improvements when compared to popular first choice multi-label, multi-modal, multi-instance algorithms that range from 20\% to 30\% in terms of accuracy. In a second phase, we analyzed the clothing parsing problem using deep learning. We propose a multi-scale convolutional neural network model. Specifically, we use different network levels where each level processes images with different dimensions, i.e., after every level the images are decomposed into smaller patches, allowing the network to capture minimal details. In the first level, bigger images are processed in a robust network. Images with low entropy already get their final class in this level, while the others with high entropy (classification still undefined) are splitted into smaller patches and go to the next one. In the third and last level, images without final classification in the second level are again divided into even smaller patches and, finally, classified.At the end, we have a class associated with each patch of the image and we can recompose it. To evaluate this approach, we use a dataset crawled from \url{chictopia.com}. Our experiments shows that our proposed approach achieves promising results. | |
