Artículos de revistas
A Rank Aggregation Framework For Video Multimodal Geocoding
Registro en:
Multimedia Tools And Applications. Kluwer Academic Publishers, v. 73, n. 3, p. 1323 - 1359, 2014.
13807501
10.1007/s11042-013-1588-4
2-s2.0-84912032077
Autor
Li L.T.
Pedronette D.C.G.
Almeida J.
Penatti O.A.B.
Calumby R.T.
Torres R.S.
Institución
Resumen
Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) This paper proposes a rank aggregation framework for video multimodal geocoding. Textual and visual descriptions associated with videos are used to define ranked lists. These ranked lists are later combined, and the resulting ranked list is used to define appropriate locations for videos. An architecture that implements the proposed framework is designed. In this architecture, there are specific modules for each modality (e.g, textual and visual) that can be developed and evolved independently. Another component is a data fusion module responsible for combining seamlessly the ranked lists defined for each modality. We have validated the proposed framework in the context of the MediaEval 2012 Placing Task, whose objective is to automatically assign geographical coordinates to videos. Obtained results show how our multimodal approach improves the geocoding results when compared to methods that rely on a single modality (either textual or visual descriptors). We also show that the proposed multimodal approach yields comparable results to the best submissions to the Placing Task in 2012 using no extra information besides the available development/training data. Another contribution of this work is related to the proposal of a new effectiveness evaluation measure. The proposed measure is based on distance scores that summarize how effective a designed/tested approach is, considering its overall result for a test dataset. 73 3 1323 1359 2009/10554-8; FAPESP; Conselho Nacional de Desenvolvimento Científico e Tecnológico; 2011/11171-5; FAPESP; Conselho Nacional de Desenvolvimento Científico e Tecnológico; 306580/2012-8; CNPq; Conselho Nacional de Desenvolvimento Científico e Tecnológico; 484254/2012-0; CNPq; Conselho Nacional de Desenvolvimento Científico e Tecnológico Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) Almeida, J., Leite, N.J., Torres R da, S., Comparison of video sequences with histograms of motion patterns. In: International conference on image processing (2011) pp 3673–3676 Andrade, F.S.P., Almeida, J., Pedrini, H., Torres R da, S., Fusion of local and global descriptors for content-based image and video retrieval. In: Iberoamerican congress on pattern recognition (CIARP’S) (2012) pp 845–853 Boureau, Y.L., Bach, F., LeCun, Y., Ponce, J., Learning mid-level features for recognition. In: Conference on computer vision and pattern recognition, pp 2559–2566 (2010) doi:10.1109/CVPR.2010.5539963 Candeias, R., Martins, B., Associating relevant photos to georeferenced textual documents through rank aggregation. In: Terra Cognita 2011 workshop (2011) In conjunction with 10th international semantic web conference Choi, J., Ekambaram, V.N., Friedland, G., ICSI/Berkeley video location estimation system. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927 (2012) CEUR-WS.org Choi, J., Lei, H., ICSI video location estimation system. In: Working notes proceedings of the MediaEval workshop (2011) vol 807 Clinchant, S., Ah-Pine, J., Csurka G (2011) Semantic combination of textual and visual information in multimedia retrieval In: International conference on multimedia retrieval, pp 44, 1-44, p. 8 Coppersmith, D., Fleischer, L.K., Rurda, A., Ordering by weighted number of wins gives a good ranking for weighted tournaments (2010) ACM Trans Algorithm, 6 (3), pp. 55:1-55:13 Cormack, G.V., Clarke, C.L.A., Buettcher, S., Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In: ACM SIGIR conference on research and development in information retrieval (2009) pp 758–759 Croft, W.B., Croft, W.B., Croft, W.B., Combining approaches to information retrieval (2002) Advances in information retrieval, the information retrieval, vol 7. Springer US, pp. 1-36 Ding, D., Zhang, B., Probabilistic model supported rank aggregation for the semantic concept detection in video. In: Proceedings of the 6th ACM international Conference on Image and Video Retrieval, CIVR ’07, pp 587–594 (2007) doi:10.1145/1282280.1282364.http://doi.acm.org/10.1145/1282280.1282364, , http://doi.acm.org/10.1145/1282280.1282364 Faria, F.A., Veloso, A., de Almeida, H.M., Valle, E., Torres R da, S., Gonçalves, M.A., Jr, W.M., Learning to rank for content-based image retrieval. In: International conference on multimedia information retrieval (2010) pp 285–294 Fishburn, P.C., (1988) Nonlinear preference and utility theory/Peter C. Fishburn, , Johns Hopkins University Press, Baltimore: Fox, E.A., Shaw, J.A., Combination of multiple searches (1994) Text REtrieval Conference (TREC-2), 500-215, pp. 243-252 Friendly, M., Corrgrams: exploratory displays for correlation matrices (2002) Am Stat, 56 (4), pp. 316-324 Hauff, C., Houben, G.J., WISTUD at MediaEval 2011: placing task. In: Working notes proceedings of the MediaEval workshop (2011) vol 807 Hays, J., Efros, A.A., (2008) im2gps: estimating geographic information from a single image, , In, Conference on computer vision and pattern recognition: Jones, C.B., Purves, R.S., Geographical information retrieval (2008) Int J Geogr Inf Sci, 22 (3), pp. 219-228 Kalantidis, Y., Tolias, G., Avrithis, Y., Phinikettos, M., Spyrou, E., Mylonas, P., Kollias, S., Viral: visual image retrieval and localization (2011) Multimed Tools Appl, 51, pp. 555-592 Kelm, P., Schmiedeke, S., Sikora, T., A hierarchical, multi-modal approach for placing videos on the map using millions of flickr photographs. In: Workshop on Social and Behavioural Networked Media Access, SBNMA ’11 (2011) pp 15–20 Kelm, P., Schmiedeke, S., Sikora, T., Multi-modal, multi-resource methods for placing Flickr videos on the map (2011) In: International conference on multimedia retrieval Kelm, P., Schmiedeke, S., Sikora, T., How spatial segmentation improves the multimodal geo-tagging. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927 (2012) CEUR-WS.org Kelm, P., Schmiedeke, S., Sikora, T., Multimodal geo-tagging in social media websites using hierarchical spatial segmentation (2012) LBSN ’12, pp. 32-39. , ACM, New York, NY: 10.1145/2442796.2442805 Khudyak, K.A., Kurland, O., Cluster-based fusion of retrieved lists. In: Proceedings of the 34th international ACM SIGIR conference on research and development in information retrieval, SIGIR ’11 (2011) pp 893–902 Klementiev, A., Roth, D., Small, K., A framework for unsupervised rank aggregation. In: Proc. of the ACM SIGIR conference (SIGIR) workshop on learning to rank for information retrieval (2008) pp 32–39, , http://cogcomp.cs.illinois.edu/papers/KlementievRoSm08a.pdf Kludas, J., Bruno, E., Marchand-Maillet, S., Information fusion in multimedia information retrieval (2008) Adaptive multimedial retrieval: retrieval, user, and semantics, pp. 147-159. , Boujemaa N, Detyniecki M, Nürnberger A, (eds), Springer, New York: Kokar, M.M., Tomasik, J.A., Weyman, J., Formalizing classes of information fusion systems (2004) Inform Fusion, 5 (3), pp. 189-202 Laere, O.V., Schockaert, S., Dhoedt, B., Ghent university at the 2011 placing task. In: Working notes proceedings of the MediaEval workshop (2011) vol 807 Laere, O.V., Schockaert, S., Quinn, J.A., Langbein, F.C., Dhoedt, B., Ghent and cardiff university at the 2012 placing task. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927 (2012) CEUR-WS.org Larson, M., Soleymani, M., Serdyukov, P., Rudinac, S., Wartena, C., Murdock, V., Friedland, G., Ordelman, R., Jones GJF (2011) Automatic tagging and geotagging in video collections and communities In: International conference on multimedia retrieval, pp 51, 1-51, p. 8 Larson, R.R., Geographic information retrieval and digital libraries (2009) European conference on research and advanced technology for digital libraries, 5714, pp. 461-464 Li, L.T., Almeida, J., Pedronette, D.C.G., Penatti, O.A.B., Torres R da, S., A multimodal approach for video geocoding. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927 (2012) CEUR-WS.org Li, L.T., Almeida, J., da, T.R., In: Working notes proceedings of the MediaEval workshop (2011) vol 807 Li, L.T., Pedronette, D.C.G., Almeida, J., Penatti, O.A.B., Calumby, R.T., Torres R da, S., Multimedia multimodal geocoding. In: ACM SIGSPATIAL international conference on advances in geographic information systems (2012) pp 474–477 Li, X., Hauff, C., Larson, M., Hanjalic, A., Preliminary exploration of the use of geographical information for content-based geo-tagging of social video. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927 (2012) CEUR-WS.org Luo, J., Joshi, D., Yu, J., Gallagher, A., Geotagging in multimedia and computer vision–a survey (2011) Multimed Tools Appl, 51, pp. 187-211 Manning, C.D., Raghavan, P., Schtze, H., (2008) Introduction to information retrieval, , Cambridge University Press, New York, NY: Montague, M., Aslam, J.A., Condorcet fusion for improved retrieval. In: Proceedings of the 11th international Conference on Information and Knowledge Management, CIKM ’02, pp 538–548 (2002) doi:10.1145/584792.584881.http://doi.acm.org/10.1145/584792.584881, , http://doi.acm.org/10.1145/584792.584881 Olligschlaeger, A.M., Hauptmann, A.G., Multimodal information systems and GIS: the informedia digital video library (1999) In: 1999 ESRI user conference, , http://www.informedia.cs.cmu.edu/documents/ESRI99.html Pedronette, D.C.G., Exploiting contextual information for image re-ranking and rank aggregation in image retrieval tasks. Ph.D. thesis (2012) University of Campinas (UNICAMP), , Campinas, SP: Brazil Pedronette, D.C.G., Torres RdS Exploiting clustering approaches for image re-ranking (2011) J Vis Lang Comput, 22 (6), pp. 453-466 Pedronette, D.C.G., Torres R da, S., Calumby, R.T., Using contextual spaces for image re-ranking and rank aggregation (2012) Multimed Tools Appl, pp. 1-28 Penatti, O.A.B., Li, L.T., Almeida, J., Torres R da, S., A visual approach for video geocoding using bag-of-scenes (2012) In: International conference on multimedia retrieval Poh, N., Bengio, S., How do correlation and variance of base-experts affect fusion in biometric authentication tasks? (2005) IEEE Trans Signal Proces, 53 (11), pp. 4384-4396 Popescu, A., Ballas, N., CEA LIST’s participation at mediaeval 2012 placing task. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927 (2012) CEUR-WS.org Rae, A., In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927 (2012) CEUR-WS.org Schalekamp, F., Zuylen, A., Rank aggregation: together were strong. In: Workshop on Algorithm Engineering and Experiments (ALENEX) (1998) pp 38–51 Sculley, D., Rank aggregation for similar items (2007) SIAM international conference on Data Mining (SDM, 2007, pp. 587-592 Serdyukov, P., Murdock, V., van Zwol, R., Placing flickr photos on a map. In: ACM SIGIR, pp 484–491 (2009) doi:10.1145/1571941.1572025 Trevisiol, M., Delhumeau, J., Jégou, H., Gravier, G., How INRIA/IRISA identifies geographic location of a video. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927 (2012) CEUR-WS.org Trevisiol, M., Jégou, H., Delhumeau, J., Gravier, G., Retrieving geo-location of videos with a divide & conquer hierarchical multimodal approach (2013) In: International conference on multimedia retrieval van Gemert, J.C., Veenman, C.J., Smeulders, A.W.M., Geusebroek, J.M., Visual word ambiguity (2010) IEEE Trans Pattern Anal Mach Intell, 32, pp. 1271-1283 Van Laere, O., Schockaert, S., Dhoedt, B., Finding locations of flickr resources using language models and similarity search. In: International conference on multimedia retrieval, pp 48:1–48:8 (2011) doi:10.1145/1991996.1992044 Young, H.P., An axiomatization of borda’s rule (1974) J Econ Theory, 9 (1), pp. 43-52 Zhang, H., Jiang, L., Su, J., Augmenting naive bayes for ranking. In: International conference on machine learning (2005) pp 1020–1027 Zhou, X., Depeursinge, A., Müller, H., Information fusion for combining visual and textual image retrieval in imageclef@icpr (2010) Proceedings of the 20th International Conference on Recognizing Patterns in signals, speech, images, and videos, ICPR ’10, pp. 129-137. , Springer-Verlag, Berlin, Heidelberg: