Semantic classification & clustering

Dr Krishna Chandramouli	Virginia Fernandez Arguedas	Tijana Janjusevic
Giuseppe Passino	Dr Tomas Piatrik	Dr Qianni Zhang

Summary

Multimedia Information Retrieval has been an active research area for more than few decades. To a large extent, the challenge could be attributed to the presence of a “Semantic Gap”, which is defined as the gap between the outcome of automatic analysis of multimedia data, and concepts associated by the users to similar items as presented in Figure 1.

This research challenge has been confronted, here in the Multimedia and Vision Research Group (MMV), from different directions:

• clustering and classification of multimedia content using biologically-inspired techniques;
• semantic segmentation of images aimed at studying the inner category layouts;
• (object-based) image retrieval using relevance feedback;
• intelligent visualisation techniques for efficient navigation of large multimedia databases.

Clustering and Classification of Multimedia Content Using Biologically-Inspired Techniques

Dr Krishna Chandramouli

Dr Tomas Piatrik

Clustering and classification problems have been investigated for more than four decades. However, the performance of classical methods has been constrained by the implicit optimisation techniques while optimising distances between clusters. In order to improve the performance of traditional clustering and classification algorithms, biologically inspired techniques have been studied. In particular Ant Colony Optimisation and Particle Swarm Optimisation techniques have been used for clustering and classification respectively.

In order to obtain meaningful clusters, it is imperative to use multiple features such as those defined by MPEG–7. However these descriptors are not naturally compatible with each other when defining similarity between images, and therefore this research tackles the problem of multi-feature fusion using Ant Colony Optimisation (ACO) and its learning mechanism. The proposed algorithm aims at optimising the performance of different low-level features and metric spaces acting over the same image set.

Despite performance improvements achieved by Evolutionary Computation algorithms for classification, such solutions are still far away from solutions generated by human cognition in real-world. As the fundamental evolution of human cognition could be largely attributed to the social interaction of human species, Particle Swarm Optimisation models these interactions as a function in an optimisation problem. In this research aspect, the Particle Swarm Optimisation technique has been used to train a supervised classifier used for categorising images into semantic classes.

Publications

Conferences

• T. Piatrik and E. Izquierdo, "Subspace Clustering of Images using Ant Colony Optimisation", in Proc. of International Conference on Image Processing, 2009,
• E. Dumont, B. Merialdo, S. Essid, W. Bailer, H. Rehatschek, D. Byrne, H. Bredin, N. O'Connor, G. Jones, A. F. Smeaton, M. Haller, A. Krutz, T. Sikora, and T. Piatrik, "Rushes Video Summarization Using a Collaborative Approach," in Proc. TRECVID BBC Rushes Summarization Workshop at ACM Multimedia, 2008,

Book chapters

• K. Chandramouli, E. Izquierdo, "Image Retrieval Using Particle Swarm Optimisation", in book Advances in Semantic Media Adaptation and Personlisation edited by M. C. Angelides, P. Mylonas, M. Wallace, pp. 297-319, 2009

Semantic Segmentation of Images Aimed at Studying the Inner Category Layouts

Giuseppe Passino

The ultimate strategy to associate semantics with multimedia data has to rely on the analysis of what is represented in the data, towards its understanding. Therefore, this research focuses on building models to study and analyse the structure of an image in terms of part relationships. In particular the research addresses issues related to associating high-level semantic labels to low-level image patches. The approach exploits structural data, commonly defined as information associated to the co-presence and relative location of patches in an image. In the ongoing research development, a framework based on probabilistic graphical models to build a learning paradigm to infer relevant semantic cues depicted within a collection of images has been developed.

Publications

Journals

• G. Passino, I. Patras, and E. Izquierdo, "Aspect Coherence for Graph-Based Semantic Image Labelling," IET Computer Vision, to appear.

Conferences

• G. Passino, I. Patras, and E. Izquierdo, "Latent Semantics Local Distribution for CRF-based Image Semantic Segmentation," in Proc. British Machine Vision Conference, 2009

(Object - based) Image Retrieval using Relevance Feedback

Virginia Fernandez Arguedas

Dr Qianni Zhang

In addition to the presence of semantic gap information retrieval systems suffer from “human creativity” while users define and search through multimedia data. In order to model the human creativity, several approaches have been developed based on merging multiple features to define a visual model of a query. Additionally, relevance feedback (RF) techniques have been used for capturing user interest while performing content retrieval. The first aspect of the research focuses on developing a multi-objective learning mechanism for extracting key visual patterns of objects, which are then used to model user requirements while performing retrieval.

In addition, this research area focuses on object retrieval from CCTV cameras, which has been deployed in almost all the major cities of the world in recent times. In an effort to bridge the semantic gap, this research approaches the problem from a top-down perspective, i.e., using Ontologies to model the security domain and thereby detect objects such as vehicles, human, animals etc. The research methodology uses Ontologies to model and to extract previously defined high-level concepts.

Publications

Journals

• Q. Zhang and E. Izquierdo, "Combining Low-Level Features for Semantic Inference in Image Retrieval," Journal on Advances in Signal Processing, 2007.
• Q. Zhang and E. Izquierdo, "Adaptive Salient Block Based Image Retrieval in Multi-Feature Space," Signal Processing: Image Communication, vol. 22, iss. 6, pp. 591–603, 2007.

Conferences

• V. Fernandez, K. Chandramouli, E. Izquierdo, "Semantic Object based Retrieval from Surveillance Videos", 4th IEEE International Workshop on Semantic Media Adaptation and Personalization (SMAP), 2009

Intelligent Information Visualisation techniques for efficient exploration, navigation and query through large databases

Tijana Janjusevic

Information visualisation offers a unique method to reveal hidden patterns and contextual information through visual presentation and allows users to seek information in an intuitive way. Hence, in this research innovative techniques are developed to support efficient exploration, navigation and query of large multimedia databases. They are listed below.

Hierarchical Browser

The objective of this research is to develop methodologies for efficient content navigation and content exploration through hierarchically structured database. The developed framework utilises the properties of hierarchical structure for assisting users’ task through several interaction and visualization strategies. In particular, “random jump” is a novel navigational approach developed to allow users to perform non-sequential exploration of the large image/key-frame repository.

Multi-Concept Browser

This research investigates the introduction of human knowledge on semantic relations to enrich, widen and/or enhance the queries by combining the simple detected mid-level features. The developed Multi-Concept browsing tool combines user’s knowledge, with information rendering techniques based on distortion and additional geometrical transformations, in order to assist the user in:

Publications

Journals

• T. Janjusevic, S. Benini, E. Izquierdo, and R. Leonardi, "Random assisted browsing of rushes archives," Journal of Multimedia (JMM), 2009

Conferences

• T. Janjusevic and E. Izquierdo, "Visualising the Query Space of the Image Collection," in Proc. Proc. of 13th International Conference Information Visualisation (IV), 2009
• T. Janjusevic, S. Benini, E. Izquierdo, and R. Leonardi, "Random Methods for Fast Exploration of the Raw Video material," in Proc. of 20th International Workshop on Database and Expert Systems Application (DEXA), 2009