Content-based Image Retrieval
Project Description:
· Problem Statement
Image search engines become indispensable tools for users who look for images from a large-scale image collection and World-Wide Web. Its key technique is content-based image retrieval (CBIR) having the ability of searching images via automatically derived image features, such as color, texture or shape. The major difficulty of CBIR lies in the big gap between low-level image features and high-level image semantics. Fig. 1 shows an image. Human beings may identify its semantics (a balloon is flying in the sky) effortlessly. However, computers only know it contains white, black, and blue color. Hence, if one user selects Fig. 1 as the query example, the CBIR system retrieves images with similar colors to Fig. 1. The poor results are displayed by Fig. 2.
Fig. 1 An image of balloon Fig. 2 Query results based on color features
· Proposed Research
In order to address the above problem, we focus on two researches:
(1) Relevance Feedback using Machine Learning Algorithms
Relevance feedback (RF) is an interactive process to incorporate human perception subjectivity into the query process and provide users with the opportunity to evaluate the retrieval results. Fig. 3 shows the basic architecture of a RF based CBIR system.
Fig. 3 The basic architecture of a general RF based CBIR system
We consider RF from the perspective of supervised learning. Given a query, the system first retrieves a list of ranked images using a similarity metric. Then, the user selects a set of positive and negative examples from the returned results. The systems learn from labeled examples to train a classifier. Many classical machine learning schemes may be applied to train the classifier, such as Bayesian learning, Support Vector Machines (SVM), Boosting, and so on. The retrieval performance is gradually improved after several feedback iterations.
Fig. 4 illustrates a RF iteration. (a) shows the results of the first round of retrieval. The top left image is the query example. The images marked by "√" are the positive example, which are relevant to the query image. On the contrary, images marked by "×" are the negative example, which are irrelevant to the query image. (b) provides the results after this feedback iteration.
(a) The results of the first round retrieval (b) The results after one feedback iteration
Fig. 4 An example of the RF process
(2) Memory Learning by Accumulating User Feedback Log
To further improve the performance of CBIR systems, we propose a memory learning (ML) framework. Its basic idea is to learn semantics from previous users’ feedback knowledge instead of image contents. The architecture of ML framework is shown in Fig. 5.
Fig. 5 The architecture of ML framework
Comparing with general RF, our work has four contributions:
o A feedback knowledge memory model is presented to gather the users’ feedback information;
o A learning strategy based on the memorized information is proposed. It can estimate the hidden semantic relationships among images.
o A seamless combination of normal RF (low-level features based) and the memory learning (semantics based) is proposed to improve the retrieval performance.
o A semantics based annotation propagation scheme is proposed using learned semantics.
Project Investigators:
Prof. Ngan, King Ngi, Dr. Han, Jun Wei
Email: {knngan, jwhan}@ee.cuhk.edu.hk
Homepage: http://www.ee.cuhk.edu.hk/~knngan
Address: Department of Electronic
Engineering
The Chinese University of Hong Kong
Shatin, New Territorie,
HONG KONG
Tel: +852 26098255
Current Project Achievements
·
Publications
J. Han, K.N. Ngan, M. Li and H. Zhang, “A memory learning framework for effective image retrieval,” To appear in IEEE Transactions on Image Processing, U.S.A.
J. Han, K.N. Ngan, M. Li and H. Zhang, "Learning semantic concepts from user feedback log for image retrieval," To appear in IEEE International Conference on Multimedia and Expo 2004, Taipei, Taiwan.