Release 8.1.5
A67293-01
 Library |
 Product |
 Contents |
 Index |
| Oracle8i Visual Information Retrieval User's Guide and Reference Release 8.1.5 A67293-01 |
|
This chapter explains, at a high level, why and how to use content-based retrieval. It covers the following topics:
Inexpensive image-capture and storage technologies have allowed massive collections of digital images to be created. However, as a database grows, the difficulty of finding relevant images increases. Two general approaches to this problem have been developed, both of which use metadata for image retrieval:
With Visual Information Retrieval, you can combine both approaches in designing a table to accommodate images: use traditional text columns to describe the semantic significance of the image (for example, that the pictured automobile won a particular award, or that its engine has six or eight cylinders), and use the Visual Information Retrieval type for the image, to permit content-based queries based on intrinsic attributes of the image (for example, how closely its color and shape match a picture of a specific automobile).
As an alternative to defining image-related attributes in columns separate from the image, a database designer could create a specialized composite data type that combines Visual Information Retrieval and the appropriate text, numeric, and date attributes.
The primary benefit of using content-based retrieval is reduced time and effort required to obtain image-based information. With frequent adding and updating of images in massive databases, it is often not practical to require manual entry of all attributes that might be needed for queries, and content-based retrieval provides increased flexibility and practical value. It is also useful in providing the ability to query on attributes such as texture or structure that are difficult to represent using keywords.
Examples of database applications where content-based retrieval is useful -- where the query is semantically of the form, "find objects that look like this one" -- include:
For example, a Web-based interface to a retail clothing catalog might allow users to search by traditional categories (such as style or price range) and also by image properties (such as color or texture). Thus, a user might ask for formal shirts in a particular price range that are off-white with pin stripes. Similarly, fashion designers could use a database with images of fabric swatches, designs, concept sketches, and finished garments to facilitate their creative processes.
A content-based retrieval system processes the information contained in image data and creates an abstraction of its content in terms of visual attributes. Any query operations deal solely with this abstraction rather than with the image itself. Thus, every image inserted into the database is analyzed, and a compact representation of its content is stored in a feature vector, or signature.
The signature contains information about the following visual attributes:
Feature data for all these visual attributes is stored in the signature, whose size typically ranges from 1000 to 2000 bytes. For better performance with large image databases, you can create an index based on the signatures of your images. See Section 2.4 for more information on indexing.
Images in the database can be retrieved by matching them with a comparison image. The comparison image can be: any image inside or outside the current database, a sketch, an algorithmically generated image, and so forth.
The matching process requires that signatures be generated for the comparison image and each image to be compared with it. Images are seldom identical, and therefore matching is based on a similarity-measuring function for the visual attributes and a set of weights for each attribute. The score is the relative distance between two images being compared. The score for each attribute is used to determine the degree of similarity when images are compared, with a smaller distance reflecting a closer match, as explained in Section 2.3.3.
Global color reflects the distribution of colors within the entire image, whereas local color reflects color distributions and where they occur in an image. To illustrate the difference between global color and local color, consider Figure 2-1.
Image 1 and Image 2 are the same size and are filled with solid colors. In Image 1, the top left quarter (25%) is red, the bottom left quarter (25%) is blue, and the right half (50%) is yellow. In Image 2, the top right quarter (25%) is blue, the bottom right quarter (25%) is red, and the left half (50%) is yellow.
If the two images are compared first solely on global color and then solely on local color, the following are the similarity results:
Thus, if you need to select images based on the dominant color or colors (for example, to find apartments with blue interiors), give greater relative weight to global color. If you need to find images with common colors in common locations (for example, red dominant in the upper portion to find sunsets), give greater relative weight to local color.
Figure 2-2 shows two images very close (score = 0.0) in global color. Figure 2-3 shows two images very close (score = 0.02461) in local color.
Texture is most useful for full images of textures, such as catalogs of wood grains, marble, sand, or stones. These images are generally hard to categorize using keywords alone because our vocabulary for textures is limited. Texture can be used effectively alone (without color) for pure textures, but also with a little bit of global color for some kinds of textures, like wood or fabrics. Figure 2-4 shows two similar fabric samples (score = 4.1).
Structure is not strictly confined to certain sizes or positions. However, when objects are of the same size or position, they have a lower score (greater similarity) than objects of different sizes. Structure is useful to capture objects such as horizon lines in landscapes, rectangular structures in buildings, and organic structures like trees. Structure is very useful for querying on simple shapes (like circles, polygons, or diagonal lines) especially when the query image is drawn by hand and color is not considered important when the drawing is made. Figure 2-5 shows two images very close (score = 0.61939) in structure.
Visual Information Retrieval supports face recognition software developed by Viisage Technology, Inc. This third-party software analyzes images of faces and generates a facial signature based on various unique biometric characteristics.
After you have generated facial signatures with the Viisage software, you can use Visual Information Retrieval Convert( ), VIRScore( ), and VIRSimilar( ) operators to compare the images.
See the Viisage product documentation for more details.
When you match images, you assign an importance measure, or weight, to each of the visual attributes, and Visual Information Retrieval calculates a similarity measure for each visual attribute.
Each weight value reflects how sensitive the matching process should be to the degree of similarity or dissimilarity between two images. For example, if you want global color to be completely ignored in matching, assign a weight of 0.0 to global color; in this case, any similarity or difference between the color of the two images is totally irrelevant in matching. On the other hand, if global color is extremely important, assign it a weight greater than any of the other attributes; this will cause any similarity or dissimilarity between the two images with respect to global color to contribute greatly to whether or not the two images match.
Weight values can be any positive real numbers. During processing, the values are normalized such that they sum to 100%. To avoid confusion, you should get into a habit of always using the same scale (0.00 to 1.00, or 0 to 100) when providing weight values. At least one of the visual attribute weights must be set greater than zero. See Section 2.3.3 for details of the calculation.
The similarity measure for each visual attribute is calculated as the score or distance between the two images with respect to that attribute. The score can range from 0 (no difference) to 100 (maximum possible difference). Thus, the more similar two images are with respect to a visual attribute, the smaller the score will be for that attribute.
As an example of how distance is determined, assume that the dots in Figure 2-6 represent scores for three images with respect to two visual attributes, such as global color and structure, plotted along the X and Y axes of a graph.
For matching, assume Image 1 is the comparison image, and Image 2 and Image 3 are each being compared with Image 1. With respect to the global color attribute plotted on the X axis, the distance between Image 1 and Image 2 is relatively small (for example, 15), whereas the distance between Image 1 and Image 3 is much greater (for example, 75). If the global color attribute is given more weight, then the fact that the two distance values differ by a great deal will probably be very important in determining whether or not Image 2 and Image 3 match Image 1. However, if global color is minimized and the structure attribute is emphasized instead, then Image 3 will match Image 1 better than Image 2 matches Image 1.
In Section 2.3.2, Figure 2-6 showed a graph of only two of the attributes that Visual Information Retrieval can consider. In reality, when images are matched, the degree of similarity depends on a weighted sum reflecting the weight and distance of all four of the visual attributes of the comparison image and the test image.
For example, assume that for the comparison image (Image 1) and one of the images being tested for matching (Image 2), Table 2-1 lists the relative distances between the two images for each attribute. Note that you would never see these individual numbers unless you computed four separate scores, each time highlighting one attribute and setting the others to zero.
| Visual Attribute | Distance |
|---|---|
|
Global color |
15 |
|
Local color |
90 |
|
Texture |
5 |
|
Structure |
50 |
In this example, the two images are most similar with respect to texture (distance = 5) and most different with respect to local color (distance = 90).
Assume that for the matching process, the following weights have been assigned to each visual attribute:
The weights are typically supplied in the range of 0.0 to 1.0. Within this range, a weight of 1 indicates the strongest emphasis, and a weight of 0 means the attribute should be ignored. You can use a different range (such as 0 to 100), but be careful not to accidentally combine different ranges. The values you supply are automatically normalized such that the weights total 100 percent, still maintaining the ratios you have supplied. In this example, the weights were specified such that normalization was not necessary.
The following formula is used to calculate the weighted sum of the distances, which is used to determine the degree of similarity between two images:
weighted_sum = global_color_weight * global_color_distance + local_color_weight * local_color_distance + texture_weight * texture_distance + structure_weight * structure_distance
The degree of similarity between two images in this case is computed as:
0.1*gc_distance + 0.6*lc_distance + 0.2*tex_distance + 0.1*struc_distance
Using the supplied values, this becomes:
(0.1*15 + 0.6*90 + 0.2*5 + 0.1*50) = (1.5 + 54.0 + 1.0 + 5.0) = 61.5
To illustrate the effect of different weights in this case, assume that the weights for global color and local color were reversed. In this case, the degree of similarity between two images is computed as:
0.6*gc_distance + 0.1*lc_distance +0.2*tex_distance + 0.1*struc_distance
That is:
(0.6*15 + 0.1*90 + 0.2*5 + 0.1*50) = (9.0 +9.0 + 1.0 + 5.0) = 24.0
In this second case, the images are considered to be more similar than in the first case, because the overall score (24.0) is smaller than in the first case (61.5). Whether or not the two images are considered matching depends on the threshold value (explained in Section 2.3.4): if the weighted sum is less than or equal to the threshold, the images match; if the weighted sum is greater than the threshold, the images do not match.
In these two cases, the correct weight assignments depend on what you are looking for in the images. If local color is extremely important, then the first set of weights is a better choice than the second, because the first set of weights grants greater significance to the disparity between these two specific images with respect to local color (weighted sum of 24 versus 61.5). Thus, with the first set of weights, these two images are less likely to match -- and a key goal of content-based retrieval is to eliminate uninteresting images so that you can focus on images containing what you are looking for.
When you match images, you assign a threshold value. If the weighted sum of the distances for the visual attributes is less than or equal to the threshold, the images match; if the weighted sum is greater than the threshold, the images do not match.
Using the examples in Section 2.3.3, if you assign a threshold of 50, the images do not match when the weighted sum is 61.5, but they do match when the weighted sum is 24. If the threshold is 20, the images do not match in either case; and if the threshold is 61.5 or greater, the images match in both cases.
The following example shows a cursor (getphotos) that selects the photo_id, annotation, and photograph from the Pictures table where the threshold value is 20 for comparing photographs with a comparison image:
CURSOR getphotos IS SELECT photo_id, annotation, photo FROM Pictures WHERE ORDSYS.VIRSimilar(photo.Signature, comparison_sig, 'globalcolor="0.4",
localcolor="0.2", texture="0.10", structure="0.3"', 20)=1;
Before the cursor executes, the Analyze( ) operator must be used to compute the signature of the comparison image (comparison_sig), and to compute signatures for each image in the table. Chapter 4 describes all the operators, including Analyze( ) and Similar( ).
The number of matches returned generally increases as the threshold increases. Setting the threshold to 100 would return all images as matches. Such a result, of course, defeats the purpose of content-based retrieval. If your images are all very similar, you may find that even a threshold of 50 returns too many (or all) images as matches. Through trial and error, adjust the threshold to an appropriate value for your application.
You will probably want to experiment with different weights for the visual attributes and different threshold values, to see which combinations retrieve the kinds and approximate number of matches you want.
A domain index, or extensible index, is a new approach for supporting complex data objects. Oracle8i and Visual Information Retrieval cooperate to define, build, and maintain an index for image data. This index is of type ORDVIRIDX. Once created, the index automatically updates itself every time an image is inserted or removed from the database table. The index is created, managed, and accessed by routines supplied by the index type.
For better performance with large image databases, you should always create and use an index for searching through the image signatures. The default search model compares the signature of the query image to the signatures of all images stored in the database. This works well for simple queries such as, "does this picture of an automobile match the image stored with the client's insurance records?" However, if you want to compare that image with thousands or millions of images in order to determine what kind of vehicle it is, then a linear search though the database would be impractical. In this case, an index based on the image signatures would greatly improve performance.
Assume you have a table T containing fabric ID numbers and pattern photographs:
CREATE TABLE T (fabric_id NUMBER, pattern_photo ORDSYS.ORDVIR);
Populate the table with images, and process each image using the Analyze( ) operator to generate the signatures.
Once the signatures have been created, the following command creates an index on this table, based on the data in the photo column. Note that you only build an index on image signatures generated by Visual Information Retrieval. You cannot build an index on facial signatures.
CREATE INDEX idx1 ON T(pattern_photo.signature) INDEXTYPE IS ORDSYS.ORDVIRIDX PARAMETERS ('ORDVIR_DATA_TABLESPACE = <name>,
ORDVIR_INDEX_TABLESPACE = <name>');
The index name is limited to 24 or fewer characters.1 As with any Oracle table, do not use pound signs (#) or dollar signs ($) in the name. Also as usual, the tablespace must be created prior to creating the table.
The index data resides in two tablespaces. The first contains the actual index data, and the second is an internal index created on that data. See Section 3.7 for suggestions concerning the sizes of these tablespaces.
Finally, as with other Oracle indexes, you should analyze the new index as follows:
ANALYZE INDEX idx1 COMPUTE STATISTICS;
Two operators, VIRSimilar( ) and VIRScore( ) support queries using the index. The operators automatically use the index if it is present. See Section 4.3 for syntax information and examples.
The human mind is infinitely smarter than a computer in matching images. If we are near a street and want to identify all red automobiles, we can easily do so because our minds rapidly adjust for the following factors:
However, for a computer to find red automobiles (retrieving all red automobiles and no or very few images that are not red or not automobiles), it is helpful if all the automobile images have the automobile occupy almost the entire image, have no extraneous elements (people, plants, decorations, and so on), and have the automobiles pointing in the same direction. In this case, a match emphasizing global color and structure would produce useful results. However, if the pictures show automobiles in different locations, with different relative sizes in the image, pointing in different directions, and with different backgrounds, it will be difficult to perform content-based retrieval with these images.
The following are some suggestions for selecting images or preparing images for comparison. The list is not exhaustive, but the basic principle to keep in mind is this: Know what you are looking for, and use common sense. If possible, crop and edit images in accordance with these suggestions before performing content-based retrieval.
1
The standard Oracle restriction is 30 characters for table or index names. However, Visual Information Retrieval requires an extra 6 characters for internal processing of the domain index.
