PARP Research Group University of Murcia, Spain


Image processing

The group Image processing contains all the image processing functionality offered by the QVision, most of the which is built over the Intel Performance Primitives library.

In the QVision framework, images are represented as objects. These objects are created from the QVImage class. They contain the pixel values corresponding to a raster or bitmap image representation.

QVImage objects

The QVImage class is parametrized with the type of the values contained in each pixel, and the number of channels of the image. For example, the following code will create an 8 bit-depth image, with 3 color channels:

QVision<uChar, 3> image;

In the case of one channel images, the second template parameter for the QVImage class can be omited. For example, the following two variable declarations are equivalent:

QVision<uChar, 1> imageA;
QVision<uChar> imageB;

The uChar type is an alias for the unsigned char C++ type. Technically you can create image objects with an arbitrarily number of channels, but in practice you generally won't find practical functionallity applied to images with a channel number other than 1 or 3.

As the uChar type, the QVision defines several ad-hoc types for bit depth and precision image types. A comprehensive list can be seen in the next table:

C++ type QVision type Bit depth
unsigned char uChar 8
unsigned short uShort 16
unsigned int uInt 32
signed char sChar 8
signed short sShort 16
signed int sInt 32
typedef float sFloat 32
typedef double sDouble 64

We encourage programmers to use them for image types, instead of the corresponding C++ types.

Image creation

The class QVImage has several different constructors. The first one creates the image by specifying its size in rows and cols. For example:

QVImage<sInt, 3> imageA(320, 240);

That line will create a 3-channel image with 32 bit-depth, of size 320x240 pixels. The second is the copy constructor. It initializes the size and content from another image, of the same bit-depth and channel number:

QVImage<sInt, 3> imageB = imageA;

The third one is the convert constructor. It is used in the same fashion than the copy constructor, and can be used to initialize the size and the content of another image, with different bit-depth and/or channel number:

QVImage<uChar, 1> imageC = imageB;

Also, the size and content of image objects can be initialized loading the content of an image file stored in the filesystem, using the image file loader constructor. It only requires the file name as the input parameter:

QVImage<uChar, 3> imageD("lena.png");

That instruction will load the content of the file lena.png on the object imageD, providing the file exists in the datapath and is readable by the application.

Copying images

The copy operator has a similar use to the copy and convert constructors. It can be used to copy the size and content of an image onto an already initialized one. An example usage follows:

QVImage<uChar, 3> imageE("lena.jpg"), imageF;   // Construct image objects.
QVImage<sInt, 1> imageG;

[...]

imageF = imageE;
imageG = imageF;

After the above instructions the object imageF will contain a copy of the contents of the object imageE, and the object imageG will contain a 32 bit-depth version of the same contents.

Image pixel access

The QVision provides several ways to access the pixels of an image. The simplest and less efficient of them is using the parenthesis operator. The following code shows how it can be used to retrieve and update the value of the pixels in an image:

QVImage<uChar, 3> imageA("lena.png");

QVImage<uChar, 3> imageB(imageA.getCols(), imageA.getRows());
for(int i = 0; i < imageA.getCols(); i++)
        for(int j = 0; j < imageA.getRows(); j++)
                {
                imageB(i, j, 0) = imageA(i, j, 1);
                imageB(i, j, 1) = imageA(i, j, 0);
                imageB(i, j, 2) = imageA(i, j, 2);
                }

That code swaps the content of the first and the second channels of the image contained in the file lena.png. The default value for the third parameter of the parenthesis operator is 0. Thus if the image has only one channel, the third parameter of the function call operator can be ommited:

QVImage<uChar, 3> imageA("lena.png");

QVImage<uChar, 1> imageB(imageA.getCols(), imageA.getRows());
for(int i = 0; i < imageA.getCols(); i++)
        for(int j = 0; j < imageA.getRows(); j++)
                imageB(i, j) =  0.33* imageA(i, j, 0) +
                                0.33* imageA(i, j, 1) +
                                0.33* imageA(i, j, 2);

This code obtains a gray-scale version of the lena.png image in the object imageB.

A more efficient way of accessing the contents of an image is using the following macros:

Macros QVIMAGE_INIT_READ and QVIMAGE_INIT_WRITE are used to initialize access to an image in read or read/write mode respectively. Macro QVIMAGE_PIXEL is used to access a pixel in an image, previously initialized with the other four macros. The previous code example can be rewritten using the macros in the following way:

QVImage<uChar, 3> imageA("lena.png");
QVImage<uChar, 1> imageB(imageA.getCols(), imageA.getRows());

QVIMAGE_INIT_READ(uChar, imageA);
QVIMAGE_INIT_WRITE(uChar, imageB);
for(int i = 0; i < imageA.getCols(); i++)
        for(int j = 0; j < imageA.getRows(); j++)
                QVIMAGE_PIXEL(imageB, i, j, 0) = 
                imageB(i, j) =  0.33* QVIMAGE_PIXEL(imageA, i, j, 0) +
                                0.33* QVIMAGE_PIXEL(imageA, i, j, 1) +
                                0.33* QVIMAGE_PIXEL(imageA, i, j, 2);

The code becomes somewhat less legible, but time performance is better this way.

Macros QVIMAGE_PTR_INIT_READ and QVIMAGE_PTR_INIT_WRITE are equivalent to QVIMAGE_INIT_READ and QVIMAGE_INIT_WRITE, but can receive a pointer to an image, as the second input parameter. An equivalent code to the previous example, using image pointers and those macros would be the following:

// Input and output images
QVImage<uChar, 3> imageA("lena.png");
QVImage<uChar, 1> imageB(imageA.getCols(), imageA.getRows());

// Get pointers to images.
QVImage<uChar, 3> *ptrImageA = &imageA;
QVImage<uChar, 1> *ptrImageB = &imageB;

// Read the input image
QVIMAGE_PTR_INIT_READ(uChar, ptrImageA);
QVIMAGE_PTR_INIT_WRITE(uChar, ptrImageB);
for(int i = 0; i < ptrImageA->getCols(); i++)
        for(int j = 0; j < ptrImageA->getRows(); j++)
                QVIMAGE_PIXEL(ptrImageB, i, j, 0) = 
                imageB(i, j) =  0.33* QVIMAGE_PIXEL(ptrImageA, i, j, 0) +
                                0.33* QVIMAGE_PIXEL(ptrImageA, i, j, 1) +
                                0.33* QVIMAGE_PIXEL(ptrImageA, i, j, 2);

Note:
The macros QVIMAGE_INIT_READ, QVIMAGE_INIT_WRITE, QVIMAGE_PTR_INIT_READ and QVIMAGE_PTR_INIT_WRITE must always receive as second parameter a plain variable identificator, not an expression of type QVImage. Thus an image pointer cannot be used with the QVIMAGE_INIT_READ or QVIMAGE_INIT_WRITE, because that would imply including the derreference operator * in the second parameter.

Image input/output

Besides the image file constructor previously discussed, the QVision offers some functionallity to load the contents of image files and YUV4MPEG2 video files on image objects.

The first application example seen in section First application under the QVision includes the header file qvio.h. This file contains much of this functionallity. It offers the functions writeQVImageToFile and readQVImageFromFile, which can be used to read the content of an image file from and to a QVImage object. It also provides functions writeYUV4MPEG2Header, writeYUV4MPEG2Frame, writeYUV4MPEG2Frame, readYUV4MPEG2Frame and readYUV4MPEG2Header to read and write image frames from and to YUV4MPEG2 video files.

The documentation of these functions can be seen in the group Video and image input/output group. Besides the aforementioned functions, that group includes some I/O classes, which can be used to create input and output blocks for a QVision block oriented application. The creation of this kind of applications, along with the use of these block objects will be described in section Block programming. There we will also describe the complete video input system of QVision, which is mainly based on mplayer.

QVision wrapper functions for the IPP library

The IPP wrapper functions is a subgroup included in the Image processing group. It contains a set of wrapper functions for the Intel's Performance Primitives library, in order to increase its usability in the developing of new applications using the QVision.

Each one of these functions corresponds to a function in the IPP library. Most of the functions from the IPP library perform computations on one or two input images, returning another image as a result. When directly using the IPP library, the developer must generally provide a couple of parameters for each input/output image to the corresponding IPP function: a pointer to the data buffer of the image and its step -number of bytes between consecutive rows of the image-.

This can be considered a "C-like", low-level approach. Instead, the QVision framework, which is more object oriented, defines the class QVImage to create image objects which encapsulate all the information needed to work with an image: amongst them, of course, the needed pointer to the data buffer, and the step of the image. The IPP wrapper functions group contains wrapper functions which can receive QVImage objects instead of the pointer and the step of each image, simplifying the interface of the IPP function by avoiding the use of pointers and extra parameters, and freeing the developer fo working directly with error-prone low level pointers. Thus, this wrapper package is included to offer a clearer and object oriented alternative to direct usage of the IPP functions.

Example of IPP wrapper function

Take for instance the functions used for adding images in the IPP library. There is a whole family of functions, denoted with the prefix ippiAdd_, followed by a subfix indicating some relevant information, like the storage type for the elements of the image, its channel number, and so on. An example is the function ippiAdd_8u_C1RSfs, used to add two 8-bit unsigned data type images composed by one single channel and store the result in a destination image of the same type. This function has the following header:

IppStatus ippiAdd_<mod>(
        const Ipp<datatype>* pSrc1, int src1Step,
        const Ipp<datatype>* pSrc2, int src2Step,
        Ipp<datatype>* pDst, int dstStep,
        IppiSize roiSize,
        int scaleFactor);

The package IPP wrapper functions contains a corresponding wrapper function named Add, which uses the following header:

void Add(       const QVImage<uChar, 1> & qvimage_pSrc1,
                const QVImage<uChar, 1> & qvimage_pSrc2,
                QVImage<uChar, 1> & qvimage_pDst,
                const int scaleFactor = 1,
                const QPoint &destROIOffset = QPoint(0,0));

The IPP wrapper functions package substitutes the input parameters pSrc1 and src1Step from the IPP function ippiAdd_8u_C1RSfs, which correspond to the first input image, by the input image object type parameter qvimage_pSrc1 in the Add function. The latter function will basically call the IPP function, mapping the pointer to the data buffer of the image qvimage_pSrc1 and its step value respectively to the input parameters pSrc1 and src1Step of the IPP function.

There is a double advantage with this approach. Besides the clearer interface of the Add function, which allows an easier and more robust, object-oriented usage, the name of the function gets simplified by eliminating the function subfix. This can be done by overloading the Add function, to apply the proper version of the ippiAdd function of the IPP library depending on the number, type, and channel numbers of the input parameters provided by the developer using the function.

The following table illustrates the type equivalence between IPP image types and QVImage types:

QVision type IPP type
uChar Ipp8u
uShort Ipp16u
uInt Ipp32u
sChar Ipp8s
sShort Ipp16s
sInt Ipp32s
sFloat Ipp32f
sDouble Ipp64f

Image features group

This group contains several medium-level image processing functions, which obtain some kind high-level data structures from the images. Image features are pieces of information which can be extracted from an image, and are relevant for solving a given computer vision task.

By now, QVision includes several interesting image feature extraction algorithms: MSER, borders, corners, etc... We keep constantly adding new methods in this group. You can check the documentation of the group Image features for a full description the implemented functions.

Regions of Interest (ROI)

Most of the IPP functions restrict their operations to a rectangular region of the image, which can be either the whole image, or some part of it. QVImage objects contain a property named ROI, which can be modified and read using several methods, like QVImage::getROI, QVImage::setROI, and so on. This property contains a rectangle inside the image, which will be used as the Region of Interest by the IPP functions which operate with it, in the IPP wrapper functions.

Example ROI Usage:

The following code illustrates the usage of the Copy function, which performs operations on the ROI of the input image:

#include <qvipp.h>      // QVIPP functions are included in this header file.
[...]

QVImage<uChar> sourceImage("tajMahal.png");
sourceimage.setROI(10,10,90,90);        // This sets the ROI of the input image.

QVImage<uChar> destinationImage(200, 200);
Copy(sourceimage, destinationImage);

[...]

Figure below shows the result of that code on an input image. The left image is the source image, containing the tajMahal image. The right image shows the result of the Copy operation. The black area can contain unspecified content, not necesarily black, if it was not previously initialized so.

tajmahalcopyroi.png

The ROI is commonly used to specify the portion of the image which contains valid information. Thus, after the copy operation of the previous example, the ROI of the destinationImage will be set to contain only the part of the image which really shows a part of the Taj Mahal -that is, that contains "valid" data-.

Note:
If the destination image is not large enough to contain the ROI, it will be resized. The new content of the image, outside the original image size, will be unspecified.
Each image processing function should update the ROI of the output image or images, so that they contain the correct area computed by the operations, and which can be used in posterior processing. The following code is useful to illustrate this:

[...]

QVImage<uChar> sourceImage("lena.png");
sourceImage.setROI(100, 110, 90, 40);   // This sets the ROI of the input image.

QVImage<uChar> temporaryImage(256, 256);
FilterGauss(sourceImage, temporaryImage, 5);

QVImage<uChar> destinationImage(256, 256);
FilterGauss(temporaryImage, destinationImage, 5);

[...]

It applies two Gaussian filter on the lena image. The first call to FilterGauss will store in the ROI of the temporaryImage the area of the image which contains valid data from its operations, so the next call to FilterGauss will only work with that part of the image -the one that now is supposed to be valid-.

lenagauss_5_5roi.png

Of course, and if needed, the programmer can anyway set another ROI at any time by explicitly calling the method QVImage::setROI().

Destination ROI pointer

In some cases the developer may want to indicate a location for the ROI in the destination image. The (optional) parameter destROIOffset can be used to indicate a point in the destination image where the IPP wrapper functions function must locate the top left corner of the ROI.

This parameter is included in the header of several functions from the package IPP wrapper functions, the Copy function amongst them. The following code illustrates the usage of the destROIOffset parameter:

[...]

QVImage<uChar> sourceImage("tajMahal.png");
sourceimage.setROI(10,10,90,90);        // This sets the ROI of the input image.

QVImage<uChar> destinationImage(200, 200);
Copy(sourceimage, destinationImage, QPoint(60,30));

[...]

The QPoint(60,30) parameter in the call to the Copy function indicates that it should copy the content of the ROI in the source image to the destination location, starting at the point (60,30):

tajmahalcopyroianchor.png

Note:
When dealing with several input images, the IPP wrapper functions functions usually perform a ROI size checking to ensure that the ROI of all the input images is adecuate. For example, the Add function checks that both of the input images have the same ROI size. A correct example usage of the Add function would be this:
[...]
// lena is an image of size 256x256. tajMahal is an image of size 200x200.
QVImage<uChar> tajMahal("tajMahal.png"), lena("lena.png");

lena.setROI(45,65,155,85);                      // Set initial ROI for image sourceImage1.
tajMahal.setROI(30,20,155,85);                  // Set initial ROI for image sourceImage2.
                                                // ROI's have a dimention of 155x85 pixels
QVImage<uChar> destinationImage(230,230);       // 'destinationImage' has size 230x230

Add(lena, tajMahal, destinationImage, QPoint(10,20));
[...]
The following picture illustrates the result of that code:
tajmahallenacopyroianchor.png
If the developer provides the Add function two images with an incorrect ROI size, and if the program executes in DEBUG mode, a Q_ASSERT will be raised and the program will stop, indicating the error.



QVision framework. PARP research group, copyright 2007, 2008.