Performing Advanced Image Processing

This chapter illustrates how to perform sophisticated image processing.

The source code for the C# versions of this sample program can be found in the under samples\C# *\Advanced Image Processing in your My Documents/IC Imaging Control 3.5 directory.

Overview

This chapter illustrates how to:

The sample described in this chapter lets you draw a rectangle on the live image using the mouse. The region specified by the rectangle is used to check whether the live image has changed. If so, the changed image is displayed. The threshold for the difference between the images can be set by the user.

Setting up the Project

Create a new project and add IC Imaging Control to the form. Before you run the program, select the video device, input and video format as shown in the First Steps Visual Studio .NET 2010 chapter. Alternatively, run the program without selecting a device. In this case, the program shows the device selection dialog provided by IC Imaging Control. If you close this dialog without making a selection, the displays error message and terminates.

Add two buttons to the form and label them Device and Settings. Name them cmdDevice and cmdSettings respectively. If you click the "Device" button, a device selection dialog is displayed. After you have selected a valid device, UpdateDeviceSettings is called. This is a helper function, which sets up a FrameQueueSink that is used to receive images. The sink is initialized with a queued buffer count of 5 and Y800 as color format. ICImagingControl.LiveDisplay is disabled, because the buffers are drawn by hand in this application. The Settings button shows a dialog that allows you to adjust the VCDProperties of the currently selected device. The code for both buttons looks as follows: 

[C#]
private void cmdDevice_Click( object sender, EventArgs e )
{
    icImagingControl1.ShowDeviceSettingsDialog();
    UpdateDeviceSettings();
}
/// <summary>
/// UpdateDeviceSettings
///
/// Setup the sink and some runtime variables.
/// </summary>
private void UpdateDeviceSettings()
{
    cmdStart.Enabled = icImagingControl1.DeviceValid;
    cmdSettings.Enabled = icImagingControl1.DeviceValid;
    if( !icImagingControl1.DeviceValid )
    {
        return;
    }
    // Set the size of ICImagingControl to the width and height
    // of the currently selected video format.
    icImagingControl1.Size = icImagingControl1.VideoFormatCurrent.Size;
    _userROI.Top = 0;
    _userROI.Left = 0;
    _userROI.Bottom = icImagingControl1.Width;
    _userROI.Right = icImagingControl1.Height;
}


[C#]
private void cmdSettings_Click( object sender, EventArgs e )
{
    icImagingControl1.ShowPropertyDialog();
}

Declaring Form member variables

Create a user data type named RECT to hold the coordinates for a rectangle by inserting the following code at the beginning of your Form class. 

[C#]
private struct RECT
{
    public int Left;
    public int Top;
    public int Right;
    public int Bottom;
}

Now add several variables to the class. 

[C#]
private IFrameQueueBuffer _currentlyDisplayedBuffer;
private RECT _userROI;
private bool _userROICommited = false;
private int _threshold = 0;
private FrameQueueSink _sink;

Adding the Mouse Event Procedures

Add event procedures for ICImagingControl's MouseDown, MouseUp, and MouseMove events. Insert the following code into the event procedures:

MouseDown: 

[C#]
if( !_userROICommited && (e.Button == MouseButtons.Left) )
{
    _userROI.Left = e.Location.X;
    _userROI.Top = e.Location.Y;
}

MouseUp: 

[C#]
if( !_userROICommited && !(e.Button == MouseButtons.Left) )
{
    _userROI.Right = e.Location.X;
    _userROI.Bottom = e.Location.Y;
}

MouseMove: 

[C#]
if( !_userROICommited && (e.Button == MouseButtons.Left) )
{
    _userROI.Right = e.Location.X;
    _userROI.Bottom = e.Location.Y;
}

Because we use Y800 as sink color format, the mouse positions are identical to the pixel positions in the image buffers. Thus, we can set the UserROI.bottom and .top members to YPos.

When using a bottom-up sink color format such as RGB8, you had to transform the mouse positions to ICImagingControl1.Height - YPos.

Adding Functionality to the Application

As described in Displaying Buffers via FrameQueueSink, create two buttons ( Start / Stop ), the Form_Load event handler a 

[C#]
FrameQueuedResult NewBufferCallback( IFrameQueueBuffer buffer )
{
    RECT region = NormalizeRect( _userROI, buffer.FrameType.Size );
    if( !_userROICommited )
    {
        ReceiveFrameInContinuousMode( buffer, region );
    }
    else
    {
        ReceiveFrameInCompareMode( buffer, region );
    }
    return FrameQueuedResult.SkipReQueue;
}

This event handler uses the function NormalizeRect to make sure the rectangle's coordinates are not exchanged, e.g. left greater than right. It also calls the method ReceiveFrameInContinuousMode to paint that rectangle onto the image buffer. NormalizeRect is implemented as follows: 

[C#]
private RECT NormalizeRect( RECT val, Size fmtDim )
{
    RECT r = val;
    if( r.Top > r.Bottom )
    {
        int Tmp = r.Top;
        r.Top = r.Bottom;
        r.Bottom = Tmp;
    }
    if( r.Left > r.Right )
    {
        int Tmp = r.Left;
        r.Left = r.Right;
        r.Right = Tmp;
    }
    if( r.Top < 0 )
    {
        r.Top = 0;
    }
    if( r.Left < 0 )
    {
        r.Left = 0;
    }
    if( r.Bottom >= fmtDim.Height )
    {
        r.Bottom = fmtDim.Height - 1;
    }
    if( r.Right >= fmtDim.Width )
    {
        r.Right = fmtDim.Width - 1;
    }
    return r;
}

The method ReceiveFrameInContinuousMode is implemented in the following way: 

[C#]
private void ReceiveFrameInContinuousMode( IFrameQueueBuffer buffer, RECT Region )
{
    if( _currentlyDisplayedBuffer != null )
    {
        _sink.QueueBuffer(_currentlyDisplayedBuffer);
    }
    _currentlyDisplayedBuffer = buffer;
    DrawRectangleY8( buffer, Region );
    icImagingControl1.DisplayImageBuffer( _currentlyDisplayedBuffer );
}

First we check if we have a 'currently displayed buffer' and queue it back into the sink via FrameQueueSink.QueueBuffer. Then we save the current frame in _currentlyDisplayedBuffer. Now we draw the region of interest rectangle directly into the frame data and then update the displayed image via ICImagingControl.DisplayImageBuffer.

Implemented this way, IC Imaging Control enables the user to select a region by painting a rectangle on the control while a live video is running.

Inserting the "CompareMode"

Now, we add the functionality to only update the display, if something changes in the region of interest. Create a button, named cmdROICommit and add the following code to it's click event procedure: 

[C#]
private void cmdROICommit_Click( object sender, EventArgs e )
{
    if( !_userROICommited )
    {
        _userROICommited = true;
        cmdROICommit.Text = "Reset ROI";
    }
    else
    {
        _userROICommited = false;
        cmdROICommit.Text = "Set current ROI";
    }
}

When this button is clicked, the region of interest is committed. The application then switches to the "CompareMode". In this mode, the display is only updated, if something within the region of interest has changed. When the "CompareMode" is already running, the button toggles to "ContinousMode".

Updating the Code in the Event Procedures

Add the the following code to the cmdStart_Click event procedure: 

[C#]
cmdROICommit.Enabled = true;

This enables the painting of the ROI every time you start the live video. This is named "ContinousMode" in this example.

Add the following code to the cmdStop_Click event procedure:. 

[C#]
try
{
    cmdStart.Enabled = true;
    cmdStop.Enabled = false;
    cmdDevice.Enabled = true;
    if( _userROICommited )
    {
        cmdROICommit_Click( sender, e );
    }
    cmdROICommit.Enabled = false;
    icImagingControl1.LiveStop();
    _currentlyDisplayedBuffer = null;
}
catch( Exception ex )
{
    MessageBox.Show( ex.Message );
}

This will reset the mode from "CompareMode" to "ContinousMode" when the "Stop" button is clicked and the "CompareMode" is active. Further, add the line cmdROICommit.Enabled = False to the Form_Load event procedure.

Then alter the code in the ImageAvailable event procedure: 

[C#]
FrameQueuedResult NewBufferCallback( IFrameQueueBuffer buffer )
{
    RECT region = NormalizeRect( _userROI, buffer.FrameType.Size );
    if( !_userROICommited )
    {
        ReceiveFrameInContinuousMode( buffer, region );
    }
    else
    {
        ReceiveFrameInCompareMode( buffer, region );
    }
    return FrameQueuedResult.SkipReQueue;
}

This calls the appropriate methods, depending on the value of _userROICommited.

The "ReceiveFrameInCompareMode" method

The new ImageAvailable event procedure calls the method ReceiveFrameInCompareMode, which is implemented as follows: 

[C#]
private void ReceiveFrameInCompareMode( IFrameQueueBuffer newFrame, RECT Region )
{
    IFrameQueueBuffer oldBuffer = _currentlyDisplayedBuffer;
    if( oldBuffer == null || CompareRegion(oldBuffer, newFrame, Region, _threshold) )
    {
        if( oldBuffer != null )
        {
            _sink.QueueBuffer(oldBuffer);
        }
        _currentlyDisplayedBuffer = newFrame;
        DrawRectangleY8(newFrame, Region);
        icImagingControl1.DisplayImageBuffer(newFrame);
    }
    else
    {
        _sink.QueueBuffer(newFrame);
    }
}

This method calls the CompareRegion function, which compares the region of interest of the frames. If they differ 'enough', the previous frame is queued in the FrameQueueSink and the new frame in newFrame displayed via DisplayImageBuffer.

The CompareRegion Function

The CompareRegion function compares the region of interest of the two image buffers Buf and Buf2. The differences of the pixels in the specified Region are added. The result is saved in the variable greyscaleDifferenceAccu. After accumulating the differences in greyscaleDifferenceAccu, it is divided by the count of pixels in the region. greyscaleDifferenceAccu is compared to the threshold value. If greyscaleDifferenceAccu is greater then threshold, the regions differ 'enough' and the function returns true. Otherwise false is returned. The CompareRegion function is implemented as follows: 

[C#]
private bool CompareRegion( IFrame buf, IFrame buf2, RECT region, int threshold )
{
    int PixelCount = (region.Bottom - region.Top) * (region.Right - region.Left);
    if (PixelCount <= 0)
    {
        return false;
    }
    long greyscaleDifferenceAccu = 0;
    for( int y = region.Top; y <= region.Bottom; y++ )
    {
        for( int x = region.Left; x <= region.Right; x++ )
        {
            greyscaleDifferenceAccu += Math.Abs( GetY8PixelAt( buf, x, y ) - GetY8PixelAt( buf2, x, y ) );
        }
    }
    var greyscaleDifference = greyscaleDifferenceAccu / PixelCount;
    if( greyscaleDifference > threshold )
    {
        return true;
    }
    else
    {
        return false;
    }
}

<< Programmer's Guide