Frame Filters

With frame filters, a versatile and powerful way to manipulate image data is introduced. Besides other features, frame filters can be used to change or convert the video format of the image stream, implement image processing and control which frames are processed.

Introduction

In contrast to other DirectShow libraries, the frame filters of IC Imaging Control are not DirectShow filters. But they are executed in the context of a DirectShow thread that processes the image stream. This saves the programmer the hassle of creating a DirectShow filter module, registering it and defining interfaces for the parameters. Frame filters are implemented in C++ as part of a filter module. They are loaded on demand by the OCX from the module and may be inserted at three locations in the image stream. Frame filters are basically far easier to implement and to use as DirectShow filters by providing the same performance with regard to image processing. C# allows frame filters to be implemented as part of a program. In order to create a frame filter module, C++ has to be used.

Filter Categories

Frame filters open the door to a vast number of new IC Imaging Control based applications. In order to show you what is possible with this new concept, we discuss categories of tasks that can be implemented with frames filters. For some of these tasks, frame filters are shipped with IC Imaging Control.

Gate:
Gate filters drop unwanted frames based on image analysis or other information like the frame count or time stamps.
Color Space Conversion:
All kinds of transformations from one color format to another fall in to this category. In addition, the reduction of the pixel depth, e.g. extracting the 8 most significant bits out of a 10 or 12 bit monochrome format is regarded as a color space conversion. The reduction of the pixel depth is used to display a 10 or 12 bit monochrome image. A good example for a color space conversion is the DeBayer Filter contained in stdfilters.ftf. It converts color raw data in Bayer format to RGB.
ROI:
Filters in this category allow a Region Of Interest to be cropped out of an image stream. The ROI Filter that is contained in stdfilters.ftf implements this functionality.
Rotation:
This category includes rotation and flipping. The Rotate Flip Filter that is contained in stdfilters.ftf provides rotation by 90°, 180° and 270°, as well as flipping on the horizontal and vertical axis. There is a filter in the DemoFilter.ftf that provides rotation by any angle, but this filter is a demo implementation and very slow.
Scaling:
Scaling up and down may be necessary to match a predefined display size or to save disk space for surveillance applications.
Sink:
Sink filters are consumers. They process the image stream in their Transform method completely and do not send frames to the IC Imaging Control sink. If you want to use professional high performance codecs, you would implement a sink filter in order to interface the codec's API. This approach is necessary because most professional codecs come as a DLL instead of a DirectShow codec. These codecs provides functionality to compress image data and write it to a file at the same time, which makes a sink obsolete. Of course, you could use the ImageBuffers of a FrameHandlerSink to access the image data and send it to the codec's API. In this case, the image data would have been copied by the sink to the ImageBuffers which is not the case, if the codec is interfaced using a frame filter. In other words, the sink filter approach saves one copy operation per frame, which can increase the performance significantly, especially if large frames are processed.
Extraction:
Extraction filters take specific frames out of the image stream and apply processing on them, e.g. saving them to disk as image files. The image stream itself remains unchanged and is sent further to the sink. Such a filter allows specific images to be saved uncompressed, while a MediaStreamSink generates a compressed video file.
Image Processing:
All image processing functions that produce one image as output can be implemented easily by a frame filter. For functions that produce more than one image, special concepts have to be developed. One possible concept would be to write the output images one after the other into the resulting image. If data other than images is generated by the function (e.g. data generated by a blob analysis), the application could read out this data by using filter parameters.
Buffer Format Conversion:
Most image processing libraries have a special internal image buffer format. Frame filters can be used to convert image buffers of the image stream to such a special image buffer format. Doing so, the elements in the ring buffer of the FrameHandlerSink contain data in the special format. Therefore, the data of these buffers may be passed directly to the according image processing library. This saves one copy step and, therefore, increases the performance of the system.

Using Existing Frame Filters

As described earlier, frame filters can be created directly in an application or loaded from a filter module. IC Imaging Control provides some filters in the module stdfilters.ftf. In order to obtain a list of the frame filters in all currently available filter modules, examine the FrameFilterInfo.FrameFilterInfos collection. With the FrameFilterInfo.FrameFilterPath property, an additional path for filter modules may be specified. A call to FrameFilter.Create creates an instance of a filter that was selected from the list of available filters. Now that we have a frame filter instance, we can insert it at three different locations in the image stream:

Please note that the same instance of a frame filter should not be used at more than one place in the image stream because its internal state may get inconsistent, especially if FrameFilterImpl.NotifyStart and FrameFilterImpl.NotifyStop are used. If it is necessary to insert the same filter at more than one location in the image stream, different instances of the frame filter should be used.

The internal data of a frame filter can be accessed through the parameter interface. With the methods FrameFilter.SetIntParameter, FrameFilter.GetIntParameter, FrameFilter.SetBoolParameter, FrameFilter.GetBoolParameter, FrameFilter.SetStringParameter, FrameFilter.GetStringParameter, FrameFilter.SetFloatParameter, FrameFilter.GetFloatParameter, the following basic data types can be read from or written to the filter: bool, long, float and string. A block of binary data can be exchanged with a frame filter with the FrameFilter.SetDataParameter and FrameFilter.GetDataParameter methods. Since the transform method of a filter is executed within a DirectShow thread, it is very important to synchronize external and internal access to the filter's data. Therefore, it is very important that a sequence of calls to the four methods mentioned above is preceded by a call to FrameFilter.BeginParameterTransfer and is followed by a call to FrameFilter.EndParameterTransfer:

If a frame filter is used by generic applications, such as the Filter Inspector, a dialog should be implemented that allows the parameters to be altered interactively. A frame filter indicates that it provides a parameter dialog by returning true in the method FrameFilter.HasDialog. The dialog should be displayed when the method FrameFilter.ShowDialog is called.

To serialize the settings of a frame filter, use the FrameFilter.FilterData property. To save the settings directly to a file, use the FrameFilter.SaveFilterData and FrameFilter.LoadFilterData methods.

<< Technical Articles