Frame Filters
With frame filters, a versatile and powerful way to manipulate image data is
introduced. Besides other features, frame filters can be used to change or convert the video format
of the image stream, implement image processing and control which frames are processed.
Introduction
In contrast to other DirectShow libraries, the frame filters of IC Imaging Control are not
DirectShow filters. But they are executed in the context of a DirectShow thread that processes the
image stream. This saves the programmer the hassle of creating a DirectShow filter module,
registering it and defining interfaces for the parameters. Frame filters are implemented by deriving from
the class FrameFilterImpl or FrameUpdateFilterImpl. Easy access to the
parameters of a frame filter is provided by the class IFrameFilter. On the one hand,
frame filters may be implemented as part of a filter module. On the other hand, a frame filter can be created
directly in an application, if its class is known.
Filter Categories
Frame filters open the door to a vast number of new IC Imaging Control based applications.
In order to show you what is possible with this new concept, we discuss categories of tasks that can be implemented
with frames filters. For some of these tasks, frame filters are shipped with IC Imaging Control.
- Gate:
-
Gate filters drop unwanted frames based on image analysis or other information like the frame count or
time stamps.
- Color Space Conversion:
-
All kinds of transformations from one color format to another fall in to this category. In addition,
the reduction of the pixel depth, e.g. extracting the 8 most significant bits out of a 10 or 12 bit
monochrome format is regarded as a color space conversion. The reduction of the pixel depth is used
to display a 10 or 12 bit monochrome image.
A good example for a color space conversion is the
DeBayer Filter
contained in stdfilters.ftf. It converts color raw data in Bayer format to RGB.
- ROI:
-
Filters in this category allow a Region Of Interest to be cropped out of an image stream.
The ROI Filter that is contained in stdfilters.ftf implements this functionality.
- Rotation:
-
This category includes rotation and flipping. The Rotate Flip Filter that is contained
in stdfilters.ftf provides rotation by 90°, 180° and 270°, as well as flipping on the horizontal
and vertical axis. There is a filter in the DemoFilter.ftf that provides rotation by any angle, but this
filter is a demo implementation and very slow.
- Scaling:
-
Scaling up and down may be necessary to match a predefined display size or to save disk space for
surveillance applications.
- Sink:
-
Sink filters are consumers. They process the image stream in their transform method
completely and do not send frames to the IC Imaging Control sink. If you want to use professional high
performance codecs, you would implement a sink filter in order to interface the codec API. This approach
is necessary because most professional codecs come as a DLL instead of a DirectShow codec.
These codecs provides functionality to compress image data
and write it to a file at the same time, which makes a sink obsolete. Of course, you could use
the MembufferCollection of a FrameHandlerSink to access the image
data and send it to the codec API. In this case, the image data would have been copied by the sink to
the MembufferCollection which is not the case, if the codec is interfaced using a
frame filter. In other words, the sink filter approach saves one copy operation per frame, which can
increase the performance significantly, especially if large frames are processed.
- Extraction:
-
Extraction filters take specific frames out of the image stream and apply processing on them, e.g.
saving them to disk as image files. The image stream itself remains unchanged and is sent further
to the sink. Such a filter allows specific images to be saved uncompressed, while a
MediaStreamSink generates a compressed video file.
- Image Processing:
-
All image processing functions that produce one image as output can be implemented easily by a
frame filter. For functions
that produce more than one image, special concepts have to be developed. One possible concept would
be to write the output images one after the other into the resulting image. If data other than images is
generated by the function (e.g. data generated by a blob analysis), the application could read out
this data by using filter parameters.
- Buffer Format Conversion:
-
Most image processing libraries have a special internal image buffer format. Frame filters can be used
to convert image buffers of the image stream to such a special image buffer format. Doing so, the
elements in the ring buffer of the FrameHandlerSink contain data in the special format.
Therefore, the data of these buffers may be passed directly to the according image processing library.
This saves one copy step and, therefore, increases the performance of the system.
Filter Types
There are 2 basic types of frame filters which will be discussed in the following. Both types provide the
same way of parameter access and allow frames to be dropped explicitly.
- Transform Filter
-
This type of frame filter has to be derived from FrameFilterImpl
and allows to implement a conversion from one video format to another. This includes changing the color
format (e.g. eY800 to RGB24), as well as changing the width and height.
- Update Filter
- This type of filter has to be derived from FrameUpdateFilterImpl.
In contrast to a transform filter, an update filter may not change the color format nor the width and height.
In order to speed up the processing of the image stream, an update filter can tell the framework that it
will only read image data. This allows the framework to make certain optimizations. As a rule of thumb,
update filters should only be used, if the image data is not changed at all or only a few pixels are modified. This
is because the framework has to copy image buffers before it calls the
FrameUpdateFilterImpl::updateInPlace method, if the filter has indicated that it will modify
image data by returning true in the method
FrameUpdateFilterImpl::modifiesData.
Using Existing Frame Filters
As described earlier, frame filters can be created directly in an application or loaded from a filter module.
IC Imaging Control provides some filters in the module stdfilters.ftf. In order to
obtain a list of the frame filters in all currently available filter modules, call the method
FilterLoader::getAvailableFrameFilters. With the method FilterLoader::setLoadPath,
an additional path for filter modules may be specified. A call to FilterLoader::createFilter
creates an instance of a filter that was selected from the list of available filters.
Now that we have a frame filter instance, we can insert it at three different locations in the image stream:
Please note that the same instance of a frame filter should not be used at more than one place in the image
stream because its internal state may get inconsistent, especially if FrameFilterImpl::notifyStart
and FrameFilterImpl::notifyStop are used. If it is necessary to insert the same filter at more
than one location in the image stream, different instances of the frame filter should be used.
The internal data of a frame filter can be accessed through the parameter interface. With the method
IFrameFilter::getParameter and IFrameFilter::setParameter, the following basic
data types can be read from or written to the filter: bool, int, long float and std::string. A block of
binary data can be exchanged with a frame filter by the methods IFrameFilter::getData
and IFrameFilter::setData. Since the transform method of a filter is executed within a
DirectShow thread, it is very important to synchronize external and internal access to the filter's data.
Therefore, it is very important that a sequence of calls to the four methods
mentioned above is preceded by a call to IFrameFilter::beginParamTransfer and is
followed by a call to IFrameFilter::endParamTransfer:
pFrameFilter->beginParamTransfer();
// All calls to setParameter and getParameter here...
pFrameFilter->endParamTransfer();
If a frame filter is used by generic applications, such as the Filter Inspector,
a dialog should be implemented that allows the parameters to be altered interactively. A frame filter
indicates that it provides a parameter dialog by returning true in the method
IFrameFilter::hasDialog. The dialog should be displayed when the method
IFrameFilter::callDialog is called.
Every frame filter should provide the serialization of its internal data by implementing the methods
IFrameFilter::getSettings and IFrameFilter::setSettings. The representation
of the data has to be string compatible.
<< Technical Articles