Most FxFactory plug-ins outside the Pro Pack are designed to consume and produce RGB + alpha images at 8 bit per channel. This reflects the general trend of all third-party plug-ins for Final Cut Studio, and a testament to the dominance of output formats where 8 bit of color information per channel provides enough room for presenting video. Because FxFactory is based on FxPlug, it also allows plug-ins to support 16 bit "half-float" and 32 bit float rendering in RGBA. In Final Cut Pro, the same color depths are available for native YUV (Y'CbCr) processing as well.

The first important distinction to be made is the difference between the number of bits per channel used for representation and the number of bits per channel used for computation. Equally important is the distinction between integer and floating point methods of numeric representation and computation. It is generally understood that floating point is much better suited for computation of effects, but for historical reasons there exists mainstream graphics software which still uses integer methods for computation (it consumes less memory and used to run faster, at the expense of quality).
Although an FxFactory plug-in may be designed for 8 bit RGB input/output in integer format (where the value of each channel goes from 0 to 255), the bulk of the effects are based on Core Image, a graphics technology designed by Apple whose computations happen at 32 bit float. On modern graphics hardware, this means that intermediate steps of an effect are computed and stored using 32 bit float values for each channel. Moreover, these computations happen entirely on the GPU, thus enabling effects to be rendered faster, and with equal precision to the traditional approaches.

Whereas internal plug-in computations are hidden from Final Cut Pro and Motion, each plug-in still has a choice of capabilities to "advertise". The basic setup provided by FxFactory ("RGB + alpha, 8 bit integer") should prove sufficient for most plug-in development, but there are situations where the better quality provided by high-precision input/output may be desired.

In Motion, the high-precision rendering pipeline is enabled via the "Bit Depth" popup menu, in the Project properties window:

Motion only supports the RGB color format. Because the Motion rendering engine is designed around modern graphics hardware, Motion supports the 16 bit float option (commonly referred to as "half-float"). This option is not available in Final Cut Pro, since regular processors cannot handle this format natively.

In Final Cut Pro, the high-precision rendering pipeline is enabled by turning on the "Render all YUV material in high-precision YUV" option under the "Video Processing" tab in the Sequence Settings window.

When the option is enabled, Final Cut Pro will attempt to process images at 32 bit float precision. Despite the misleading name of that option, Final Cut Pro will also feed plug-ins 32 bit float RGBA data, whenever the plug-in is incapable of handling YUV color information directly. More details on the Final Cut Pro approach to rendering third-party effects may be found here.

High-precision Rendering in Quartz Composer

It won't suffice to turn on high-precision rendering in the host in order to get better-quality effects. In fact, turning on high-precision rendering for 8 bit capable plug-ins will simply force Motion and Final Cut Pro to down-sample all color information as it gets fed to the plug-in (potentially losing information along the way). More changes are required to the plug-in itself, starting with the options available under the "Renderer" section:

The capabilities of a plug-in are changed via options under the Renderer section.

The plug-in will advertise support for each selected color format to the host. Due to certain limitations of current graphics hardware, the 16 bit half-float format is the recommended way to support high-precision rendering. Not surprisingly, FxFactory is designed to map 32 bit float inputs/output to 16 bit float, in order to get the best compromise between high-precision rendering and graphics card features.

All FxFactory plug-ins are based on a Quartz Composition. Quartz Composer currently allows only one way to process floating point images directly. A composition should render its output using only the "Billboard" renderer patch:

Moreover, the Billboard patch settings should have the "Enable Direct Core Image Rendering" option enabled:

Only when these two conditions are met a plug-in may accept (and produce) images using floating point precision. Other factors may play a role, such as the complexity of the effect chain and the type of patches used by the composition. When other types of renderer are used, or when the "Render in Image" patch is used, Quartz Composer will render the intermediate results at 8 bit integer, thus losing color information along the way. This holds true for all Quartz compositions rendered under Mac OS X 10.4.x "Tiger".

Because of limitations in graphics hardware, certain features may not work consistently when floating point textures are being used. Generally speaking, the more complicated a composition is, the less likely it is to run correctly on older graphics cards. (At the time of this writing, the best consumer card available for desktop Macs is the ATI Radeon X1900, which is quite capable of rendering complex scenes at 16 bit half-float.)

When YUV support is enabled, the composition will be fed YUV + alpha information rather than the usual RGB + alpha pixels. Under this scenario, the meaning of the values in each channel are given by the particular format used by Final Cut Pro to encode Y'CbCr data, known as r4fl.

It is generally hard to design Quartz compositions that can handle YUV data correctly, since Core Image was not designed for this purpose. When in doubt, it is always best to declare support for high-precision images in RGB + alpha only, and let Final Cut Pro perform the conversions.