# Better Derivative Estimates by Pre-filtering

Numerical Methods for Derivative Estimation

In this section, we want to consider a *pre-filtering*
strategy that eliminates any undesirable frequencies from the
input data stream prior to applying the Central Difference derivative
estimator. If there is no high frequency disturbance, the estimator
will have nothing to amplify. But before committing
to this idea fully, let's first test the concept in a relatively
simple, brute-force manner. We will construct a lowpass digital
filter that clears away high frequencies, but leaves low frequencies
intact. Then, apply the Central Difference formula.

## Selecting a lowpass filter

We want the input data set to be as undamaged as possible in
the frequency band below the "20% of Nyquist Limit." Thus,
the filter should not attenuate or shift the data in
that band. But above that 20% limit, as much response as possible
should be attenuated away. The tricky part is getting an appropriate
*flat lowpass filter characteristic* without excessive
computational effort.

This is a nontrivial design problem, and this is not a tutorial in digital filter design, so it is not feasible to present all the details here. The filter characteristic is intended to satisfy the following criteria:

- preserve frequencies below 20% of the Nyquist limit
- attenuate frequencies above 20% of the Nyquist limit
- vigorously attenuate frequencies above 50% of the Nyquist limit

Preconditioning filter for derivative estimation (29 terms)

-0.0000106 -0.0000634 0.0002528 0.0011268 0.0000000 -0.0049328 -0.0054297 0.0090374 0.0231928 -0.0000000 -0.0542194 -0.0518104 0.0862069 0.2966433 0.4000126 0.2966433 0.0862069 -0.0518104 -0.0542194 -0.0000000 0.0231928 0.0090374 -0.0054297 -0.0049328 0.0000000 0.0011268 0.0002528 -0.0000634 -0.0000106

The form of the filter is a "linear convolution filter" very much like the "Central Difference" estimator filter, except with an even symmetry instead of an odd symmetry. To apply this, align the center of the filter to the point in your data set where you want a filtered value. Multiply the aligned terms from your data set with the corresponding filter coefficients, pairwise, and sum these intermediate products to obtain the smoothed data stream value.

## Results of prefiltering the input stream

The preconditioning filter is first applied to the "noisy" data stream, and then the Central Differences derivative estimator is applied to the cleaned data stream.

Here is signal data before (green) and after the pre-filtering (blue). Notice that the changes are subtle.

Now compare the derivative estimates obtained using the pre-filtered data (in blue) to the derivative estimates obtained using the original unfiltered data (red).

The results with filtered data show a definite improvement. However, it is unclear how exactly to quantify the benefit, or how to determine necessary filter adjustments to best align with the performance objectives of the derivative analysis.

## Further improvements?

It seems like we should be able to do better. The cost of the two-stage pre-filtered estimation is rather high. The experimental lowpass filter used 29 filter coefficients, quite a lot compared to the Central Differences estimator which uses 7.

Both filters have a similar "linear convolution kernel" form. Perhaps it is possible to construct a single hybrid filter characteristic that achieves both the derivative estimation and noise rejection objectives simultaneously, and maybe more efficiently as well? We are not done yet.

Footnotes: