Blur, sharpen and convolution

These operations compute each pixel from its neighbors. They keep the size, the pixel format, the frames and the metadata of the image, apply to every frame and to the poster frame, and also exist for one frame.

Blur and sharpen

public static void CleanUpScan(string inputPath, string cleanedPath, string backgroundPath)
{
    using var image = Image.Load(inputPath);

    // The median of every 3x3 square removes isolated specks and keeps edges. The unsharp mask then adds half the
    // difference from a Gaussian blur of deviation 1.5, except where it is under 2% of the range: flat paper stays flat
    image.MedianBlur(1);
    image.Sharpen(1.5, amount: 0.5, threshold: 0.02);
    image.Save(cleanedPath);

    // A box blur costs the same whatever its radius; a separable matrix is applied to the rows, then to the columns
    image.BoxBlur(25, ConvolutionEdgeMode.Mirror);
    image.Convolve(new ConvolutionOptions(ConvolutionKernel.CreateSeparable([0.25, 0.5, 0.25], [0.25, 0.5, 0.25])));
    image.Save(backgroundPath);
}

• The threshold of Sharpen keeps flat areas and their noise as they are, while the details get more contrast. Its result is clamped to the range of the samples: the ringing beyond black and white is clipped, except for the colors of floating-point samples.
• GaussianBlur, BoxBlur and Sharpen take an edge mode and a working space, as a convolution does (see below). MedianBlur takes an edge mode and filters every sample, alpha included, as stored: it computes nothing, it only picks one of the samples. So does PercentileFilter, with the same cost.

Add a motion blur

ConvolutionKernel.CreateMotionBlur gives the matrix of a straight trail of length pixels at an angle (clockwise from the horizontal, as Rotate turns), for Convolve: the blur of a camera or a subject that moved while the shutter was open. Every pixel weighs the length of the trail inside it, so the brightness is kept, and the trail is centered on each pixel, so the image does not move. The cost grows with the number of pixels the trail crosses.

Smooth noise and keep the edges

A blur softens the edges as much as the noise. BilateralFilter and GuidedFilter smooth what varies a little (sensor noise, compression artifacts, skin texture) and keep what varies a lot (the edges):

public static void DenoisePhoto(string inputPath, string outputPath)
{
    using var photo = Image.Load(inputPath);

    // The bilateral filter averages the pixels of a disk of radius 4 whose colors differ by less than about 8% of the
    // range (20 levels of 255): noise is smoothed, edges are not
    photo.BilateralFilter(radius: 4, spatialSigma: 2, rangeSigma: 0.08);

    // The guided filter flattens what varies less than √epsilon (here 5%) in windows of 17x17 pixels, in a constant
    // time per pixel whatever the radius
    photo.GuidedFilter(radius: 8, epsilon: 0.0025);
    photo.Save(outputPath);
}

On gray images, the bilateral filter gives the result of OpenCV's cv2.bilateralFilter, and the guided filter the result of cv2.ximgproc.guidedFilter with the edge mode Mirror. Colors are filtered premultiplied by alpha, so the hidden color of a transparent pixel does not spread.

Add a zoom or a spin blur

RadialBlur blurs along lines through a center (RadialBlurKind.Zoom, the streaks of a zoom during the exposure) or along circles around it (RadialBlurKind.Spin, a rotation). The center stays sharp, and the streaks grow with the distance to it:

public static void ZoomBurst(string inputPath, string outputPath)
{
    using var photo = Image.Load(inputPath);

    // Every pixel is blurred toward the center over 20% of its distance to the center; a spin blur
    // (RadialBlurKind.Spin) turns around the center by an angle instead
    var center = new PointD(photo.Width / 2.0, photo.Height / 2.0);
    photo.RadialBlur(new RadialBlurOptions(RadialBlurKind.Zoom, center, amount: 0.2));
    photo.Save(outputPath);
}

The Amount is a fraction of the distance to the center for a zoom, and an angle in degrees for a spin. The center is in pixel coordinates, where the center of a 100x80 image is (50, 40), and can be outside the image. Long streaks are sampled at most one pixel apart, up to MaxSampleCount samples per pixel (64 by default): raise it for very long streaks, at the cost of time.

Apply a convolution matrix

Convolve replaces every sample by the weighted sum of the same sample of the neighboring pixels: sharpening, blurring, embossing and edge detection are all matrices. A ConvolutionKernel has odd dimensions, its middle weight is the one of the pixel itself, and its weights are given row by row:

public static void SharpenAndDetectEdges(string inputPath, string sharpenedPath, string edgesPath)
{
    using var image = Image.Load(inputPath);

    // The matrix is applied as written, row by row; weights that add up to 1 keep the overall brightness
    var sharpen = new ConvolutionKernel(3, 3, [0, -1, 0, -1, 5, -1, 0, -1, 0]);
    using (var sharpened = image.Clone())
    {
        sharpened.Convolve(new ConvolutionOptions(sharpen));
        sharpened.Save(sharpenedPath);
    }

    // Weights that add up to 0 would also filter alpha down to 0 (transparent black): keep the alpha instead, and
    // add half of the range so that negative sums show as darker than mid-gray instead of black
    var options = new ConvolutionOptions(ConvolutionKernel.Laplacian3x3) { PreserveAlpha = true, EdgeMode = ConvolutionEdgeMode.Mirror, Bias = 0.5 };
    image.Convolve(options);
    image.Save(edgesPath);
}

The matrix is applied as written: the top-left weight multiplies the top-left neighbor (it is not flipped), and the weights are used as given. Divide them by their sum to keep the overall brightness. ConvolutionKernel has the published matrices SobelX, SobelY, ScharrX, ScharrY, PrewittX, PrewittY (horizontal and vertical gradients), Laplacian3x3, Laplacian5x5, LaplacianOfGaussian, Emboss and Sharpen3x3.

ConvolutionOptions chooses:

The sums are computed in double precision at the storage precision, then rounded and clamped to the range of the samples: negative sums become 0. A separable matrix, whose weights are the products of a row and a column of weights (a box, a binomial, Sobel), is applied as two passes, rows then columns: its cost grows with the sum of its dimensions instead of their product. ConvolutionKernel.CreateSeparable(rowWeights, columnWeights) makes one, and IsSeparable tells whether a matrix is one.

Detect edges

DetectEdges(detector) replaces every frame by its edges: bright where the colors change, black where they are flat:

public static void DetectEdges(string inputPath, string edgesPath)
{
    using var image = Image.Load(inputPath);

    // White edges on black, computed on the brightness (grayscale: false keeps the edges of every color); alpha is kept
    image.DetectEdges(EdgeDetector.Sobel);
    image.Save(edgesPath);
}

By default, DetectEdges converts every frame to grayscale first, which gives white edges on black; with grayscale: false, every color has its own edges. Alpha is kept. The edge mode Clamp (default) finds no edge along the border of the image, and Zero reads black beyond it.

Precision and cost

16-bit images are filtered with 16-bit precision, and the colors of floating-point images are not clamped. The working memory is charged to the allocation limit of the image. Convolve, the blurs and Sharpen rent it before any pixel changes, so an exceeded limit leaves the image unchanged. So do the edge-preserving filters and RadialBlur, which also copy each frame as single-precision samples. The EXIF thumbnail, now stale, is removed.

The cost of GaussianBlur grows with its deviation, the cost of BoxBlur does not depend on its radius while its square fits in the image (beyond, each row and column costs a time proportional to the radius), and the cost of a matrix grows with its number of non-zero weights, unless it is separable. Large frames can be split between several threads: see Performance.