ImageSimilarity Class

Every member reads the pixels as 8-bit RGBA, whatever the pixel format (except the overloads of SSIM and PSNR that take ImageSimilarityOptions, and ImageSimilarity.CreateDifferenceMap), with the conversion of Image.CloneAs: 16-bit samples are rounded to 8 bits, gray samples are copied to the three color channels, and formats without alpha are opaque. Floating-point samples (RgbaFloat, RgbaHalf) are rounded to 8 bits as they are stored, whatever their transfer function label, like the samples of the other formats: a sample outside [0, 1], or that is not a number, has no 8-bit value and is rejected with an UnsupportedImageFeatureException, never clipped (convert such an image with Image.CloneAs and PixelConversionOptions.OutOfRangeSamples or PixelConversionOptions.ToneMapping first). Two frames of different pixel formats can be compared. Nothing is written to the pixels; the frames are leased for the duration of the call, so a frame cannot be measured while a lease is active on it. A frame can be compared with itself.

The results are deterministic: SSIM and the hashes depend only on the pixels, not on the hardware. PSNR, SSIMULACRA 2 and Butteraugli use the logarithm, power and cube root functions of the platform, so their last digits can differ between platforms. The temporary buffers are charged to the allocation scope of the image of the frames. An animated image is compared frame by frame.

SSIM. The score is computed in every 7×7 window that fits inside the frame, moved one pixel at a time, with uniform weights, the sample variance and covariance, and K1 = 0.01, K2 = 0.03, L = 255: for opaque frames, it is the mean over the three color channels of what scikit-image's structural_similarity(expected, actual, data_range=255, channel_axis=-1) computes. A frame narrower or shorter than 7 pixels uses a window as wide or as tall as the frame. The color channels are premultiplied by alpha, so a fully transparent pixel scores the same whatever color it hides; the alpha channel is scored on its own, and the result is the lower of the color score and the alpha score.

Hashes. Both hashes start from an exact, area-weighted luminance thumbnail (BT.601 weights; 9×8 cells for the dHash, 32×32 for the pHash), so a change anywhere in the frame affects it. A dHash bit is set when a cell is brighter than its right neighbor by more than half a luminance level; a pHash bit is set when a low-frequency DCT coefficient of the thumbnail exceeds the median coefficient by more than one level. These tolerances keep flat areas from flipping bits on invisible noise. The hashes describe the structure, not the brightness: a hash distance is therefore the Hamming distance plus the difference between the mean luminances on the same scale (about 4 levels out of 255 per unit). A frame that is not fully opaque is evaluated composited over black and over white, and the larger distance is kept. The distance is capped at ImageSimilarity.MaxHashDistance.

PSNR. The ratio, in decibels, of the peak value (255) to the mean squared error of the R, G and B channels premultiplied by alpha (as for SSIM): for opaque frames, it is what scikit-image's peak_signal_noise_ratio(expected, actual, data_range=255) computes. The alpha channel gives its own ratio, and the result is the lower of the two. Identical frames have an infinite ratio.

SSIMULACRA 2. A port of the ssimulacra2 tool of libjxl 0.12.0, a perceptual metric tuned on subjective scores: the frames are compared in the XYB color space of JPEG XL at 6 scales, with an SSIM map and two edge maps (new artifacts and lost details). The planes are 32-bit floats, and the arithmetic of the tool is reproduced (its sRGB curve, cube root and blur, with fused multiply-adds): the scores agree with the 8 decimals it prints. A frame that is not fully opaque is composited over a dark and a bright gray, and the lower score is kept, as the tool does.

Butteraugli. A port of the Butteraugli metric of libjxl 0.12.0 (by Jyrki Alakuijala), with the parameters of its butteraugli_main tool for SDR images (80 nits): a psychovisual model of the differences per frequency band, masked by the activity of the frames, that gives a distance for every pixel. The results are within about 0.01% of the tool. A frame that is not fully opaque is composited over black and over white, and every pixel keeps the larger distance, as libjxl does. A frame smaller than 8×8 is extended to 8×8 by repeating its edges, as libjxl's ButteraugliDiffmap does. It is the slowest member, and it needs about 120 bytes per pixel.

Parallelism. SSIM, PSNR, SSIMULACRA 2 and Butteraugli use up to ImageConfiguration.MaxDegreeOfParallelism workers (the configuration of the image of the expected frame) for frames of at least 65,536 pixels, and the result is identical whatever the number of workers: the floating-point sums are made by bands of 64 rows, added in order. With several workers, each one has its own row buffers.