| # Corruption Detection | |
| **Name:** | |
| "Corruption Detection"; "Extension for Automatic Detection of Video Corruptions" | |
| **Formal name:** | |
| <http://www.webrtc.org/experiments/rtp-hdrext/corruption-detection> | |
| **Status:** This extension is defined here to allow for experimentation. | |
| **Contact:** <sprang@google.com> | |
| NOTE: This explainer is a work in progress and may change without notice. | |
| The Corruption Detection (sometimes referred to as automatic corruption | |
| detection or ACD) extension is intended to be a part of a system that allows | |
| estimating a likelihood that a video transmission is in a valid state. That is, | |
| the input to the video encoder on the send side corresponds to the output of the | |
| video decoder on the receive side with the only difference being the expected | |
| distortions from lossy compression. | |
| The goal is to be able to detect outright coding errors caused by things such as | |
| bugs in encoder/decoders, malformed packetization data, incorrect relay | |
| decisions in SFU-type servers, incorrect handling of packet loss/reordering, and | |
| so forth. We want to accomplish this with a high signal-to-noise ratio while | |
| consuming a minimum of resources in terms of bandwidth and/or computation. It | |
| should be noted that it is _not_ a goal to be able to e.g. gauge general video | |
| quality using this method. | |
| This explainer contains two parts: | |
| 1) A definition of the RTP header extension itself and how it is to be parsed. | |
| 2) The intended usage and implementation details for a WebRTC sender and | |
| receiver respectively. | |
| If this extension has been negotiated, all the client behavior outlined in this | |
| doc MUST be adhered to. | |
| ## RTP Header Extension Format | |
| ### Data Layout Overview | |
| The message format of the header extension: | |
| 0 1 2 3 | |
| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 | |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |
| |B| seq index | std dev | Y err | UV err| sample 0 | | |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |
| | sample 1 | sample 2 | … up to sample <=12 | |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |
| ### Data Layout Details | |
| * B (1 bit): If the sequence number should be interpreted as the MSB or LSB | |
| of the full size 14 bit sequence index described in the next point. | |
| * seq index (7 bits): The index into the Halton sequence (used to locate | |
| where the samples should be drawn from). | |
| * If B is set: the 7 most significant bits of the true index. The 7 least | |
| significant bits of the true index shall be interpreted as 0. This is | |
| because this is the point where we can guarantee that the sender and | |
| receiver has the same full index. B MUST be set on keyframes. On droppable | |
| frames B MUST NOT be set. | |
| * If B is not set: The 7 LSB of the true index. The 7 most significant bits | |
| should be inferred based on the most recent message. | |
| * std dev (8 bits): The standard deviation of the Gaussian filter used | |
| to weigh the samples. The value is scaled using a linear map: | |
| 0 = 0.0 to 255 = 40.0. A std dev of 0 is interpreted as directly using | |
| just the sample value at the desired coordinate, without any weighting. | |
| * Y err (4 bits): The allowed error for the luma channel. | |
| * UV err (4 bits): The allowed error for the chroma channels. | |
| * Sample N (8 bits): The N:th filtered sample from the input image. Each | |
| sample represents a new point in one of the image planes, the plane and | |
| coordinates being determined by index into the Halton sequence (starting at | |
| seq# index and is incremented by one for each sample). Each sample has gone | |
| through a Gaussian filter with the std dev specified above. The samples | |
| have been floored to the nearest integer. | |
| A special case is the so-called "synchronization" message. Such a message | |
| only contains the first byte. They are used to keep the sender and receiver in | |
| sync even if no "full" message has been received for a while. Such messages | |
| MUST NOT be sent on droppable frames. | |
| ### A note on encryption | |
| Privacy and security are core parts of nearly every WebRTC-based application, | |
| which means that some sort of encryption needs to be present. The most common | |
| form of encryption is SRTP, defined in RFC 3711. However, as mentioned in | |
| section 9.4 of that RFC, RTP header extensions are considered part of the header | |
| and are thus not encrypted. | |
| The automatic corruption detection header extension is different from most other | |
| header extensions in that it provides not only metadata about the media stream | |
| being transmitted but in practice comprises an extremely sparse representation | |
| of the actual video stream itself. Given a static scene and enough time, a crude | |
| image of the encrypted video can rather trivially be constructed. | |
| As such, most applications should use this extension with SRTP only if | |
| additional security is present to protect it. That could be for example in the | |
| form of explicit header extension encryption provided by RFC 6904/RFC 9335, or | |
| by encapsulating the entire RTP stream in an additional layer such as IPSec. | |
| ## Usage & Guidelines | |
| In this section we’ll first look at a general overview of the intended usage of | |
| this header extensions, followed by more details around the expected | |
| implementation. | |
| ### Overview | |
| The premise of the extension described here is that we can validate the state of | |
| the video pipeline by quasi-randomly selecting a few samples from the raw input | |
| frame to an encoder, and then checking them against the output of a decoder. | |
| Assuming that a lossless codec is used we can follow these steps: | |
| 1) In an image that is to be encoded, quasi-randomly select N sampling positions | |
| and store the samples values for those positions from the raw input image. | |
| 2) Encode the image, and attach the selected sample values to the RTP packets | |
| containing the encoded bitstream of that image. | |
| 3) Transmit the RTP packets to a remote receiver. | |
| 4) At the receiver, collect the attached sample values from the RTP packets when | |
| assembling the frame, and then pass the bitstream to a decoder. | |
| 5) Using the same quasi-random sequence as in (1), calculate the corresponding N | |
| sampling positions. | |
| 6) Take the output of the decoder and check the values of the samples from the | |
| RTP packets. If they differ significantly, it is likely that an image | |
| corruption has occurred. | |
| Lossless encoding is however rarely used in practice, and that introduces | |
| problems for the above algorithm. | |
| * Quantization causes values to be different from the desired value. | |
| * Whole blocks of pixels might be shifted somewhat due to inaccuracies in motion | |
| vectors. | |
| * Inaccuracies caused by in-loop or post-process filtering. | |
| * etc. | |
| We must therefore take these distortions into consideration, as they are merely | |
| a natural side-effect of the compression and their effect is not to be | |
| considered an “invalid state”. We aim to accomplish this using two tools. | |
| First, instead of a sample being a single raw sample value let it be a filtered | |
| one: a weighted average of samples in the vicinity of the desired location, with | |
| the weights being a 2D Gaussian centered at that location and the variance | |
| adjusted depending on the magnitude of the expected distortions | |
| (higher distortion => higher variance). This smoothes out inaccuracies caused by | |
| both quantization and motion compensation. | |
| Secondly, even with a very large filter kernel the new sample might not converge | |
| towards the exact desired value. For that reason, set an “allowed error | |
| threshold” that removes small magnitude differences. Since chroma and luma | |
| channels have different scales, separate error thresholds are available for | |
| them. | |
| ### Sequence Index Handling | |
| The quasi-random sequence of choice for this extension is a 2D | |
| [Halton Sequence](https://en.wikipedia.org/wiki/Halton_sequence). | |
| The index into the Halton Sequence is indicated by the header extension and | |
| results in a 14 bit unsigned integer which on overflow will wrap around back to | |
| 0. | |
| For each sample contained within the extension, the sequence index should be | |
| considered to be incremented by one. Thus the sequence index at the start of the | |
| header should be considered “the sequence index for the next sample to be | |
| drawn”. | |
| The ACD extension may be sent containing either the 7 most significant bits | |
| (B = true) or the 7 least significant bits (B = false) of the sequence index. | |
| Key-frames MUST be populated with the ACD extension, and those MUST use B = true | |
| indicating only the 7 most significant bits are transmitted. | |
| The sender may choose any arbitrary starting point. The biggest reason to not | |
| always start with (B = true, seq index = 0) is that with frequent/periodic | |
| keyframes you might end up always sampling the same small subset of image | |
| locations over and over. | |
| If B = false and the LSB seq index + number of samples exceeds the capacity of | |
| the 7-bit field (i.e. > 0x7F), then the most significant bits of the 14 bit | |
| sequence counter should be considered to be implicitly incremented by the | |
| overflow. | |
| Delta-frames may be encoded as “droppable” or “non-droppable”. Consider for | |
| example temporal layering using the | |
| [L1T3](https://www.w3.org/TR/webrtc-svc/#L1T3*) mode. In that scenario, | |
| key-frames and all T0 frames are non-droppable, while all T1 and T2 frames are | |
| droppable. | |
| For non-droppable frames, B MAY be set to true even though there is often little | |
| utility for it. | |
| For droppable frames B MUST NOT be set to true, since a receiver could otherwise | |
| easily end up out of sync with the sender. | |
| A receiver must store a state containing the last sequence index used. If an ACD | |
| extension is receiver with B = false but the LSB does not match the last known | |
| sequence index state, this indicates that an instrumented frame has been | |
| dropped. The receiver should recover from this by incrementing the last known | |
| sequence index until the 7 least significant bits match. | |
| Because of this, the sender MUST send ACD messages on non-droppable frames such | |
| that the delta between their sequence indexing (from the last sample of the | |
| previous packet to the first of the next) indexing does not exceed 0x7F. A | |
| synchronization message may be used for this purpose if there is no wish to | |
| instrument the non-droppable frame. | |
| It is not required to add the ACD extension to every frame. Indeed, for | |
| performance reasons it may be reasonable to only instrument a small subset of | |
| frames, for example using only one frame per second. | |
| Additionally, when encoding a structure that has independent decode targets | |
| (e.g. L3T3_KEY) - the sender should generate an independent stream ACD sequence | |
| per target resolution so that a receiver can validate the state of the | |
| sub-stream they receive. | |
| // TODO: Add concrete examples. | |
| ### Sample Selection | |
| As mentioned above, a Halton Sequence is used to generate sampling coordinates. | |
| Base 2 is used for selecting the rows, and base 3 is used for selecting columns. | |
| Each sample in the ACD extension represents a single image sample, meaning it | |
| belongs to a single channel rather than e.g. being an RGB pixel. | |
| The initial version of the ACD extension supports only the I420 chroma | |
| subsampling format. When determining which plane a location belongs to, it is | |
| easiest to visualize it as the chroma planes being “stacked” to the side of the | |
| luma plane: | |
| +------+---+ | |
| | | U | | |
| + Y +---+ | |
| | | V | | |
| +------+---+ | |
| In pseudo code: | |
| ``` | |
| row = GetHaltonSequence(seq_index, /*base=*/2) * image_height; | |
| col = GetHaltonSequence(seq_index, /*base=*/3) * image_width * 1.5; | |
| if (col < image_width) { | |
| HandleSample(Y_PLANE, row, col); | |
| } else if (row < image_height / 2) { | |
| HandleSample(U_PLANE, row, col - image_width); | |
| } else { | |
| HandleSample(V_PLANE, row - (image_height / 2), col - image_width); | |
| } | |
| seq_index++; | |
| ``` | |
| Support for other layout types may be added in later versions of this extension. | |
| Note that the image dimensions are not explicitly a part of the ACD extension - | |
| that has to be inferred from the raw image itself. | |
| ### Sample Filtering | |
| As mentioned above, when filtering a sample we create a weighted average around | |
| the desired location. Only samples in the same plane are considered. The | |
| weighting consists of a 2D Gaussian centered on the desired location, with the | |
| standard deviation specified in the ACD extension header. | |
| If the standard deviation is specified as 0.0 - we consider only a singular | |
| sample. Otherwise, we first determine a cutoff distance below which the weights | |
| are considered too small to matter. For now, we have set the weight cutoff to | |
| 0.2 - meaning the maximum distance from the center sample we need to consider is | |
| max_d = ceil(sqrt(-2.0 * ln(0.2) * stddev^2) - 1. | |
| Any samples outside the plane are considered to have weight 0. | |
| In pseudo-code, that means we get the following: | |
| ``` | |
| sample_sum = 0; | |
| weight_sum = 0; | |
| for (y = max(0, row - max_d) to min(plane_height, row + max_d) { | |
| for (x = max(0, col - max_d) to min(plane_width, col + max_d) { | |
| weight = e^(-1 * ((y - row)^2 + (x - col)^2) / (2 * stddev^2)); | |
| sample_sum += SampleAt(x, y) * weight; | |
| weight_sum += weight; | |
| } | |
| } | |
| filtered_sample = sample_sum / weight_sum; | |
| ``` | |
| ### Receive Side Considerations | |
| When a frame has been decoded and an ACD message is present, the receiver | |
| performs the following steps: | |
| * Update the sequence index so that it is consistent with the ACD message. | |
| * Calculate the sample positions from the Halton sequence. | |
| * Filter each sample of the decoded image using the standard deviation provided | |
| in the ACD message. | |
| We then need to compare the actual samples present in the ACD message and the | |
| samples generated from the locally decoded frame, and take the allowed error | |
| into account: | |
| ``` | |
| for (i = 0 to num_samples) { | |
| // Allowed error from ACD message, depending on which plane sample i is in. | |
| allowed_error = SampleType(i) == Y_PLANE ? Y_ERR : UV_ERR; | |
| delta_i = max(0, abs(RemoteSample(i) - LocalSample(i)) - allowed_error); | |
| } | |
| ``` | |
| It is then up to the receiver how to interpret these deltas. A suggested method | |
| is to calculate a “corruption score” by calculating sum(delta(i)^2), where | |
| delta(i) is the delta for i:th sample in the message, and then scaling and | |
| capping that result to a maximum of 1.0. By squaring the sample, we make sure | |
| that even singular samples that are way outside their expected values cause a | |
| noticeable shift in the score. Another possible way is to calculate the distance | |
| and cap it using a sigmoid function. | |
| This extension message format does not make recommendations about what a | |
| receiver should do with the corruption scores, but some possibilities are: | |
| * Expose it as a statistics connected to the video receive stream. Let the | |
| application decide what to do with the information. | |
| * Let the WebRTC application use a corruption signal to take proactive measures. | |
| E.g. request a key-frame in order to recover, or try to switch to another | |
| codec type or implementation. | |
| ### Determining Filter Settings & Error Thresholds | |
| It is up to the sender to estimate how large the filter kernel and the allowed | |
| error thresholds should be. | |
| One method to do this is to analyze example outputs from different encoders and | |
| map the average frame QP to suitable settings. There will of course have to be | |
| different such mapping for e.g. AV1 compared to VP8 - but it’s also possible to | |
| get “tighter” values with knowledge of the exact implementation used. E.g. a | |
| mapping designed just for libaom encoder version X running with speed setting Y. | |
| Another method is to use the actual reconstructor state from the encoder. That | |
| of course means the encoder has to expose that state, which is not common. | |
| A benefit of doing it that way is that the filter size and allowed error can be | |
| very small (really only post-processing could introduce distortions in that | |
| scenario). A drawback is if the reconstructed state already contains corruption | |
| due to an encoder bug - then we would not be able to detect that corruption at | |
| all. | |
| There are also possibly more accurate but probably much more costly alternatives | |
| as well, such as training an ML model to determine the settings based on both | |
| the content of the source frame and any metadata present in the encoded | |
| bitstream. | |
| Regardless of method, the implementation at the send side SHOULD strive to set | |
| the filter size and error thresholds such that 99.5% of filtered samples end up | |
| with a delta <= the error threshold for that plane, based on a representative | |
| set of test clips and bandwidth constraints. | |
| Notes: The extension must not be present in more than 1 packet per video frame. |