video/g3doc/adaptation.md

<?% config.freshness.reviewed = '2021-04-13' %?>
<?% config.freshness.owner = 'eshr' %?>

# Video Adaptation

Video adaptation is a mechanism which reduces the bandwidth or CPU consumption
by reducing encoded video quality.

## Overview

Adaptation occurs when a _Resource_ signals that it is currently underused or
overused. When overused, the video quality is decreased and when underused, the
video quality is increased. There are currently two dimensions in which the
quality can be adapted: frame-rate and resolution. The dimension that is adapted
is based on the degradation preference for the video track.

## Resources

_Resources_ monitor metrics from the system or the video stream. For example, a
resource could monitor system temperature or the bandwidth usage of the video
stream. A resource implements the [Resource][resource.h] interface. When a
resource detects that it is overused, it calls `SetUsageState(kOveruse)`. When
the resource is no longer overused, it can signal this using
`SetUsageState(kUnderuse)`.

There are two resources that are used by default on all video tracks: Quality
scaler resource and encode overuse resource.

### QP Scaler Resource

The quality scaler resource monitors the quantization parameter (QP) of the
encoded video frames for video send stream and ensures that the quality of the
stream is acceptable for the current resolution. After each frame is encoded the
[QualityScaler][quality_scaler.h] is given the QP of the encoded frame. Overuse
or underuse is signalled when the average QP is outside of the
[QP thresholds][VideoEncoder::QpThresholds]. If the average QP is above the
_high_ threshold, the QP scaler signals _overuse_, and when below the _low_
threshold the QP scaler signals _underuse_.

The thresholds are set by the video encoder in the `scaling_settings` property
of the [EncoderInfo][EncoderInfo].

*Note:* that the QP scaler is only enabled when the degradation preference is
`MAINTAIN_FRAMERATE` or `BALANCED`.

### Encode Usage Resource

The [encoder usage resource][encode_usage_resource.h] monitors how long it takes
to encode a video frame. This works as a good proxy measurement for CPU usage as
contention increases when CPU usage is high, increasing the encode times of the
video frames.

The time is tracked from when frame encoding starts to when it is completed. If
the average encoder usage exceeds the thresholds set, *overuse* is triggered.

### Injecting other Resources

A custom resource can be injected into the call using the
[Call::AddAdaptationResource][Call::AddAdaptationResource] method.

## Adaptation

When a a *resource* signals the it is over or underused, this signal reaches the
`ResourceAdaptationProcessor` who requests an `Adaptation` proposal from the
[VideoStreamAdapter][VideoStreamAdapter]. This proposal is based on the
degradation preference of the video stream. `ResourceAdaptationProcessor` will
determine if the `Adaptation` should be applied based on the current adaptation
status and the `Adaptation` proposal.

### Degradation Preference

There are 3 degradation preferences, described in the
[RtpParameters][RtpParameters] header. These are

*   `MAINTIAIN_FRAMERATE`: Adapt video resolution
*   `MAINTIAIN_RESOLUTION`: Adapt video frame-rate.
*   `BALANCED`: Adapt video frame-rate or resolution.

The degradation preference is set for a video track using the
`degradation_preference` property in the [RtpParameters][RtpParameters].

## VideoSinkWants and video stream adaptation

Once an adaptation is applied it notifies the video stream. The video stream
converts this adaptation to a [VideoSinkWants][VideoSinkWants]. These sink wants
indicate to the video stream that some restrictions should be applied to the
stream before it is sent to encoding. It has a few properties, but for
adaptation the properties that might be set are:

*   `target_pixel_count`: The desired number of pixels for each video frame. The
    actual pixel count should be close to this but does not have to be exact so
    that aspect ratio can be maintained.
*   `max_pixel_count`: The maximum number of pixels in each video frame. This
    value can not be exceeded if set.
*   `max_framerate_fps`: The maximum frame-rate for the video source. The source
    is expected to drop frames that cause this threshold to be exceeded.

The `VideoSinkWants` can be applied by any video source, or one may use the
[AdaptedVideoTraceSource][adapted_video_track_source.h] which is a base class
for sources that need video adaptation.

[RtpParameters]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/rtp_parameters.h?q=%22RTC_EXPORT%20RtpParameters%22
[resource.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/adaptation/resource.h
[Call::AddAdaptationResource]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/call.h?q=Call::AddAdaptationResource
[quality_scaler.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/quality_scaler.h
[VideoEncoder::QpThresholds]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::QpThresholds
[EncoderInfo]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::EncoderInfo
[encode_usage_resource.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/adaptation/encode_usage_resource.h
[VideoStreamAdapter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/adaptation/video_stream_adapter.h
[adaptation_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/adaptation/adaptation_constraint.h
[bitrate_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/adaptation/bitrate_constraint.h
[AddOrUpdateSink]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_source_interface.h?q=AddOrUpdateSink
[VideoSinkWants]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_source_interface.h?q=%22RTC_EXPORT%20VideoSinkWants%22
[adapted_video_track_source.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/media/base/adapted_video_track_source.h