| <!-- go/cmark --> | |
| <!--* freshness: {owner: 'eshr' reviewed: '2021-04-13'} *--> | |
| # Video Adaptation | |
| Video adaptation is a mechanism which reduces the bandwidth or CPU consumption | |
| by reducing encoded video quality. | |
| ## Overview | |
| Adaptation occurs when a _Resource_ signals that it is currently underused or | |
| overused. When overused, the video quality is decreased and when underused, the | |
| video quality is increased. There are currently two dimensions in which the | |
| quality can be adapted: frame-rate and resolution. The dimension that is adapted | |
| is based on the degradation preference for the video track. | |
| ## Resources | |
| _Resources_ monitor metrics from the system or the video stream. For example, a | |
| resource could monitor system temperature or the bandwidth usage of the video | |
| stream. A resource implements the [Resource][resource.h] interface. When a | |
| resource detects that it is overused, it calls `SetUsageState(kOveruse)`. When | |
| the resource is no longer overused, it can signal this using | |
| `SetUsageState(kUnderuse)`. | |
| There are two resources that are used by default on all video tracks: Quality | |
| scaler resource and encode overuse resource. | |
| ### QP Scaler Resource | |
| The quality scaler resource monitors the quantization parameter (QP) of the | |
| encoded video frames for video send stream and ensures that the quality of the | |
| stream is acceptable for the current resolution. After each frame is encoded the | |
| [QualityScaler][quality_scaler.h] is given the QP of the encoded frame. Overuse | |
| or underuse is signalled when the average QP is outside of the | |
| [QP thresholds][VideoEncoder::QpThresholds]. If the average QP is above the | |
| _high_ threshold, the QP scaler signals _overuse_, and when below the _low_ | |
| threshold the QP scaler signals _underuse_. | |
| The thresholds are set by the video encoder in the `scaling_settings` property | |
| of the [EncoderInfo][EncoderInfo]. | |
| *Note:* that the QP scaler is only enabled when the degradation preference is | |
| `MAINTAIN_FRAMERATE` or `BALANCED`. | |
| ### Encode Usage Resource | |
| The [encoder usage resource][encode_usage_resource.h] monitors how long it takes | |
| to encode a video frame. This works as a good proxy measurement for CPU usage as | |
| contention increases when CPU usage is high, increasing the encode times of the | |
| video frames. | |
| The time is tracked from when frame encoding starts to when it is completed. If | |
| the average encoder usage exceeds the thresholds set, *overuse* is triggered. | |
| ### Injecting other Resources | |
| A custom resource can be injected into the call using the | |
| [Call::AddAdaptationResource][Call::AddAdaptationResource] method. | |
| ## Adaptation | |
| When a a *resource* signals the it is over or underused, this signal reaches the | |
| `ResourceAdaptationProcessor` who requests an `Adaptation` proposal from the | |
| [VideoStreamAdapter][VideoStreamAdapter]. This proposal is based on the | |
| degradation preference of the video stream. `ResourceAdaptationProcessor` will | |
| determine if the `Adaptation` should be applied based on the current adaptation | |
| status and the `Adaptation` proposal. | |
| ### Degradation Preference | |
| There are 3 degradation preferences, described in the | |
| [RtpParameters][RtpParameters] header. These are | |
| * `MAINTIAIN_FRAMERATE`: Adapt video resolution | |
| * `MAINTIAIN_RESOLUTION`: Adapt video frame-rate. | |
| * `BALANCED`: Adapt video frame-rate or resolution. | |
| The degradation preference is set for a video track using the | |
| `degradation_preference` property in the [RtpParameters][RtpParameters]. | |
| ## VideoSinkWants and video stream adaptation | |
| Once an adaptation is applied it notifies the video stream. The video stream | |
| converts this adaptation to a [VideoSinkWants][VideoSinkWants]. These sink wants | |
| indicate to the video stream that some restrictions should be applied to the | |
| stream before it is sent to encoding. It has a few properties, but for | |
| adaptation the properties that might be set are: | |
| * `target_pixel_count`: The desired number of pixels for each video frame. The | |
| actual pixel count should be close to this but does not have to be exact so | |
| that aspect ratio can be maintained. | |
| * `max_pixel_count`: The maximum number of pixels in each video frame. This | |
| value can not be exceeded if set. | |
| * `max_framerate_fps`: The maximum frame-rate for the video source. The source | |
| is expected to drop frames that cause this threshold to be exceeded. | |
| The `VideoSinkWants` can be applied by any video source, or one may use the | |
| [AdaptedVideoTraceSource][adapted_video_track_source.h] which is a base class | |
| for sources that need video adaptation. | |
| [RtpParameters]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/rtp_parameters.h?q=%22RTC_EXPORT%20RtpParameters%22 | |
| [resource.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/adaptation/resource.h | |
| [Call::AddAdaptationResource]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/call.h?q=Call::AddAdaptationResource | |
| [quality_scaler.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/utility/quality_scaler.h | |
| [VideoEncoder::QpThresholds]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::QpThresholds | |
| [EncoderInfo]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::EncoderInfo | |
| [encode_usage_resource.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/adaptation/encode_usage_resource.h | |
| [VideoStreamAdapter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/adaptation/video_stream_adapter.h | |
| [adaptation_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/call/adaptation/adaptation_constraint.h | |
| [bitrate_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/video/adaptation/bitrate_constraint.h | |
| [AddOrUpdateSink]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_source_interface.h?q=AddOrUpdateSink | |
| [VideoSinkWants]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video/video_source_interface.h?q=%22RTC_EXPORT%20VideoSinkWants%22 | |
| [adapted_video_track_source.h]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/media/base/adapted_video_track_source.h |