Add documentation for video/adaptation
Bug: webrtc:12564
Change-Id: I24e807be6e7bbf1cd6d8b7ed0fa25bde6b257f34
Reviewed-on: https://webrtc-review.googlesource.com/c/src/+/215078
Commit-Queue: Evan Shrubsole <eshr@google.com>
Reviewed-by: Artem Titov <titovartem@webrtc.org>
Reviewed-by: Henrik Andreassson <henrika@webrtc.org>
Reviewed-by: Henrik Boström <hbos@webrtc.org>
Cr-Commit-Position: refs/heads/master@{#33728}
diff --git a/g3doc/sitemap.md b/g3doc/sitemap.md
index 5e0a9a8..95671df 100644
--- a/g3doc/sitemap.md
+++ b/g3doc/sitemap.md
@@ -20,6 +20,7 @@
* AudioProcessingModule
* [APM](/modules/audio_processing/g3doc/audio_processing_module.md)
* Video
+ * [Adaptation](/video/g3doc/adaptation.md)
* DataChannel
* PeerConnection
* Desktop capture
diff --git a/video/g3doc/adaptation.md b/video/g3doc/adaptation.md
new file mode 100644
index 0000000..e2dc3379
--- /dev/null
+++ b/video/g3doc/adaptation.md
@@ -0,0 +1,115 @@
+<?% config.freshness.reviewed = '2021-04-13' %?>
+
+<?% config.freshness.owner = 'eshr' %?>
+
+# Video Adaptation
+
+Video adaptation is a mechanism which reduces the bandwidth or CPU consumption
+by reducing encoded video quality.
+
+## Overview
+
+Adaptation occurs when a _Resource_ signals that it is currently underused or
+overused. When overused, the video quality is decreased and when underused, the
+video quality is increased. There are currently two dimensions in which the
+quality can be adapted: frame-rate and resolution. The dimension that is adapted
+is based on the degradation preference for the video track.
+
+## Resources
+
+_Resources_ monitor metrics from the system or the video stream. For example, a
+resource could monitor system temperature or the bandwidth usage of the video
+stream. A resource implements the [Resource][resource.h] interface. When a
+resource detects that it is overused, it calls `SetUsageState(kOveruse)`. When
+the resource is no longer overused, it can signal this using
+`SetUsageState(kUnderuse)`.
+
+There are two resources that are used by default on all video tracks: Quality
+scaler resource and encode overuse resource.
+
+### QP Scaler Resource
+
+The quality scaler resource monitors the quantization parameter (QP) of the
+encoded video frames for video send stream and ensures that the quality of the
+stream is acceptable for the current resolution. After each frame is encoded the
+[QualityScaler][quality_scaler.h] is given the QP of the encoded frame. Overuse
+or underuse is signalled when the average QP is outside of the
+[QP thresholds][VideoEncoder::QpThresholds]. If the average QP is above the
+_high_ threshold, the QP scaler signals _overuse_, and when below the _low_
+threshold the QP scaler signals _underuse_.
+
+The thresholds are set by the video encoder in the `scaling_settings` property
+of the [EncoderInfo][EncoderInfo].
+
+*Note:* that the QP scaler is only enabled when the degradation preference is
+`MAINTAIN_FRAMERATE` or `BALANCED`.
+
+### Encode Usage Resource
+
+The [encoder usage resource][encode_usage_resource.h] monitors how long it takes
+to encode a video frame. This works as a good proxy measurement for CPU usage as
+contention increases when CPU usage is high, increasing the encode times of the
+video frames.
+
+The time is tracked from when frame encoding starts to when it is completed. If
+the average encoder usage exceeds the thresholds set, *overuse* is triggered.
+
+### Injecting other Resources
+
+A custom resource can be injected into the call using the
+[Call::AddAdaptationResource][Call::AddAdaptationResource] method.
+
+## Adaptation
+
+When a a *resource* signals the it is over or underused, this signal reaches the
+`ResourceAdaptationProcessor` who requests an `Adaptation` proposal from the
+[VideoStreamAdapter][VideoStreamAdapter]. This proposal is based on the
+degradation preference of the video stream. `ResourceAdaptationProcessor` will
+determine if the `Adaptation` should be applied based on the current adaptation
+status and the `Adaptation` proposal.
+
+### Degradation Preference
+
+There are 3 degradation preferences, described in the
+[RtpParameters][RtpParameters] header. These are
+
+* `MAINTIAIN_FRAMERATE`: Adapt video resolution
+* `MAINTIAIN_RESOLUTION`: Adapt video frame-rate.
+* `BALANCED`: Adapt video frame-rate or resolution.
+
+The degradation preference is set for a video track using the
+`degradation_preference` property in the [RtpParameters][RtpParameters].
+
+## VideoSinkWants and video stream adaptation
+
+Once an adaptation is applied it notifies the video stream. The video stream
+converts this adaptation to a [VideoSinkWants][VideoSinkWants]. These sink wants
+indicate to the video stream that some restrictions should be applied to the
+stream before it is sent to encoding. It has a few properties, but for
+adaptation the properties that might be set are:
+
+* `target_pixel_count`: The desired number of pixels for each video frame. The
+ actual pixel count should be close to this but does not have to be exact so
+ that aspect ratio can be maintained.
+* `max_pixel_count`: The maximum number of pixels in each video frame. This
+ value can not be exceeded if set.
+* `max_framerate_fps`: The maximum frame-rate for the video source. The source
+ is expected to drop frames that cause this threshold to be exceeded.
+
+The `VideoSinkWants` can be applied by any video source, or one may use the
+[AdaptedVideoTraceSource][adapted_video_track_source.h] which is a base class
+for sources that need video adaptation.
+
+[RtpParameters]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/rtp_parameters.h?q=%22RTC_EXPORT%20RtpParameters%22
+[resource.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/adaptation/resource.h
+[Call::AddAdaptationResource]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/call/call.h?q=Call::AddAdaptationResource
+[quality_scaler.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/utility/quality_scaler.h
+[VideoEncoder::QpThresholds]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::QpThresholds
+[EncoderInfo]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video_codecs/video_encoder.h?q=VideoEncoder::EncoderInfo
+[encode_usage_resource.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/adaptation/encode_usage_resource.h
+[VideoStreamAdapter]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/call/adaptation/video_stream_adapter.h
+[adaptation_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/call/adaptation/adaptation_constraint.h
+[bitrate_constraint.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/video/adaptation/bitrate_constraint.h
+[AddOrUpdateSink]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_source_interface.h?q=AddOrUpdateSink
+[VideoSinkWants]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/api/video/video_source_interface.h?q=%22RTC_EXPORT%20VideoSinkWants%22
+[adapted_video_track_source.h]: https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/media/base/adapted_video_track_source.h
