Artem Titov | a617867 | 2023-01-30 10:51:01 | [diff] [blame] | 1 | <!-- go/cmark --> |
| 2 | <!--* freshness: {owner: 'sprang' reviewed: '2021-04-12'} *--> |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 3 | |
| 4 | # Paced Sending |
| 5 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 6 | The paced sender, often referred to as just the "pacer", is a part of the WebRTC |
| 7 | RTP stack used primarily to smooth the flow of packets sent onto the network. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 8 | |
| 9 | ## Background |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 10 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 11 | Consider a video stream at 5Mbps and 30fps. This would in an ideal world result |
| 12 | in each frame being ~21kB large and packetized into 18 RTP packets. While the |
| 13 | average bitrate over say a one second sliding window would be a correct 5Mbps, |
| 14 | on a shorter time scale it can be seen as a burst of 167Mbps every 33ms, each |
| 15 | followed by a 32ms silent period. Further, it is quite common that video |
| 16 | encoders overshoot the target frame size in case of sudden movement especially |
| 17 | dealing with screensharing. Frames being 10x or even 100x larger than the ideal |
| 18 | size is an all too real scenario. These packet bursts can cause several issues, |
| 19 | such as congesting networks and causing buffer bloat or even packet loss. Most |
| 20 | sessions have more than one media stream, e.g. a video and an audio track. If |
| 21 | you put a frame on the wire in one go, and those packets take 100ms to reach the |
| 22 | other side - that means you have now blocked any audio packets from reaching the |
| 23 | remote end in time as well. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 24 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 25 | The paced sender solves this by having a buffer in which media is queued, and |
| 26 | then using a _leaky bucket_ algorithm to pace them onto the network. The buffer |
| 27 | contains separate fifo streams for all media tracks so that e.g. audio can be |
| 28 | prioritized over video - and equal prio streams can be sent in a round-robin |
| 29 | fashion to avoid any one stream blocking others. |
| 30 | |
| 31 | Since the pacer is in control of the bitrate sent on the wire, it is also used |
| 32 | to generate padding in cases where a minimum send rate is required - and to |
| 33 | generate packet trains if bitrate probing is used. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 34 | |
| 35 | ## Life of a Packet |
| 36 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 37 | The typical path for media packets when using the paced sender looks something |
| 38 | like this: |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 39 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 40 | 1. `RTPSenderVideo` or `RTPSenderAudio` packetizes media into RTP packets. |
| 41 | 2. The packets are sent to the [RTPSender] class for transmission. |
| 42 | 3. The pacer is called via [RtpPacketSender] interface to enqueue the packet |
| 43 | batch. |
| 44 | 4. The packets are put into a queue within the pacer awaiting opportune moments |
| 45 | to send them. |
| 46 | 5. At a calculated time, the pacer calls the `PacingController::PacketSender()` |
| 47 | callback method, normally implemented by the [PacketRouter] class. |
| 48 | 6. The router forwards the packet to the correct RTP module based on the |
| 49 | packet's SSRC, and in which the `RTPSenderEgress` class makes final time |
| 50 | stamping, potentially records it for retransmissions etc. |
| 51 | 7. The packet is sent to the low-level `Transport` interface, after which it is |
| 52 | now out of scope. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 53 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 54 | Asynchronously to this, the estimated available send bandwidth is determined - |
Erik Språng | 8f722ca | 2022-06-09 09:00:09 | [diff] [blame] | 55 | and the target send rate is set on the `RtpPacketPacer` via the `void |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 56 | SetPacingRates(DataRate pacing_rate, DataRate padding_rate)` method. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 57 | |
| 58 | ## Packet Prioritization |
| 59 | |
| 60 | The pacer prioritized packets based on two criteria: |
| 61 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 62 | * Packet type, with most to least prioritized: |
| 63 | 1. Audio |
| 64 | 2. Retransmissions |
| 65 | 3. Video and FEC |
| 66 | 4. Padding |
| 67 | * Enqueue order |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 68 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 69 | The enqueue order is enforced on a per stream (SSRC) basis. Given equal |
| 70 | priority, the [RoundRobinPacketQueue] alternates between media streams to ensure |
| 71 | no stream needlessly blocks others. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 72 | |
| 73 | ## Implementations |
| 74 | |
Erik Språng | 8f722ca | 2022-06-09 09:00:09 | [diff] [blame] | 75 | The main class to use is called [TaskQueuePacedSender]. It uses a task queue to |
| 76 | manage thread safety and schedule delayed tasks, but delegates most of the actual |
| 77 | work to the `PacingController` class. |
| 78 | This way, it's possible to develop a custom pacer with different scheduling |
| 79 | mechanism - but ratain the same pacing logic. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 80 | |
| 81 | ## The Packet Router |
| 82 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 83 | An adjacent component called [PacketRouter] is used to route packets coming out |
| 84 | of the pacer and into the correct RTP module. It has the following functions: |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 85 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 86 | * The `SendPacket` method looks up an RTP module with an SSRC corresponding to |
| 87 | the packet for further routing to the network. |
| 88 | * If send-side bandwidth estimation is used, it populates the transport-wide |
| 89 | sequence number extension. |
| 90 | * Generate padding. Modules supporting payload-based padding are prioritized, |
| 91 | with the last module to have sent media always being the first choice. |
| 92 | * Returns any generated FEC after having sent media. |
| 93 | * Forwards REMB and/or TransportFeedback messages to suitable RTP modules. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 94 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 95 | At present the FEC is generated on a per SSRC basis, so is always returned from |
| 96 | an RTP module after sending media. Hopefully one day we will support covering |
| 97 | multiple streams with a single FlexFEC stream - and the packet router is the |
| 98 | likely place for that FEC generator to live. It may even be used for FEC padding |
| 99 | as an alternative to RTX. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 100 | |
| 101 | ## The API |
| 102 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 103 | The section outlines the classes and methods relevant to a few different use |
| 104 | cases of the pacer. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 105 | |
| 106 | ### Packet sending |
| 107 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 108 | For sending packets, use |
| 109 | `RtpPacketSender::EnqueuePackets(std::vector<std::unique_ptr<RtpPacketToSend>> |
| 110 | packets)` The pacer takes a `PacingController::PacketSender` as constructor |
| 111 | argument, this callback is used when it's time to actually send packets. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 112 | |
| 113 | ### Send rates |
| 114 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 115 | To control the send rate, use `void SetPacingRates(DataRate pacing_rate, |
| 116 | DataRate padding_rate)` If the packet queue becomes empty and the send rate |
| 117 | drops below `padding_rate`, the pacer will request padding packets from the |
| 118 | `PacketRouter`. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 119 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 120 | In order to completely suspend/resume sending data (e.g. due to network |
| 121 | availability), use the `Pause()` and `Resume()` methods. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 122 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 123 | The specified pacing rate may be overriden in some cases, e.g. due to extreme |
| 124 | encoder overshoot. Use `void SetQueueTimeLimit(TimeDelta limit)` to specify the |
| 125 | longest time you want packets to spend waiting in the pacer queue (pausing |
| 126 | excluded). The actual send rate may then be increased past the pacing_rate to |
| 127 | try to make the _average_ queue time less than that requested limit. The |
| 128 | rationale for this is that if the send queue is say longer than three seconds, |
| 129 | it's better to risk packet loss and then try to recover using a key-frame rather |
| 130 | than cause severe delays. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 131 | |
| 132 | ### Bandwidth estimation |
| 133 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 134 | If the bandwidth estimator supports bandwidth probing, it may request a cluster |
| 135 | of packets to be sent at a specified rate in order to gauge if this causes |
Per Kjellander | 88af203 | 2022-05-16 17:58:40 | [diff] [blame] | 136 | increased delay/loss on the network. Use the `void CreateProbeCluster(...)` |
| 137 | method - packets sent via this `PacketRouter` will be marked with the |
| 138 | corresponding cluster_id in the attached `PacedPacketInfo` struct. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 139 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 140 | If congestion window pushback is used, the state can be updated using |
| 141 | `SetCongestionWindow()` and `UpdateOutstandingData()`. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 142 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 143 | A few more methods control how we pace: * `SetAccountForAudioPackets()` |
| 144 | determines if audio packets count into bandwidth consumed. * |
| 145 | `SetIncludeOverhead()` determines if the entire RTP packet size counts into |
| 146 | bandwidth used (otherwise just media payload). * `SetTransportOverhead()` sets |
| 147 | an additional data size consumed per packet, representing e.g. UDP/IP headers. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 148 | |
| 149 | ### Stats |
| 150 | |
| 151 | Several methods are used to gather statistics in pacer state: |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 152 | |
Artem Titov | 3ab7a55 | 2021-04-14 14:23:10 | [diff] [blame] | 153 | * `OldestPacketWaitTime()` time since the oldest packet in the queue was |
| 154 | added. |
| 155 | * `QueueSizeData()` total bytes currently in the queue. |
| 156 | * `FirstSentPacketTime()` absolute time the first packet was sent. |
| 157 | * `ExpectedQueueTime()` total bytes in the queue divided by the send rate. |
Erik Språng | 32347b5 | 2021-04-13 16:18:55 | [diff] [blame] | 158 | |
Tony Herre | b0ed120 | 2021-07-22 15:40:44 | [diff] [blame] | 159 | [RTPSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/source/rtp_sender.h;drc=77ee8542dd35d5143b5788ddf47fb7cdb96eb08e |
| 160 | [RtpPacketSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/include/rtp_packet_sender.h;drc=ea55b0872f14faab23a4e5dbcb6956369c8ed5dc |
| 161 | [RtpPacketPacer]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/rtp_packet_pacer.h;drc=e7bc3a347760023dd4840cf6ebdd1e6c8592f4d7 |
| 162 | [PacketRouter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/packet_router.h;drc=3d2210876e31d0bb5c7de88b27fd02ceb1f4e03e |
Tony Herre | b0ed120 | 2021-07-22 15:40:44 | [diff] [blame] | 163 | [TaskQueuePacedSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/task_queue_paced_sender.h;drc=5051693ada61bc7b78855c6fb3fa87a0394fa813 |
| 164 | [RoundRobinPacketQueue]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/round_robin_packet_queue.h;drc=b571ff48f8fe07678da5a854cd6c3f5dde02855f |