blob: 69f1e695134384b029a353c2fff8c2db399174f4 [file] [log] [blame] [view]
Artem Titova6178672023-01-30 10:51:011<!-- go/cmark -->
2<!--* freshness: {owner: 'sprang' reviewed: '2021-04-12'} *-->
Erik Språng32347b52021-04-13 16:18:553
4# Paced Sending
5
Artem Titov3ab7a552021-04-14 14:23:106The paced sender, often referred to as just the "pacer", is a part of the WebRTC
7RTP stack used primarily to smooth the flow of packets sent onto the network.
Erik Språng32347b52021-04-13 16:18:558
9## Background
Erik Språng32347b52021-04-13 16:18:5510
Artem Titov3ab7a552021-04-14 14:23:1011Consider a video stream at 5Mbps and 30fps. This would in an ideal world result
12in each frame being ~21kB large and packetized into 18 RTP packets. While the
13average bitrate over say a one second sliding window would be a correct 5Mbps,
14on a shorter time scale it can be seen as a burst of 167Mbps every 33ms, each
15followed by a 32ms silent period. Further, it is quite common that video
16encoders overshoot the target frame size in case of sudden movement especially
17dealing with screensharing. Frames being 10x or even 100x larger than the ideal
18size is an all too real scenario. These packet bursts can cause several issues,
19such as congesting networks and causing buffer bloat or even packet loss. Most
20sessions have more than one media stream, e.g. a video and an audio track. If
21you put a frame on the wire in one go, and those packets take 100ms to reach the
22other side - that means you have now blocked any audio packets from reaching the
23remote end in time as well.
Erik Språng32347b52021-04-13 16:18:5524
Artem Titov3ab7a552021-04-14 14:23:1025The paced sender solves this by having a buffer in which media is queued, and
26then using a _leaky bucket_ algorithm to pace them onto the network. The buffer
27contains separate fifo streams for all media tracks so that e.g. audio can be
28prioritized over video - and equal prio streams can be sent in a round-robin
29fashion to avoid any one stream blocking others.
30
31Since the pacer is in control of the bitrate sent on the wire, it is also used
32to generate padding in cases where a minimum send rate is required - and to
33generate packet trains if bitrate probing is used.
Erik Språng32347b52021-04-13 16:18:5534
35## Life of a Packet
36
Artem Titov3ab7a552021-04-14 14:23:1037The typical path for media packets when using the paced sender looks something
38like this:
Erik Språng32347b52021-04-13 16:18:5539
Artem Titov3ab7a552021-04-14 14:23:10401. `RTPSenderVideo` or `RTPSenderAudio` packetizes media into RTP packets.
412. The packets are sent to the [RTPSender] class for transmission.
423. The pacer is called via [RtpPacketSender] interface to enqueue the packet
43 batch.
444. The packets are put into a queue within the pacer awaiting opportune moments
45 to send them.
465. At a calculated time, the pacer calls the `PacingController::PacketSender()`
47 callback method, normally implemented by the [PacketRouter] class.
486. The router forwards the packet to the correct RTP module based on the
49 packet's SSRC, and in which the `RTPSenderEgress` class makes final time
50 stamping, potentially records it for retransmissions etc.
517. The packet is sent to the low-level `Transport` interface, after which it is
52 now out of scope.
Erik Språng32347b52021-04-13 16:18:5553
Artem Titov3ab7a552021-04-14 14:23:1054Asynchronously to this, the estimated available send bandwidth is determined -
Erik Språng8f722ca2022-06-09 09:00:0955and the target send rate is set on the `RtpPacketPacer` via the `void
Artem Titov3ab7a552021-04-14 14:23:1056SetPacingRates(DataRate pacing_rate, DataRate padding_rate)` method.
Erik Språng32347b52021-04-13 16:18:5557
58## Packet Prioritization
59
60The pacer prioritized packets based on two criteria:
61
Artem Titov3ab7a552021-04-14 14:23:1062* Packet type, with most to least prioritized:
63 1. Audio
64 2. Retransmissions
65 3. Video and FEC
66 4. Padding
67* Enqueue order
Erik Språng32347b52021-04-13 16:18:5568
Artem Titov3ab7a552021-04-14 14:23:1069The enqueue order is enforced on a per stream (SSRC) basis. Given equal
70priority, the [RoundRobinPacketQueue] alternates between media streams to ensure
71no stream needlessly blocks others.
Erik Språng32347b52021-04-13 16:18:5572
73## Implementations
74
Erik Språng8f722ca2022-06-09 09:00:0975The main class to use is called [TaskQueuePacedSender]. It uses a task queue to
76manage thread safety and schedule delayed tasks, but delegates most of the actual
77work to the `PacingController` class.
78This way, it's possible to develop a custom pacer with different scheduling
79mechanism - but ratain the same pacing logic.
Erik Språng32347b52021-04-13 16:18:5580
81## The Packet Router
82
Artem Titov3ab7a552021-04-14 14:23:1083An adjacent component called [PacketRouter] is used to route packets coming out
84of the pacer and into the correct RTP module. It has the following functions:
Erik Språng32347b52021-04-13 16:18:5585
Artem Titov3ab7a552021-04-14 14:23:1086* The `SendPacket` method looks up an RTP module with an SSRC corresponding to
87 the packet for further routing to the network.
88* If send-side bandwidth estimation is used, it populates the transport-wide
89 sequence number extension.
90* Generate padding. Modules supporting payload-based padding are prioritized,
91 with the last module to have sent media always being the first choice.
92* Returns any generated FEC after having sent media.
93* Forwards REMB and/or TransportFeedback messages to suitable RTP modules.
Erik Språng32347b52021-04-13 16:18:5594
Artem Titov3ab7a552021-04-14 14:23:1095At present the FEC is generated on a per SSRC basis, so is always returned from
96an RTP module after sending media. Hopefully one day we will support covering
97multiple streams with a single FlexFEC stream - and the packet router is the
98likely place for that FEC generator to live. It may even be used for FEC padding
99as an alternative to RTX.
Erik Språng32347b52021-04-13 16:18:55100
101## The API
102
Artem Titov3ab7a552021-04-14 14:23:10103The section outlines the classes and methods relevant to a few different use
104cases of the pacer.
Erik Språng32347b52021-04-13 16:18:55105
106### Packet sending
107
Artem Titov3ab7a552021-04-14 14:23:10108For sending packets, use
109`RtpPacketSender::EnqueuePackets(std::vector<std::unique_ptr<RtpPacketToSend>>
110packets)` The pacer takes a `PacingController::PacketSender` as constructor
111argument, this callback is used when it's time to actually send packets.
Erik Språng32347b52021-04-13 16:18:55112
113### Send rates
114
Artem Titov3ab7a552021-04-14 14:23:10115To control the send rate, use `void SetPacingRates(DataRate pacing_rate,
116DataRate padding_rate)` If the packet queue becomes empty and the send rate
117drops below `padding_rate`, the pacer will request padding packets from the
118`PacketRouter`.
Erik Språng32347b52021-04-13 16:18:55119
Artem Titov3ab7a552021-04-14 14:23:10120In order to completely suspend/resume sending data (e.g. due to network
121availability), use the `Pause()` and `Resume()` methods.
Erik Språng32347b52021-04-13 16:18:55122
Artem Titov3ab7a552021-04-14 14:23:10123The specified pacing rate may be overriden in some cases, e.g. due to extreme
124encoder overshoot. Use `void SetQueueTimeLimit(TimeDelta limit)` to specify the
125longest time you want packets to spend waiting in the pacer queue (pausing
126excluded). The actual send rate may then be increased past the pacing_rate to
127try to make the _average_ queue time less than that requested limit. The
128rationale for this is that if the send queue is say longer than three seconds,
129it's better to risk packet loss and then try to recover using a key-frame rather
130than cause severe delays.
Erik Språng32347b52021-04-13 16:18:55131
132### Bandwidth estimation
133
Artem Titov3ab7a552021-04-14 14:23:10134If the bandwidth estimator supports bandwidth probing, it may request a cluster
135of packets to be sent at a specified rate in order to gauge if this causes
Per Kjellander88af2032022-05-16 17:58:40136increased delay/loss on the network. Use the `void CreateProbeCluster(...)`
137method - packets sent via this `PacketRouter` will be marked with the
138corresponding cluster_id in the attached `PacedPacketInfo` struct.
Erik Språng32347b52021-04-13 16:18:55139
Artem Titov3ab7a552021-04-14 14:23:10140If congestion window pushback is used, the state can be updated using
141`SetCongestionWindow()` and `UpdateOutstandingData()`.
Erik Språng32347b52021-04-13 16:18:55142
Artem Titov3ab7a552021-04-14 14:23:10143A few more methods control how we pace: * `SetAccountForAudioPackets()`
144determines if audio packets count into bandwidth consumed. *
145`SetIncludeOverhead()` determines if the entire RTP packet size counts into
146bandwidth used (otherwise just media payload). * `SetTransportOverhead()` sets
147an additional data size consumed per packet, representing e.g. UDP/IP headers.
Erik Språng32347b52021-04-13 16:18:55148
149### Stats
150
151Several methods are used to gather statistics in pacer state:
Erik Språng32347b52021-04-13 16:18:55152
Artem Titov3ab7a552021-04-14 14:23:10153* `OldestPacketWaitTime()` time since the oldest packet in the queue was
154 added.
155* `QueueSizeData()` total bytes currently in the queue.
156* `FirstSentPacketTime()` absolute time the first packet was sent.
157* `ExpectedQueueTime()` total bytes in the queue divided by the send rate.
Erik Språng32347b52021-04-13 16:18:55158
Tony Herreb0ed1202021-07-22 15:40:44159[RTPSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/source/rtp_sender.h;drc=77ee8542dd35d5143b5788ddf47fb7cdb96eb08e
160[RtpPacketSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/include/rtp_packet_sender.h;drc=ea55b0872f14faab23a4e5dbcb6956369c8ed5dc
161[RtpPacketPacer]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/rtp_packet_pacer.h;drc=e7bc3a347760023dd4840cf6ebdd1e6c8592f4d7
162[PacketRouter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/packet_router.h;drc=3d2210876e31d0bb5c7de88b27fd02ceb1f4e03e
Tony Herreb0ed1202021-07-22 15:40:44163[TaskQueuePacedSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/task_queue_paced_sender.h;drc=5051693ada61bc7b78855c6fb3fa87a0394fa813
164[RoundRobinPacketQueue]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/round_robin_packet_queue.h;drc=b571ff48f8fe07678da5a854cd6c3f5dde02855f