modules/pacing/g3doc/index.md - src - Git at Google

 <!-- go/cmark -->
 <!--* freshness: {owner: 'sprang' reviewed: '2021-04-12'} *-->

 # Paced Sending

 The paced sender, often referred to as just the "pacer", is a part of the WebRTC
 RTP stack used primarily to smooth the flow of packets sent onto the network.

 ## Background

 Consider a video stream at 5Mbps and 30fps. This would in an ideal world result
 in each frame being ~21kB large and packetized into 18 RTP packets. While the
 average bitrate over say a one second sliding window would be a correct 5Mbps,
 on a shorter time scale it can be seen as a burst of 167Mbps every 33ms, each
 followed by a 32ms silent period. Further, it is quite common that video
 encoders overshoot the target frame size in case of sudden movement especially
 dealing with screensharing. Frames being 10x or even 100x larger than the ideal
 size is an all too real scenario. These packet bursts can cause several issues,
 such as congesting networks and causing buffer bloat or even packet loss. Most
 sessions have more than one media stream, e.g. a video and an audio track. If
 you put a frame on the wire in one go, and those packets take 100ms to reach the
 other side - that means you have now blocked any audio packets from reaching the
 remote end in time as well.

 The paced sender solves this by having a buffer in which media is queued, and
 then using a _leaky bucket_ algorithm to pace them onto the network. The buffer
 contains separate fifo streams for all media tracks so that e.g. audio can be
 prioritized over video - and equal prio streams can be sent in a round-robin
 fashion to avoid any one stream blocking others.

 Since the pacer is in control of the bitrate sent on the wire, it is also used
 to generate padding in cases where a minimum send rate is required - and to
 generate packet trains if bitrate probing is used.

 ## Life of a Packet

 The typical path for media packets when using the paced sender looks something
 like this:

 1.  `RTPSenderVideo` or `RTPSenderAudio` packetizes media into RTP packets.
 2.  The packets are sent to the [RTPSender] class for transmission.
 3.  The pacer is called via [RtpPacketSender] interface to enqueue the packet
     batch.
 4.  The packets are put into a queue within the pacer awaiting opportune moments
     to send them.
 5.  At a calculated time, the pacer calls the `PacingController::PacketSender()`
     callback method, normally implemented by the [PacketRouter] class.
 6.  The router forwards the packet to the correct RTP module based on the
     packet's SSRC, and in which the `RTPSenderEgress` class makes final time
     stamping, potentially records it for retransmissions etc.
 7.  The packet is sent to the low-level `Transport` interface, after which it is
     now out of scope.

 Asynchronously to this, the estimated available send bandwidth is determined -
 and the target send rate is set on the `RtpPacketPacer` via the `void
 SetPacingRates(DataRate pacing_rate, DataRate padding_rate)` method.

 ## Packet Prioritization

 The pacer prioritized packets based on two criteria:

 *   Packet type, with most to least prioritized:
     1.  Audio
     2.  Retransmissions
     3.  Video and FEC
     4.  Padding
 *   Enqueue order

 The enqueue order is enforced on a per stream (SSRC) basis. Given equal
 priority, the [RoundRobinPacketQueue] alternates between media streams to ensure
 no stream needlessly blocks others.

 ## Implementations

 The main class to use is called [TaskQueuePacedSender]. It uses a task queue to
 manage thread safety and schedule delayed tasks, but delegates most of the actual
 work to the `PacingController` class.
 This way, it's possible to develop a custom pacer with different scheduling
 mechanism - but ratain the same pacing logic.

 ## The Packet Router

 An adjacent component called [PacketRouter] is used to route packets coming out
 of the pacer and into the correct RTP module. It has the following functions:

 *   The `SendPacket` method looks up an RTP module with an SSRC corresponding to
     the packet for further routing to the network.
 *   If send-side bandwidth estimation is used, it populates the transport-wide
     sequence number extension.
 *   Generate padding. Modules supporting payload-based padding are prioritized,
     with the last module to have sent media always being the first choice.
 *   Returns any generated FEC after having sent media.
 *   Forwards REMB and/or TransportFeedback messages to suitable RTP modules.

 At present the FEC is generated on a per SSRC basis, so is always returned from
 an RTP module after sending media. Hopefully one day we will support covering
 multiple streams with a single FlexFEC stream - and the packet router is the
 likely place for that FEC generator to live. It may even be used for FEC padding
 as an alternative to RTX.

 ## The API

 The section outlines the classes and methods relevant to a few different use
 cases of the pacer.

 ### Packet sending

 For sending packets, use
 `RtpPacketSender::EnqueuePackets(std::vector<std::unique_ptr<RtpPacketToSend>>
 packets)` The pacer takes a `PacingController::PacketSender` as constructor
 argument, this callback is used when it's time to actually send packets.

 ### Send rates

 To control the send rate, use `void SetPacingRates(DataRate pacing_rate,
 DataRate padding_rate)` If the packet queue becomes empty and the send rate
 drops below `padding_rate`, the pacer will request padding packets from the
 `PacketRouter`.

 In order to completely suspend/resume sending data (e.g. due to network
 availability), use the `Pause()` and `Resume()` methods.

 The specified pacing rate may be overriden in some cases, e.g. due to extreme
 encoder overshoot. Use `void SetQueueTimeLimit(TimeDelta limit)` to specify the
 longest time you want packets to spend waiting in the pacer queue (pausing
 excluded). The actual send rate may then be increased past the pacing_rate to
 try to make the _average_ queue time less than that requested limit. The
 rationale for this is that if the send queue is say longer than three seconds,
 it's better to risk packet loss and then try to recover using a key-frame rather
 than cause severe delays.

 ### Bandwidth estimation

 If the bandwidth estimator supports bandwidth probing, it may request a cluster
 of packets to be sent at a specified rate in order to gauge if this causes
 increased delay/loss on the network. Use the `void CreateProbeCluster(...)`
 method - packets sent via this `PacketRouter` will be marked with the
 corresponding cluster_id in the attached `PacedPacketInfo` struct.

 If congestion window pushback is used, the state can be updated using
 `SetCongestionWindow()` and `UpdateOutstandingData()`.

 A few more methods control how we pace: * `SetAccountForAudioPackets()`
 determines if audio packets count into bandwidth consumed. *
 `SetIncludeOverhead()` determines if the entire RTP packet size counts into
 bandwidth used (otherwise just media payload). * `SetTransportOverhead()` sets
 an additional data size consumed per packet, representing e.g. UDP/IP headers.

 ### Stats

 Several methods are used to gather statistics in pacer state:

 *   `OldestPacketWaitTime()` time since the oldest packet in the queue was
     added.
 *   `QueueSizeData()` total bytes currently in the queue.
 *   `FirstSentPacketTime()` absolute time the first packet was sent.
 *   `ExpectedQueueTime()` total bytes in the queue divided by the send rate.

 [RTPSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/source/rtp_sender.h;drc=77ee8542dd35d5143b5788ddf47fb7cdb96eb08e
 [RtpPacketSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/include/rtp_packet_sender.h;drc=ea55b0872f14faab23a4e5dbcb6956369c8ed5dc
 [RtpPacketPacer]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/rtp_packet_pacer.h;drc=e7bc3a347760023dd4840cf6ebdd1e6c8592f4d7
 [PacketRouter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/packet_router.h;drc=3d2210876e31d0bb5c7de88b27fd02ceb1f4e03e
 [TaskQueuePacedSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/task_queue_paced_sender.h;drc=5051693ada61bc7b78855c6fb3fa87a0394fa813
 [RoundRobinPacketQueue]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/round_robin_packet_queue.h;drc=b571ff48f8fe07678da5a854cd6c3f5dde02855f
	<!-- go/cmark -->
	<!--* freshness: {owner: 'sprang' reviewed: '2021-04-12'} *-->

	# Paced Sending

	The paced sender, often referred to as just the "pacer", is a part of the WebRTC
	RTP stack used primarily to smooth the flow of packets sent onto the network.

	## Background

	Consider a video stream at 5Mbps and 30fps. This would in an ideal world result
	in each frame being ~21kB large and packetized into 18 RTP packets. While the
	average bitrate over say a one second sliding window would be a correct 5Mbps,
	on a shorter time scale it can be seen as a burst of 167Mbps every 33ms, each
	followed by a 32ms silent period. Further, it is quite common that video
	encoders overshoot the target frame size in case of sudden movement especially
	dealing with screensharing. Frames being 10x or even 100x larger than the ideal
	size is an all too real scenario. These packet bursts can cause several issues,
	such as congesting networks and causing buffer bloat or even packet loss. Most
	sessions have more than one media stream, e.g. a video and an audio track. If
	you put a frame on the wire in one go, and those packets take 100ms to reach the
	other side - that means you have now blocked any audio packets from reaching the
	remote end in time as well.

	The paced sender solves this by having a buffer in which media is queued, and
	then using a _leaky bucket_ algorithm to pace them onto the network. The buffer
	contains separate fifo streams for all media tracks so that e.g. audio can be
	prioritized over video - and equal prio streams can be sent in a round-robin
	fashion to avoid any one stream blocking others.

	Since the pacer is in control of the bitrate sent on the wire, it is also used
	to generate padding in cases where a minimum send rate is required - and to
	generate packet trains if bitrate probing is used.

	## Life of a Packet

	The typical path for media packets when using the paced sender looks something
	like this:

	1. `RTPSenderVideo` or `RTPSenderAudio` packetizes media into RTP packets.
	2. The packets are sent to the [RTPSender] class for transmission.
	3. The pacer is called via [RtpPacketSender] interface to enqueue the packet
	batch.
	4. The packets are put into a queue within the pacer awaiting opportune moments
	to send them.
	5. At a calculated time, the pacer calls the `PacingController::PacketSender()`
	callback method, normally implemented by the [PacketRouter] class.
	6. The router forwards the packet to the correct RTP module based on the
	packet's SSRC, and in which the `RTPSenderEgress` class makes final time
	stamping, potentially records it for retransmissions etc.
	7. The packet is sent to the low-level `Transport` interface, after which it is
	now out of scope.

	Asynchronously to this, the estimated available send bandwidth is determined -
	and the target send rate is set on the `RtpPacketPacer` via the `void
	SetPacingRates(DataRate pacing_rate, DataRate padding_rate)` method.

	## Packet Prioritization

	The pacer prioritized packets based on two criteria:

	* Packet type, with most to least prioritized:
	1. Audio
	2. Retransmissions
	3. Video and FEC
	4. Padding
	* Enqueue order

	The enqueue order is enforced on a per stream (SSRC) basis. Given equal
	priority, the [RoundRobinPacketQueue] alternates between media streams to ensure
	no stream needlessly blocks others.

	## Implementations

	The main class to use is called [TaskQueuePacedSender]. It uses a task queue to
	manage thread safety and schedule delayed tasks, but delegates most of the actual
	work to the `PacingController` class.
	This way, it's possible to develop a custom pacer with different scheduling
	mechanism - but ratain the same pacing logic.

	## The Packet Router

	An adjacent component called [PacketRouter] is used to route packets coming out
	of the pacer and into the correct RTP module. It has the following functions:

	* The `SendPacket` method looks up an RTP module with an SSRC corresponding to
	the packet for further routing to the network.
	* If send-side bandwidth estimation is used, it populates the transport-wide
	sequence number extension.
	* Generate padding. Modules supporting payload-based padding are prioritized,
	with the last module to have sent media always being the first choice.
	* Returns any generated FEC after having sent media.
	* Forwards REMB and/or TransportFeedback messages to suitable RTP modules.

	At present the FEC is generated on a per SSRC basis, so is always returned from
	an RTP module after sending media. Hopefully one day we will support covering
	multiple streams with a single FlexFEC stream - and the packet router is the
	likely place for that FEC generator to live. It may even be used for FEC padding
	as an alternative to RTX.

	## The API

	The section outlines the classes and methods relevant to a few different use
	cases of the pacer.

	### Packet sending

	For sending packets, use
	`RtpPacketSender::EnqueuePackets(std::vector<std::unique_ptr<RtpPacketToSend>>
	packets)` The pacer takes a `PacingController::PacketSender` as constructor
	argument, this callback is used when it's time to actually send packets.

	### Send rates

	To control the send rate, use `void SetPacingRates(DataRate pacing_rate,
	DataRate padding_rate)` If the packet queue becomes empty and the send rate
	drops below `padding_rate`, the pacer will request padding packets from the
	`PacketRouter`.

	In order to completely suspend/resume sending data (e.g. due to network
	availability), use the `Pause()` and `Resume()` methods.

	The specified pacing rate may be overriden in some cases, e.g. due to extreme
	encoder overshoot. Use `void SetQueueTimeLimit(TimeDelta limit)` to specify the
	longest time you want packets to spend waiting in the pacer queue (pausing
	excluded). The actual send rate may then be increased past the pacing_rate to
	try to make the _average_ queue time less than that requested limit. The
	rationale for this is that if the send queue is say longer than three seconds,
	it's better to risk packet loss and then try to recover using a key-frame rather
	than cause severe delays.

	### Bandwidth estimation

	If the bandwidth estimator supports bandwidth probing, it may request a cluster
	of packets to be sent at a specified rate in order to gauge if this causes
	increased delay/loss on the network. Use the `void CreateProbeCluster(...)`
	method - packets sent via this `PacketRouter` will be marked with the
	corresponding cluster_id in the attached `PacedPacketInfo` struct.

	If congestion window pushback is used, the state can be updated using
	`SetCongestionWindow()` and `UpdateOutstandingData()`.

	A few more methods control how we pace: * `SetAccountForAudioPackets()`
	determines if audio packets count into bandwidth consumed. *
	`SetIncludeOverhead()` determines if the entire RTP packet size counts into
	bandwidth used (otherwise just media payload). * `SetTransportOverhead()` sets
	an additional data size consumed per packet, representing e.g. UDP/IP headers.

	### Stats

	Several methods are used to gather statistics in pacer state:

	* `OldestPacketWaitTime()` time since the oldest packet in the queue was
	added.
	* `QueueSizeData()` total bytes currently in the queue.
	* `FirstSentPacketTime()` absolute time the first packet was sent.
	* `ExpectedQueueTime()` total bytes in the queue divided by the send rate.

	[RTPSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/source/rtp_sender.h;drc=77ee8542dd35d5143b5788ddf47fb7cdb96eb08e
	[RtpPacketSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/rtp_rtcp/include/rtp_packet_sender.h;drc=ea55b0872f14faab23a4e5dbcb6956369c8ed5dc
	[RtpPacketPacer]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/rtp_packet_pacer.h;drc=e7bc3a347760023dd4840cf6ebdd1e6c8592f4d7
	[PacketRouter]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/packet_router.h;drc=3d2210876e31d0bb5c7de88b27fd02ceb1f4e03e
	[TaskQueuePacedSender]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/task_queue_paced_sender.h;drc=5051693ada61bc7b78855c6fb3fa87a0394fa813
	[RoundRobinPacketQueue]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/pacing/round_robin_packet_queue.h;drc=b571ff48f8fe07678da5a854cd6c3f5dde02855f