History

doc/ffmpeg: rewrite the detailed description chapter

Split it into sections that describe in detail * the components of the transcoding pipeline * the main features it handles, in order of complexity * streamcopy * transcoding * filtering Replace the current confusing/misleading diagrams with new ones that actually reflect the program components and data flow between them.

Anton Khirnov committed 2 years ago

Commit 794308c1 parent f3391691 file · +366 −101Browse files

Unified · Split

doc/ffmpeg.texi+366−101
@@ -87,140 +87,405 @@ The format option may be needed for raw input files. · 86 unmodified lines
8787 @chapter Detailed description
8888 @c man begin DETAILED DESCRIPTION
8989
90−The transcoding process in @command{ffmpeg} for each output can be described by
91−the following diagram:
90+@command{ffmpeg} builds a transcoding pipeline out of the components listed
91+below. The program's operation then consists of input data chunks flowing from
92+the sources down the pipes towards the sinks, while being transformed by the
93+components they encounter along the way.
9294
93−@verbatim
94− _______ ______________
95−| | | |
96−| input | demuxer | encoded data | decoder
97−| file | ---------> | packets | -----+
98−|_______| |______________| |
99− v
100− _________
101− | |
102− | decoded |
103− | frames |
104− |_________|
105− ________ ______________ |
106−| | | | |
107−| output | <-------- | encoded data | <----+
108−| file | muxer | packets | encoder
109−|________| |______________|
95+The following kinds of components are available:
96+@itemize
97+@item
98+@emph{Demuxers} (short for "demultiplexers") read an input source in order to
99+extract
110100
101+@itemize
102+@item
103+global properties such as metadata or chapters;
104+@item
105+list of input elementary streams and their properties
106+@end itemize
107+
108+One demuxer instance is created for each @option{-i} option, and sends encoded
109+@emph{packets} to @emph{decoders} or @emph{muxers}.
110+
111+In other literature, demuxers are sometimes called @emph{splitters}, because
112+their main function is splitting a file into elementary streams (though some
113+files only contain one elementary stream).
111114
115+A schematic representation of a demuxer looks like this:
116+@verbatim
117+┌──────────┬───────────────────────┐
118+│ demuxer │ │ packets for stream 0
119+╞══════════╡ elementary stream 0 ├──────────────────────⮞
120+│ │ │
121+│ global ├───────────────────────┤
122+│properties│ │ packets for stream 1
123+│ and │ elementary stream 1 ├──────────────────────⮞
124+│ metadata │ │
125+│ ├───────────────────────┤
126+│ │ │
127+│ │ ........... │
128+│ │ │
129+│ ├───────────────────────┤
130+│ │ │ packets for stream N
131+│ │ elementary stream N ├──────────────────────⮞
132+│ │ │
133+└──────────┴───────────────────────┘
134+ ⯅
135+ │
136+ │ read from file, network stream,
137+ │ grabbing device, etc.
138+ │
112139 @end verbatim
113140
114−@command{ffmpeg} calls the libavformat library (containing demuxers) to read
115−input files and get packets containing encoded data from them. When there are
116−multiple input files, @command{ffmpeg} tries to keep them synchronized by
117−tracking lowest timestamp on any active input stream.
141+@item
142+@emph{Decoders} receive encoded (compressed) @emph{packets} for an audio, video,
143+or subtitle elementary stream, and decode them into raw @emph{frames} (arrays of
144+pixels for video, PCM for audio). A decoder is typically associated with (and
145+receives its input from) an elementary stream in a @emph{demuxer}, but sometimes
146+may also exist on its own (see @ref{Loopback decoders}).
118147
119−Encoded packets are then passed to the decoder (unless streamcopy is selected
120−for the stream, see further for a description). The decoder produces
121−uncompressed frames (raw video/PCM audio/...) which can be processed further by
122−filtering (see next section). After filtering, the frames are passed to the
123−encoder, which encodes them and outputs encoded packets. Finally, those are
124−passed to the muxer, which writes the encoded packets to the output file.
148+A schematic representation of a decoder looks like this:
149+@verbatim
150+ ┌─────────┐
151+ packets │ │ raw frames
152+─────────⮞│ decoder ├────────────⮞
153+ │ │
154+ └─────────┘
155+@end verbatim
125156
126−@section Filtering
127−Before encoding, @command{ffmpeg} can process raw audio and video frames using
128−filters from the libavfilter library. Several chained filters form a filter
129−graph. @command{ffmpeg} distinguishes between two types of filtergraphs:
130−simple and complex.
157+@item
158+@emph{Filtergraphs} process and transform raw audio or video @emph{frames}. A
159+filtergraph consists of one or more individual @emph{filters} linked into a
160+graph. Filtergraphs come in two flavors - @emph{simple} and @emph{complex},
161+configured with the @option{-filter} and @option{-filter_complex} options,
162+respectively.
163+
164+A simple filtergraph is associated with an @emph{output elementary stream}; it
165+receives the input to be filtered from a @emph{decoder} and sends filtered
166+output to that output stream's @emph{encoder}.
167+
168+A simple video filtergraph that performs deinterlacing (using the @code{yadif}
169+deinterlacer) followed by resizing (using the @code{scale} filter) can look like
170+this:
171+@verbatim
131172
132−@subsection Simple filtergraphs
133−Simple filtergraphs are those that have exactly one input and output, both of
134−the same type. In the above diagram they can be represented by simply inserting
135−an additional step between decoding and encoding:
173+ ┌────────────────────────┐
174+ │ simple filtergraph │
175+ frames from ╞════════════════════════╡ frames for
176+ a decoder │ ┌───────┐ ┌───────┐ │ an encoder
177+────────────⮞├─⮞│ yadif ├─⮞│ scale ├─⮞│────────────⮞
178+ │ └───────┘ └───────┘ │
179+ └────────────────────────┘
180+@end verbatim
136181
182+A complex filtergraph is standalone and not associated with any specific stream.
183+It may have multiple (or zero) inputs, potentially of different types (audio or
184+video), each of which receiving data either from a decoder or another complex
185+filtergraph's output. It also has one or more outputs that feed either an
186+encoder or another complex filtergraph's input.
187+
188+The following example diagram represents a complex filtergraph with 3 inputs and
189+2 outputs (all video):
137190 @verbatim
138− _________ ______________
139−| | | |
140−| decoded | | encoded data |
141−| frames |\ _ | packets |
142−|_________| \ /||______________|
143− \ __________ /
144− simple _\|| | / encoder
145− filtergraph | filtered |/
146− | frames |
147− |__________|
191+ ┌─────────────────────────────────────────────────┐
192+ │ complex filtergraph │
193+ ╞═════════════════════════════════════════════════╡
194+ frames ├───────┐ ┌─────────┐ ┌─────────┐ ┌────────┤ frames
195+─────────⮞│input 0├─⮞│ overlay ├─────⮞│ overlay ├─⮞│output 0├────────⮞
196+ ├───────┘ │ │ │ │ └────────┤
197+ frames ├───────┐╭⮞│ │ ╭⮞│ │ │
198+─────────⮞│input 1├╯ └─────────┘ │ └─────────┘ │
199+ ├───────┘ │ │
200+ frames ├───────┐ ┌─────┐ ┌─────┬─╯ ┌────────┤ frames
201+─────────⮞│input 2├⮞│scale├⮞│split├───────────────⮞│output 1├────────⮞
202+ ├───────┘ └─────┘ └─────┘ └────────┤
203+ └─────────────────────────────────────────────────┘
204+@end verbatim
205+Frames from second input are overlaid over those from the first. Frames from the
206+third input are rescaled, then the duplicated into two identical streams. One of
207+them is overlaid over the combined first two inputs, with the result exposed as
208+the filtergraph's first output. The other duplicate ends up being the
209+filtergraph's second output.
148210
211+@item
212+@emph{Encoders} receive raw audio, video, or subtitle @emph{frames} and encode
213+them into encoded @emph{packets}. The encoding (compression) process is
214+typically @emph{lossy} - it degrades stream quality to make the output smaller;
215+some encoders are @emph{lossless}, but at the cost of much higher output size. A
216+video or audio encoder receives its input from some filtergraph's output,
217+subtitle encoders receive input from a decoder (since subtitle filtering is not
218+supported yet). Every encoder is associated with some muxer's @emph{output
219+elementary stream} and sends its output to that muxer.
220+
221+A schematic representation of an encoder looks like this:
222+@verbatim
223+ ┌─────────┐
224+ raw frames │ │ packets
225+────────────⮞│ encoder ├─────────⮞
226+ │ │
227+ └─────────┘
149228 @end verbatim
150229
151−Simple filtergraphs are configured with the per-stream @option{-filter} option
152−(with @option{-vf} and @option{-af} aliases for video and audio respectively).
153−A simple filtergraph for video can look for example like this:
230+@item
231+@emph{Muxers} (short for "multiplexers") receive encoded @emph{packets} for
232+their elementary streams from encoders (the @emph{transcoding} path) or directly
233+from demuxers (the @emph{streamcopy} path), interleave them (when there is more
234+than one elementary stream), and write the resulting bytes into the output file
235+(or pipe, network stream, etc.).
154236
237+A schematic representation of a muxer looks like this:
155238 @verbatim
156− _______ _____________ _______ ________
157−| | | | | | | |
158−| input | ---> | deinterlace | ---> | scale | ---> | output |
159−|_______| |_____________| |_______| |________|
239+ ┌──────────────────────┬───────────┐
240+ packets for stream 0 │ │ muxer │
241+──────────────────────⮞│ elementary stream 0 ╞═══════════╡
242+ │ │ │
243+ ├──────────────────────┤ global │
244+ packets for stream 1 │ │properties │
245+──────────────────────⮞│ elementary stream 1 │ and │
246+ │ │ metadata │
247+ ├──────────────────────┤ │
248+ │ │ │
249+ │ ........... │ │
250+ │ │ │
251+ ├──────────────────────┤ │
252+ packets for stream N │ │ │
253+──────────────────────⮞│ elementary stream N │ │
254+ │ │ │
255+ └──────────────────────┴─────┬─────┘
256+ │
257+ write to file, network stream, │
258+ grabbing device, etc. │
259+ │
260+ ▼
261+@end verbatim
262+
263+@end itemize
160264
265+@section Streamcopy
266+The simplest pipeline in @command{ffmpeg} is single-stream
267+@emph{streamcopy}, that is copying one @emph{input elementary stream}'s packets
268+without decoding, filtering, or encoding them. As an example, consider an input
269+file called @file{INPUT.mkv} with 3 elementary streams, from which we take the
270+second and write it to file @file{OUTPUT.mp4}. A schematic representation of
271+such a pipeline looks like this:
272+@verbatim
273+┌──────────┬─────────────────────┐
274+│ demuxer │ │ unused
275+╞══════════╡ elementary stream 0 ├────────╳
276+│ │ │
277+│INPUT.mkv ├─────────────────────┤ ┌──────────────────────┬───────────┐
278+│ │ │ packets │ │ muxer │
279+│ │ elementary stream 1 ├─────────⮞│ elementary stream 0 ╞═══════════╡
280+│ │ │ │ │OUTPUT.mp4 │
281+│ ├─────────────────────┤ └──────────────────────┴───────────┘
282+│ │ │ unused
283+│ │ elementary stream 2 ├────────╳
284+│ │ │
285+└──────────┴─────────────────────┘
161286 @end verbatim
162287
163−Note that some filters change frame properties but not frame contents. E.g. the
164−@code{fps} filter in the example above changes number of frames, but does not
165−touch the frame contents. Another example is the @code{setpts} filter, which
166−only sets timestamps and otherwise passes the frames unchanged.
288+The above pipeline can be constructed with the following commandline:
289+@example
290+ffmpeg -i INPUT.mkv -map 0:1 -c copy OUTPUT.mp4
291+@end example
167292
168−@subsection Complex filtergraphs
169−Complex filtergraphs are those which cannot be described as simply a linear
170−processing chain applied to one stream. This is the case, for example, when the graph has
171−more than one input and/or output, or when output stream type is different from
172−input. They can be represented with the following diagram:
293+In this commandline
294+@itemize
295+
296+@item
297+there is a single input @file{INPUT.mkv};
298+
299+@item
300+there are no input options for this input;
301+
302+@item
303+there is a single output @file{OUTPUT.mp4};
304+
305+@item
306+there are two output options for this output:
173307
308+@itemize
309+@item
310+@code{-map 0:1} selects the input stream to be used - from input with index 0
311+(i.e. the first one) the stream with index 1 (i.e. the second one);
312+
313+@item
314+@code{-c copy} selects the @code{copy} encoder, i.e. streamcopy with no decoding
315+or encoding.
316+@end itemize
317+
318+@end itemize
319+
320+Streamcopy is useful for changing the elementary stream count, container format,
321+or modifying container-level metadata. Since there is no decoding or encoding,
322+it is very fast and there is no quality loss. However, it might not work in some
323+cases because of a variety of factors (e.g. certain information required by the
324+target container is not available in the source). Applying filters is obviously
325+also impossible, since filters work on decoded frames.
326+
327+More complex streamcopy scenarios can be constructed - e.g. combining streams
328+from two input files into a single output:
174329 @verbatim
175− _________
176−| |
177−| input 0 |\ __________
178−|_________| \ | |
179− \ _________ /| output 0 |
180− \ | | / |__________|
181− _________ \| complex | /
182−| | | |/
183−| input 1 |---->| filter |\
184−|_________| | | \ __________
185− /| graph | \ | |
186− / | | \| output 1 |
187− _________ / |_________| |__________|
188−| | /
189−| input 2 |/
190−|_________|
330+┌──────────┬────────────────────┐ ┌────────────────────┬───────────┐
331+│ demuxer 0│ │ packets │ │ muxer │
332+╞══════════╡elementary stream 0 ├────────⮞│elementary stream 0 ╞═══════════╡
333+│INPUT0.mkv│ │ │ │OUTPUT.mp4 │
334+└──────────┴────────────────────┘ ├────────────────────┤ │
335+┌──────────┬────────────────────┐ │ │ │
336+│ demuxer 1│ │ packets │elementary stream 1 │ │
337+╞══════════╡elementary stream 0 ├────────⮞│ │ │
338+│INPUT1.aac│ │ └────────────────────┴───────────┘
339+└──────────┴────────────────────┘
340+@end verbatim
341+that can be built by the commandline
342+@example
343+ffmpeg -i INPUT0.mkv -i INPUT1.aac -map 0:0 -map 1:0 -c copy OUTPUT.mp4
344+@end example
345+
346+The output @option{-map} option is used twice here, creating two streams in the
347+output file - one fed by the first input and one by the second. The single
348+instance of the @option{-c} option selects streamcopy for both of those streams.
349+You could also use multiple instances of this option together with
350+@ref{Stream specifiers} to apply different values to each stream, as will be
351+demonstrated in following sections.
191352
353+A converse scenario is splitting multiple streams from a single input into
354+multiple outputs:
355+@verbatim
356+┌──────────┬─────────────────────┐ ┌───────────────────┬───────────┐
357+│ demuxer │ │ packets │ │ muxer 0 │
358+╞══════════╡ elementary stream 0 ├─────────⮞│elementary stream 0╞═══════════╡
359+│ │ │ │ │OUTPUT0.mp4│
360+│INPUT.mkv ├─────────────────────┤ └───────────────────┴───────────┘
361+│ │ │ packets ┌───────────────────┬───────────┐
362+│ │ elementary stream 1 ├─────────⮞│ │ muxer 1 │
363+│ │ │ │elementary stream 0╞═══════════╡
364+└──────────┴─────────────────────┘ │ │OUTPUT1.mp4│
365+ └───────────────────┴───────────┘
192366 @end verbatim
367+built with
368+@example
369+ffmpeg -i INPUT.mkv -map 0:0 -c copy OUTPUT0.mp4 -map 0:1 -c copy OUTPUT1.mp4
370+@end example
371+Note how a separate instance of the @option{-c} option is needed for every
372+output file even though their values are the same. This is because non-global
373+options (which is most of them) only apply in the context of the file before
374+which they are placed.
193375
194−Complex filtergraphs are configured with the @option{-filter_complex} option.
195−Note that this option is global, since a complex filtergraph, by its nature,
196−cannot be unambiguously associated with a single stream or file.
376+These examples can of course be further generalized into arbitrary remappings
377+of any number of inputs into any number of outputs.
197378
198−The @option{-lavfi} option is equivalent to @option{-filter_complex}.
379+@section Trancoding
380+@emph{Transcoding} is the process of decoding a stream and then encoding it
381+again. Since encoding tends to be computationally expensive and in most cases
382+degrades the stream quality (i.e. it is @emph{lossy}), you should only transcode
383+when you need to and perform streamcopy otherwise. Typical reasons to transcode
384+are:
199385
200−A trivial example of a complex filtergraph is the @code{overlay} filter, which
201−has two video inputs and one video output, containing one video overlaid on top
202−of the other. Its audio counterpart is the @code{amix} filter.
386+@itemize
387+@item
388+applying filters - e.g. resizing, deinterlacing, or overlaying video; resampling
389+or mixing audio;
203390
204−@section Stream copy
205−Stream copy is a mode selected by supplying the @code{copy} parameter to the
206−@option{-codec} option. It makes @command{ffmpeg} omit the decoding and encoding
207−step for the specified stream, so it does only demuxing and muxing. It is useful
208−for changing the container format or modifying container-level metadata. The
209−diagram above will, in this case, simplify to this:
391+@item
392+you want to feed the stream to something that cannot decode the original codec.
393+@end itemize
394+Note that @command{ffmpeg} will transcode all audio, video, and subtitle streams
395+unless you specify @option{-c copy} for them.
210396
397+Consider an example pipeline that reads an input file with one audio and one
398+video stream, transcodes the video and copies the audio into a single output
399+file. This can be schematically represented as follows
211400 @verbatim
212− _______ ______________ ________
213−| | | | | |
214−| input | demuxer | encoded data | muxer | output |
215−| file | ---------> | packets | -------> | file |
216−|_______| |______________| |________|
401+┌──────────┬─────────────────────┐
402+│ demuxer │ │ audio packets
403+╞══════════╡ stream 0 (audio) ├─────────────────────────────────────╮
404+│ │ │ │
405+│INPUT.mkv ├─────────────────────┤ video ┌─────────┐ raw │
406+│ │ │ packets │ video │ video frames │
407+│ │ stream 1 (video) ├─────────⮞│ decoder ├──────────────╮ │
408+│ │ │ │ │ │ │
409+└──────────┴─────────────────────┘ └─────────┘ │ │
410+ ▼ ▼
411+ │ │
412+┌──────────┬─────────────────────┐ video ┌─────────┐ │ │
413+│ muxer │ │ packets │ video │ │ │
414+╞══════════╡ stream 0 (video) │⮜─────────┤ encoder ├──────────────╯ │
415+│ │ │ │(libx264)│ │
416+│OUTPUT.mp4├─────────────────────┤ └─────────┘ │
417+│ │ │ │
418+│ │ stream 1 (audio) │⮜────────────────────────────────────╯
419+│ │ │
420+└──────────┴─────────────────────┘
421+@end verbatim
422+and implemented with the following commandline:
423+@example
424+ffmpeg -i INPUT.mkv -map 0:v -map 0:a -c:v libx264 -c:a copy OUTPUT.mp4
425+@end example
426+Note how it uses stream specifiers @code{:v} and @code{:a} to select input
427+streams and apply different values of the @option{-c} option to them; see the
428+@ref{Stream specifiers} section for more details.
429+
430+
431+@section Filtering
432+
433+When transcoding, audio and video streams can be filtered before encoding, with
434+either a @emph{simple} or @emph{complex} filtergraph.
435+
436+@subsection Simple filtergraphs
217437
438+Simple filtergraphs are those that have exactly one input and output, both of
439+the same type (audio or video). They are configured with the per-stream
440+@option{-filter} option (with @option{-vf} and @option{-af} aliases for
441+@option{-filter:v} (video) and @option{-filter:a} (audio) respectively). Note
442+that simple filtergraphs are tied to their output stream, so e.g. if you have
443+multiple audio streams, @option{-af} will create a separate filtergraph for each
444+one.
445+
446+Taking the trancoding example from above, adding filtering (and omitting audio,
447+for clarity) makes it look like this:
448+@verbatim
449+┌──────────┬───────────────┐
450+│ demuxer │ │ ┌─────────┐
451+╞══════════╡ video stream │ packets │ video │ frames
452+│INPUT.mkv │ ├─────────⮞│ decoder ├─────⮞───╮
453+│ │ │ └─────────┘ │
454+└──────────┴───────────────┘ │
455+ ╭───────────⮜───────────╯
456+ │ ┌────────────────────────┐
457+ │ │ simple filtergraph │
458+ │ ╞════════════════════════╡
459+ │ │ ┌───────┐ ┌───────┐ │
460+ ╰──⮞├─⮞│ yadif ├─⮞│ scale ├─⮞├╮
461+ │ └───────┘ └───────┘ ││
462+ └────────────────────────┘│
463+ │
464+ │
465+┌──────────┬───────────────┐ video ┌─────────┐ │
466+│ muxer │ │ packets │ video │ │
467+╞══════════╡ video stream │⮜─────────┤ encoder ├───────⮜───────╯
468+│OUTPUT.mp4│ │ │ │
469+│ │ │ └─────────┘
470+└──────────┴───────────────┘
218471 @end verbatim
219472
220−Since there is no decoding or encoding, it is very fast and there is no quality
221−loss. However, it might not work in some cases because of many factors. Applying
222−filters is obviously also impossible, since filters work on uncompressed data.
473+@subsection Complex filtergraphs
474+
475+Complex filtergraphs are those which cannot be described as simply a linear
476+processing chain applied to one stream. This is the case, for example, when the
477+graph has more than one input and/or output, or when output stream type is
478+different from input. Complex filtergraphs are configured with the
479+@option{-filter_complex} option. Note that this option is global, since a
480+complex filtergraph, by its nature, cannot be unambiguously associated with a
481+single stream or file. Each instance of @option{-filter_complex} creates a new
482+complex filtergraph, and there can be any number of them.
483+
484+A trivial example of a complex filtergraph is the @code{overlay} filter, which
485+has two video inputs and one video output, containing one video overlaid on top
486+of the other. Its audio counterpart is the @code{amix} filter.
223487
488+@anchor{Loopback decoders}
224489 @section Loopback decoders
225490 While decoders are normally associated with demuxer streams, it is also possible
226491 to create "loopback" decoders that decode the output from some encoder and allow

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t