History

doc/ffmpeg: rewrite the detailed description chapter

Split it into sections that describe in detail * the components of the transcoding pipeline * the main features it handles, in order of complexity * streamcopy * transcoding * filtering Replace the current confusing/misleading diagrams with new ones that actually reflect the program components and data flow between them.

Anton Khirnov committed 2 years ago

Commit 794308c1 parent f3391691 file · +366 −101Browse files

Unified · Split

doc/ffmpeg.texi+366−101
@@ -87,140 +87,405 @@ The format option may be needed for raw input files. · 86 unmodified lines
87@chapter Detailed description87@chapter Detailed description
88@c man begin DETAILED DESCRIPTION88@c man begin DETAILED DESCRIPTION
89 89
90The transcoding process in @command{ffmpeg} for each output can be described by90@command{ffmpeg} builds a transcoding pipeline out of the components listed
91the following diagram:91below. The program's operation then consists of input data chunks flowing from
92the sources down the pipes towards the sinks, while being transformed by the
93components they encounter along the way.
92 94
93@verbatim95The following kinds of components are available:
94 _______ ______________96@itemize
95| | | |97@item
96| input | demuxer | encoded data | decoder98@emph{Demuxers} (short for "demultiplexers") read an input source in order to
97| file | ---------> | packets | -----+99extract
98|_______| |______________| |
99 v
100 _________
101 | |
102 | decoded |
103 | frames |
104 |_________|
105 ________ ______________ |
106| | | | |
107| output | <-------- | encoded data | <----+
108| file | muxer | packets | encoder
109|________| |______________|
110 100
101@itemize
102@item
103global properties such as metadata or chapters;
104@item
105list of input elementary streams and their properties
106@end itemize
107
108One demuxer instance is created for each @option{-i} option, and sends encoded
109@emph{packets} to @emph{decoders} or @emph{muxers}.
110
111In other literature, demuxers are sometimes called @emph{splitters}, because
112their main function is splitting a file into elementary streams (though some
113files only contain one elementary stream).
111 114
115A schematic representation of a demuxer looks like this:
116@verbatim
117┌──────────┬───────────────────────┐
118│ demuxer │ │ packets for stream 0
119╞══════════╡ elementary stream 0 ├──────────────────────⮞
120│ │ │
121│ global ├───────────────────────┤
122│properties│ │ packets for stream 1
123│ and │ elementary stream 1 ├──────────────────────⮞
124│ metadata │ │
125│ ├───────────────────────┤
126│ │ │
127│ │ ........... │
128│ │ │
129│ ├───────────────────────┤
130│ │ │ packets for stream N
131│ │ elementary stream N ├──────────────────────⮞
132│ │ │
133└──────────┴───────────────────────┘
134 ⯅
135 │
136 │ read from file, network stream,
137 │ grabbing device, etc.
138 │
112@end verbatim139@end verbatim
113 140
114@command{ffmpeg} calls the libavformat library (containing demuxers) to read141@item
115input files and get packets containing encoded data from them. When there are142@emph{Decoders} receive encoded (compressed) @emph{packets} for an audio, video,
116multiple input files, @command{ffmpeg} tries to keep them synchronized by143or subtitle elementary stream, and decode them into raw @emph{frames} (arrays of
117tracking lowest timestamp on any active input stream.144pixels for video, PCM for audio). A decoder is typically associated with (and
145receives its input from) an elementary stream in a @emph{demuxer}, but sometimes
146may also exist on its own (see @ref{Loopback decoders}).
118 147
119Encoded packets are then passed to the decoder (unless streamcopy is selected148A schematic representation of a decoder looks like this:
120for the stream, see further for a description). The decoder produces149@verbatim
121uncompressed frames (raw video/PCM audio/...) which can be processed further by150 ┌─────────┐
122filtering (see next section). After filtering, the frames are passed to the151 packets │ │ raw frames
123encoder, which encodes them and outputs encoded packets. Finally, those are152─────────⮞│ decoder ├────────────⮞
124passed to the muxer, which writes the encoded packets to the output file.153 │ │
154 └─────────┘
155@end verbatim
125 156
126@section Filtering157@item
127Before encoding, @command{ffmpeg} can process raw audio and video frames using158@emph{Filtergraphs} process and transform raw audio or video @emph{frames}. A
128filters from the libavfilter library. Several chained filters form a filter159filtergraph consists of one or more individual @emph{filters} linked into a
129graph. @command{ffmpeg} distinguishes between two types of filtergraphs:160graph. Filtergraphs come in two flavors - @emph{simple} and @emph{complex},
130simple and complex.161configured with the @option{-filter} and @option{-filter_complex} options,
162respectively.
163
164A simple filtergraph is associated with an @emph{output elementary stream}; it
165receives the input to be filtered from a @emph{decoder} and sends filtered
166output to that output stream's @emph{encoder}.
167
168A simple video filtergraph that performs deinterlacing (using the @code{yadif}
169deinterlacer) followed by resizing (using the @code{scale} filter) can look like
170this:
171@verbatim
131 172
132@subsection Simple filtergraphs173 ┌────────────────────────┐
133Simple filtergraphs are those that have exactly one input and output, both of174 │ simple filtergraph │
134the same type. In the above diagram they can be represented by simply inserting175 frames from ╞════════════════════════╡ frames for
135an additional step between decoding and encoding:176 a decoder │ ┌───────┐ ┌───────┐ │ an encoder
177────────────⮞├─⮞│ yadif ├─⮞│ scale ├─⮞│────────────⮞
178 │ └───────┘ └───────┘ │
179 └────────────────────────┘
180@end verbatim
136 181
182A complex filtergraph is standalone and not associated with any specific stream.
183It may have multiple (or zero) inputs, potentially of different types (audio or
184video), each of which receiving data either from a decoder or another complex
185filtergraph's output. It also has one or more outputs that feed either an
186encoder or another complex filtergraph's input.
187
188The following example diagram represents a complex filtergraph with 3 inputs and
1892 outputs (all video):
137@verbatim190@verbatim
138 _________ ______________191 ┌─────────────────────────────────────────────────┐
139| | | |192 │ complex filtergraph │
140| decoded | | encoded data |193 ╞═════════════════════════════════════════════════╡
141| frames |\ _ | packets |194 frames ├───────┐ ┌─────────┐ ┌─────────┐ ┌────────┤ frames
142|_________| \ /||______________|195─────────⮞│input 0├─⮞│ overlay ├─────⮞│ overlay ├─⮞│output 0├────────⮞
143 \ __________ /196 ├───────┘ │ │ │ │ └────────┤
144 simple _\|| | / encoder197 frames ├───────┐╭⮞│ │ ╭⮞│ │ │
145 filtergraph | filtered |/198─────────⮞│input 1├╯ └─────────┘ │ └─────────┘ │
146 | frames |199 ├───────┘ │ │
147 |__________|200 frames ├───────┐ ┌─────┐ ┌─────┬─╯ ┌────────┤ frames
201─────────⮞│input 2├⮞│scale├⮞│split├───────────────⮞│output 1├────────⮞
202 ├───────┘ └─────┘ └─────┘ └────────┤
203 └─────────────────────────────────────────────────┘
204@end verbatim
205Frames from second input are overlaid over those from the first. Frames from the
206third input are rescaled, then the duplicated into two identical streams. One of
207them is overlaid over the combined first two inputs, with the result exposed as
208the filtergraph's first output. The other duplicate ends up being the
209filtergraph's second output.
148 210
211@item
212@emph{Encoders} receive raw audio, video, or subtitle @emph{frames} and encode
213them into encoded @emph{packets}. The encoding (compression) process is
214typically @emph{lossy} - it degrades stream quality to make the output smaller;
215some encoders are @emph{lossless}, but at the cost of much higher output size. A
216video or audio encoder receives its input from some filtergraph's output,
217subtitle encoders receive input from a decoder (since subtitle filtering is not
218supported yet). Every encoder is associated with some muxer's @emph{output
219elementary stream} and sends its output to that muxer.
220
221A schematic representation of an encoder looks like this:
222@verbatim
223 ┌─────────┐
224 raw frames │ │ packets
225────────────⮞│ encoder ├─────────⮞
226 │ │
227 └─────────┘
149@end verbatim228@end verbatim
150 229
151Simple filtergraphs are configured with the per-stream @option{-filter} option230@item
152(with @option{-vf} and @option{-af} aliases for video and audio respectively).231@emph{Muxers} (short for "multiplexers") receive encoded @emph{packets} for
153A simple filtergraph for video can look for example like this:232their elementary streams from encoders (the @emph{transcoding} path) or directly
233from demuxers (the @emph{streamcopy} path), interleave them (when there is more
234than one elementary stream), and write the resulting bytes into the output file
235(or pipe, network stream, etc.).
154 236
237A schematic representation of a muxer looks like this:
155@verbatim238@verbatim
156 _______ _____________ _______ ________239 ┌──────────────────────┬───────────┐
157| | | | | | | |240 packets for stream 0 │ │ muxer │
158| input | ---> | deinterlace | ---> | scale | ---> | output |241──────────────────────⮞│ elementary stream 0 ╞═══════════╡
159|_______| |_____________| |_______| |________|242 │ │ │
243 ├──────────────────────┤ global │
244 packets for stream 1 │ │properties │
245──────────────────────⮞│ elementary stream 1 │ and │
246 │ │ metadata │
247 ├──────────────────────┤ │
248 │ │ │
249 │ ........... │ │
250 │ │ │
251 ├──────────────────────┤ │
252 packets for stream N │ │ │
253──────────────────────⮞│ elementary stream N │ │
254 │ │ │
255 └──────────────────────┴─────┬─────┘
256 │
257 write to file, network stream, │
258 grabbing device, etc. │
259 │
260 ▼
261@end verbatim
262
263@end itemize
160 264
265@section Streamcopy
266The simplest pipeline in @command{ffmpeg} is single-stream
267@emph{streamcopy}, that is copying one @emph{input elementary stream}'s packets
268without decoding, filtering, or encoding them. As an example, consider an input
269file called @file{INPUT.mkv} with 3 elementary streams, from which we take the
270second and write it to file @file{OUTPUT.mp4}. A schematic representation of
271such a pipeline looks like this:
272@verbatim
273┌──────────┬─────────────────────┐
274│ demuxer │ │ unused
275╞══════════╡ elementary stream 0 ├────────╳
276│ │ │
277│INPUT.mkv ├─────────────────────┤ ┌──────────────────────┬───────────┐
278│ │ │ packets │ │ muxer │
279│ │ elementary stream 1 ├─────────⮞│ elementary stream 0 ╞═══════════╡
280│ │ │ │ │OUTPUT.mp4 │
281│ ├─────────────────────┤ └──────────────────────┴───────────┘
282│ │ │ unused
283│ │ elementary stream 2 ├────────╳
284│ │ │
285└──────────┴─────────────────────┘
161@end verbatim286@end verbatim
162 287
163Note that some filters change frame properties but not frame contents. E.g. the288The above pipeline can be constructed with the following commandline:
164@code{fps} filter in the example above changes number of frames, but does not289@example
165touch the frame contents. Another example is the @code{setpts} filter, which290ffmpeg -i INPUT.mkv -map 0:1 -c copy OUTPUT.mp4
166only sets timestamps and otherwise passes the frames unchanged.291@end example
167 292
168@subsection Complex filtergraphs293In this commandline
169Complex filtergraphs are those which cannot be described as simply a linear294@itemize
170processing chain applied to one stream. This is the case, for example, when the graph has295
171more than one input and/or output, or when output stream type is different from296@item
172input. They can be represented with the following diagram:297there is a single input @file{INPUT.mkv};
298
299@item
300there are no input options for this input;
301
302@item
303there is a single output @file{OUTPUT.mp4};
304
305@item
306there are two output options for this output:
173 307
308@itemize
309@item
310@code{-map 0:1} selects the input stream to be used - from input with index 0
311(i.e. the first one) the stream with index 1 (i.e. the second one);
312
313@item
314@code{-c copy} selects the @code{copy} encoder, i.e. streamcopy with no decoding
315or encoding.
316@end itemize
317
318@end itemize
319
320Streamcopy is useful for changing the elementary stream count, container format,
321or modifying container-level metadata. Since there is no decoding or encoding,
322it is very fast and there is no quality loss. However, it might not work in some
323cases because of a variety of factors (e.g. certain information required by the
324target container is not available in the source). Applying filters is obviously
325also impossible, since filters work on decoded frames.
326
327More complex streamcopy scenarios can be constructed - e.g. combining streams
328from two input files into a single output:
174@verbatim329@verbatim
175 _________330┌──────────┬────────────────────┐ ┌────────────────────┬───────────┐
176| |331│ demuxer 0│ │ packets │ │ muxer │
177| input 0 |\ __________332╞══════════╡elementary stream 0 ├────────⮞│elementary stream 0 ╞═══════════╡
178|_________| \ | |333│INPUT0.mkv│ │ │ │OUTPUT.mp4 │
179 \ _________ /| output 0 |334└──────────┴────────────────────┘ ├────────────────────┤ │
180 \ | | / |__________|335┌──────────┬────────────────────┐ │ │ │
181 _________ \| complex | /336│ demuxer 1│ │ packets │elementary stream 1 │ │
182| | | |/337╞══════════╡elementary stream 0 ├────────⮞│ │ │
183| input 1 |---->| filter |\338│INPUT1.aac│ │ └────────────────────┴───────────┘
184|_________| | | \ __________339└──────────┴────────────────────┘
185 /| graph | \ | |340@end verbatim
186 / | | \| output 1 |341that can be built by the commandline
187 _________ / |_________| |__________|342@example
188| | /343ffmpeg -i INPUT0.mkv -i INPUT1.aac -map 0:0 -map 1:0 -c copy OUTPUT.mp4
189| input 2 |/344@end example
190|_________|345
346The output @option{-map} option is used twice here, creating two streams in the
347output file - one fed by the first input and one by the second. The single
348instance of the @option{-c} option selects streamcopy for both of those streams.
349You could also use multiple instances of this option together with
350@ref{Stream specifiers} to apply different values to each stream, as will be
351demonstrated in following sections.
191 352
353A converse scenario is splitting multiple streams from a single input into
354multiple outputs:
355@verbatim
356┌──────────┬─────────────────────┐ ┌───────────────────┬───────────┐
357│ demuxer │ │ packets │ │ muxer 0 │
358╞══════════╡ elementary stream 0 ├─────────⮞│elementary stream 0╞═══════════╡
359│ │ │ │ │OUTPUT0.mp4│
360│INPUT.mkv ├─────────────────────┤ └───────────────────┴───────────┘
361│ │ │ packets ┌───────────────────┬───────────┐
362│ │ elementary stream 1 ├─────────⮞│ │ muxer 1 │
363│ │ │ │elementary stream 0╞═══════════╡
364└──────────┴─────────────────────┘ │ │OUTPUT1.mp4│
365 └───────────────────┴───────────┘
192@end verbatim366@end verbatim
367built with
368@example
369ffmpeg -i INPUT.mkv -map 0:0 -c copy OUTPUT0.mp4 -map 0:1 -c copy OUTPUT1.mp4
370@end example
371Note how a separate instance of the @option{-c} option is needed for every
372output file even though their values are the same. This is because non-global
373options (which is most of them) only apply in the context of the file before
374which they are placed.
193 375
194Complex filtergraphs are configured with the @option{-filter_complex} option.376These examples can of course be further generalized into arbitrary remappings
195Note that this option is global, since a complex filtergraph, by its nature,377of any number of inputs into any number of outputs.
196cannot be unambiguously associated with a single stream or file.
197 378
198The @option{-lavfi} option is equivalent to @option{-filter_complex}.379@section Trancoding
380@emph{Transcoding} is the process of decoding a stream and then encoding it
381again. Since encoding tends to be computationally expensive and in most cases
382degrades the stream quality (i.e. it is @emph{lossy}), you should only transcode
383when you need to and perform streamcopy otherwise. Typical reasons to transcode
384are:
199 385
200A trivial example of a complex filtergraph is the @code{overlay} filter, which386@itemize
201has two video inputs and one video output, containing one video overlaid on top387@item
202of the other. Its audio counterpart is the @code{amix} filter.388applying filters - e.g. resizing, deinterlacing, or overlaying video; resampling
389or mixing audio;
203 390
204@section Stream copy391@item
205Stream copy is a mode selected by supplying the @code{copy} parameter to the392you want to feed the stream to something that cannot decode the original codec.
206@option{-codec} option. It makes @command{ffmpeg} omit the decoding and encoding393@end itemize
207step for the specified stream, so it does only demuxing and muxing. It is useful394Note that @command{ffmpeg} will transcode all audio, video, and subtitle streams
208for changing the container format or modifying container-level metadata. The395unless you specify @option{-c copy} for them.
209diagram above will, in this case, simplify to this:
210 396
397Consider an example pipeline that reads an input file with one audio and one
398video stream, transcodes the video and copies the audio into a single output
399file. This can be schematically represented as follows
211@verbatim400@verbatim
212 _______ ______________ ________401┌──────────┬─────────────────────┐
213| | | | | |402│ demuxer │ │ audio packets
214| input | demuxer | encoded data | muxer | output |403╞══════════╡ stream 0 (audio) ├─────────────────────────────────────╮
215| file | ---------> | packets | -------> | file |404│ │ │ │
216|_______| |______________| |________|405│INPUT.mkv ├─────────────────────┤ video ┌─────────┐ raw │
406│ │ │ packets │ video │ video frames │
407│ │ stream 1 (video) ├─────────⮞│ decoder ├──────────────╮ │
408│ │ │ │ │ │ │
409└──────────┴─────────────────────┘ └─────────┘ │ │
410 ▼ ▼
411 │ │
412┌──────────┬─────────────────────┐ video ┌─────────┐ │ │
413│ muxer │ │ packets │ video │ │ │
414╞══════════╡ stream 0 (video) │⮜─────────┤ encoder ├──────────────╯ │
415│ │ │ │(libx264)│ │
416│OUTPUT.mp4├─────────────────────┤ └─────────┘ │
417│ │ │ │
418│ │ stream 1 (audio) │⮜────────────────────────────────────╯
419│ │ │
420└──────────┴─────────────────────┘
421@end verbatim
422and implemented with the following commandline:
423@example
424ffmpeg -i INPUT.mkv -map 0:v -map 0:a -c:v libx264 -c:a copy OUTPUT.mp4
425@end example
426Note how it uses stream specifiers @code{:v} and @code{:a} to select input
427streams and apply different values of the @option{-c} option to them; see the
428@ref{Stream specifiers} section for more details.
429
430
431@section Filtering
432
433When transcoding, audio and video streams can be filtered before encoding, with
434either a @emph{simple} or @emph{complex} filtergraph.
435
436@subsection Simple filtergraphs
217 437
438Simple filtergraphs are those that have exactly one input and output, both of
439the same type (audio or video). They are configured with the per-stream
440@option{-filter} option (with @option{-vf} and @option{-af} aliases for
441@option{-filter:v} (video) and @option{-filter:a} (audio) respectively). Note
442that simple filtergraphs are tied to their output stream, so e.g. if you have
443multiple audio streams, @option{-af} will create a separate filtergraph for each
444one.
445
446Taking the trancoding example from above, adding filtering (and omitting audio,
447for clarity) makes it look like this:
448@verbatim
449┌──────────┬───────────────┐
450│ demuxer │ │ ┌─────────┐
451╞══════════╡ video stream │ packets │ video │ frames
452│INPUT.mkv │ ├─────────⮞│ decoder ├─────⮞───╮
453│ │ │ └─────────┘ │
454└──────────┴───────────────┘ │
455 ╭───────────⮜───────────╯
456 │ ┌────────────────────────┐
457 │ │ simple filtergraph │
458 │ ╞════════════════════════╡
459 │ │ ┌───────┐ ┌───────┐ │
460 ╰──⮞├─⮞│ yadif ├─⮞│ scale ├─⮞├╮
461 │ └───────┘ └───────┘ ││
462 └────────────────────────┘│
463 │
464 │
465┌──────────┬───────────────┐ video ┌─────────┐ │
466│ muxer │ │ packets │ video │ │
467╞══════════╡ video stream │⮜─────────┤ encoder ├───────⮜───────╯
468│OUTPUT.mp4│ │ │ │
469│ │ │ └─────────┘
470└──────────┴───────────────┘
218@end verbatim471@end verbatim
219 472
220Since there is no decoding or encoding, it is very fast and there is no quality473@subsection Complex filtergraphs
221loss. However, it might not work in some cases because of many factors. Applying474
222filters is obviously also impossible, since filters work on uncompressed data.475Complex filtergraphs are those which cannot be described as simply a linear
476processing chain applied to one stream. This is the case, for example, when the
477graph has more than one input and/or output, or when output stream type is
478different from input. Complex filtergraphs are configured with the
479@option{-filter_complex} option. Note that this option is global, since a
480complex filtergraph, by its nature, cannot be unambiguously associated with a
481single stream or file. Each instance of @option{-filter_complex} creates a new
482complex filtergraph, and there can be any number of them.
483
484A trivial example of a complex filtergraph is the @code{overlay} filter, which
485has two video inputs and one video output, containing one video overlaid on top
486of the other. Its audio counterpart is the @code{amix} filter.
223 487
488@anchor{Loopback decoders}
224@section Loopback decoders489@section Loopback decoders
225While decoders are normally associated with demuxer streams, it is also possible490While decoders are normally associated with demuxer streams, it is also possible
226to create "loopback" decoders that decode the output from some encoder and allow491to create "loopback" decoders that decode the output from some encoder and allow

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t