History

avfilter/vf_nnedi: Fix segfault when prescreening is disabled

Since c737f6edcef74a64f4d0ebcefa970bd31266d512 prescreening is nevertheless run because of a wrong check: "if (s->prescreen > 0)". s->prescreen is an array of two function pointers that is contained in the context and comparing it with 0 (i.e. NULL) is actually undefined behaviour, because NULL and s->prescreen do not point to the same object (NULL after all never points to any object). Nevertheless both Clang as well as GCC compile this to code that treat s->prescreen > 0 as true, leading to segfaults, because the code then tries to access the -1th member of an array. This commit fixes the check as well as another such check a few lines below. (Found via compiler warnings enabled by -pedantic: "ordered comparison between pointer and zero is an extension".)

Andreas Rheinhardt committed 5 years ago

Commit 2bcec401 parent 26870701 file · +2 −2Browse files

Unified · Split

libavfilter/vf_nnedi.c+2−2
@@ -637,7 +637,7 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) · 636 unmodified lines
637637 width, 1, in_scale);
638638
639639 for (int y = 0; y < slice_end - slice_start; y += 2) {
640− if (s->prescreen > 0)
640+ if (s->pscrn > 0)
641641 s->prescreen[s->pscrn > 1](ctx, srcbuf + (y / 2) * srcbuf_stride + 32,
642642 srcbuf_stride, prescreen_buf, width,
643643 &s->prescreener[s->pscrn - 1]);
@@ -649,7 +649,7 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) · 5 unmodified lines
649649 prescreen_buf, width,
650650 &s->coeffs[s->etype][s->nnsparam][s->nsize], s->qual == 2);
651651
652− if (s->prescreen > 0)
652+ if (s->pscrn > 0)
653653 interpolation(srcbuf + (y / 2) * srcbuf_stride + 32,
654654 srcbuf_stride,
655655 dstbuf + (y / 2) * dstbuf_stride,

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t