Since c737f6edcef74a64f4d0ebcefa970bd31266d512 prescreening is nevertheless run because of a wrong check: "if (s->prescreen > 0)". s->prescreen is an array of two function pointers that is contained in the context and comparing it with 0 (i.e. NULL) is actually undefined behaviour, because NULL and s->prescreen do not point to the same object (NULL after all never points to any object). Nevertheless both Clang as well as GCC compile this to code that treat s->prescreen > 0 as true, leading to segfaults, because the code then tries to access the -1th member of an array. This commit fixes the check as well as another such check a few lines below. (Found via compiler warnings enabled by -pedantic: "ordered comparison between pointer and zero is an extension".)
Unified · Split
| @@ -637,7 +637,7 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) · 636 unmodified lines | ||
| 637 | 637 | width, 1, in_scale); |
| 638 | 638 | |
| 639 | 639 | for (int y = 0; y < slice_end - slice_start; y += 2) { |
| 640 | − if (s->prescreen > 0) | |
| 640 | + if (s->pscrn > 0) | |
| 641 | 641 | s->prescreen[s->pscrn > 1](ctx, srcbuf + (y / 2) * srcbuf_stride + 32, |
| 642 | 642 | srcbuf_stride, prescreen_buf, width, |
| 643 | 643 | &s->prescreener[s->pscrn - 1]); |
| @@ -649,7 +649,7 @@ static int filter_slice(AVFilterContext *ctx, void *arg, int jobnr, int nb_jobs) · 5 unmodified lines | ||
| 649 | 649 | prescreen_buf, width, |
| 650 | 650 | &s->coeffs[s->etype][s->nnsparam][s->nsize], s->qual == 2); |
| 651 | 651 | |
| 652 | − if (s->prescreen > 0) | |
| 652 | + if (s->pscrn > 0) | |
| 653 | 653 | interpolation(srcbuf + (y / 2) * srcbuf_stride + 32, |
| 654 | 654 | srcbuf_stride, |
| 655 | 655 | dstbuf + (y / 2) * dstbuf_stride, |