In input.c and output.c and many other places, swscale follows the rule of using 15-bit intermediate if output bpc is <= 8, and 19-bit (inside int32_t) intermediate otherwise. See e.g. the comments on hyScale() on swscale_internal.h. These are also the coefficients that yuv2gbrpf32_full_X_c() is using. In contrast to this, the plane init code in slice.c (function fill_ones) is assuming that we use 35-bit intermediates (inside 64-bit integers) for this case, seemingly added by commit b4967fc71c63eae8cd96f9c46cd3e1fbd705bbf9 with no further justification. This causes a mismatch whenever the implicitly initialized plane contents leak out to the output, e.g. when converting from grayscale to RGB.
Unified · Split
| @@ -194,14 +194,10 @@ static void fill_ones(SwsSlice *s, int n, int bpc) · 193 unmodified lines | ||
| 194 | 194 | for (i = 0; i < 4; ++i) { |
| 195 | 195 | size = s->plane[i].available_lines; |
| 196 | 196 | for (j = 0; j < size; ++j) { |
| 197 | − if (bpc == 16) { | |
| 197 | + if (bpc >= 16) { | |
| 198 | 198 | end = (n>>1) + 1; |
| 199 | 199 | for (k = 0; k < end; ++k) |
| 200 | 200 | ((int32_t*)(s->plane[i].line[j]))[k] = 1<<18; |
| 201 | − } else if (bpc == 32) { | |
| 202 | − end = (n>>2) + 1; | |
| 203 | − for (k = 0; k < end; ++k) | |
| 204 | − ((int64_t*)(s->plane[i].line[j]))[k] = 1LL<<34; | |
| 205 | 201 | } else { |
| 206 | 202 | end = n + 1; |
| 207 | 203 | for (k = 0; k < end; ++k) |