History

swscale/slice: fix init of 32 bpc planes

In input.c and output.c and many other places, swscale follows the rule of using 15-bit intermediate if output bpc is <= 8, and 19-bit (inside int32_t) intermediate otherwise. See e.g. the comments on hyScale() on swscale_internal.h. These are also the coefficients that yuv2gbrpf32_full_X_c() is using. In contrast to this, the plane init code in slice.c (function fill_ones) is assuming that we use 35-bit intermediates (inside 64-bit integers) for this case, seemingly added by commit b4967fc71c63eae8cd96f9c46cd3e1fbd705bbf9 with no further justification. This causes a mismatch whenever the implicitly initialized plane contents leak out to the output, e.g. when converting from grayscale to RGB.

Niklas Haas committed 1 year ago

Commit ce457bf1 parent d2096671 file · +1 −5Browse files

Unified · Split

libswscale/slice.c+1−5
@@ -194,14 +194,10 @@ static void fill_ones(SwsSlice *s, int n, int bpc) · 193 unmodified lines
194194 for (i = 0; i < 4; ++i) {
195195 size = s->plane[i].available_lines;
196196 for (j = 0; j < size; ++j) {
197− if (bpc == 16) {
197+ if (bpc >= 16) {
198198 end = (n>>1) + 1;
199199 for (k = 0; k < end; ++k)
200200 ((int32_t*)(s->plane[i].line[j]))[k] = 1<<18;
201− } else if (bpc == 32) {
202− end = (n>>2) + 1;
203− for (k = 0; k < end; ++k)
204− ((int64_t*)(s->plane[i].line[j]))[k] = 1LL<<34;
205201 } else {
206202 end = n + 1;
207203 for (k = 0; k < end; ++k)

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t