avcodec: Add explicit capability flag for encoder flushing

Previously, there was no way to flush an encoder such that after draining, the encoder could be used again. We generally suggested that clients teardown and replace the encoder instance in these situations. However, for at least some hardware encoders, the cost of this tear down/replace cycle is very high, which can get in the way of some use-cases - for example: segmented encoding with nvenc. To help address that use case, we added support for calling avcodec_flush_buffers() to nvenc and things worked in practice, although it was not clearly documented as to whether this should work or not. There was only one previous example of an encoder implementing the flush callback (audiotoolboxenc) and it's unclear if that was intentional or not. However, it was clear that calling avocdec_flush_buffers() on any other encoder would leave the encoder in an undefined state, and that's not great. As part of cleaning this up, this change introduces a formal capability flag for encoders that support flushing and ensures a flush call is a no-op for any other encoder. This allows client code to check if it is meaningful to call flush on an encoder before actually doing it. I have not attempted to separate the steps taken inside avcodec_flush_buffers() because it's not doing anything that's wrong for an encoder. But I did add a sanity check to reject attempts to flush a frame threaded encoder because I couldn't wrap my head around whether that code path was actually safe or not. As this combination doesn't exist today, we'll deal with it if it ever comes up.

PLPhilip Langdale committed 6 years ago