transform skip vs. normal transform selection criteria might need more work, currently both are calculated for each 4x4 block and SAD+coeff_SSE is compared.
- Intra works. There is still something wrong in inter.
- Avoid horizontal deblocking of the rightmost 4 pixels in the LCU.
This is because vertical deblocking must be done for all pixels
before horizontal, but vertical deblocking can't be done for those
pixels before the next LCU is finished.
- Add separate deblocking of the rightmost pixels of the last LCU
after the LCU edge has been deblocked.
- This is a pretty ugly hack but will have to do for now.
I didn't take into account that the reference pixel on the top-left of the
LCU gets over written if we just replace the top reference pixels for
current LCU with the bottom reference pixels after doing the search.
To handle this I copy the pixel that gets overwritten to the vertical
reference pixels.
- This is necessary because after we add in-loop filters to be done per LCU,
the reconstruction buffer will have the deblocked pixels. We only need the
edge-pixels for intra prediction though so we just save those.
- Right now it only copies the pixels and passes them on to search, where
the copied pixels are asserted to be the same ones we copy from
reconstruction buffer.
- New yuv_t struct added for arrays of dynamic length. We might want to change
other buffers to use it or something like it in the future.
There is a lot of duplicated code due to handling random access and trailing
pictures separately. I merged the code for these two branches so it would be
easier to modify.
According to spec the end_of_slice_segment_flag is always coded, but in the
code it looked like it was not coded for the last LCU in picture. This was
due to the end_of_slice_segment_flag being coded inside cabac_flush, like it
is in HM. This is a bit silly so I moved it out of cabac_flush.
The relation between coefficients positions and coefficient group positions
was a big confusing due to the use of 16x16 diagonal coefficient mappings
also as coefficient group mappings.
- Moved all coefficient group mappings to their own const arrays and added
a new array the select the correct coefficient group mapping. This removes
special cases for 8x8 and 32x32 transform sizes.
- Removed all coefficient group mapping initialization from init_sig_last_scan.
- Removed 128x128 and 64x64 from regular coefficient group array as those
transform sizes don't exist anymore in HEVC.
- Moved NxN search to be done on the same level as other searches, as it's
really not any different from 2Nx2N.
- Produces working bitstream but reconstruction is different.
Moves CABAC context initialization to take place before search. This fixes
an issue with RDOQ returning different coefficients for identical adjacent
frames.
- This actually probably worsens BD-rate a little for all frames except the
first one because we were using last frames final CABAC context for every
LCU and now we are using initialized CABAC contexts. The fix is to encode
the LCU before we start compressing the next LCU so we can update CABAC
contexts.
Moves CABAC context initialization to take place before search. This fixes
an issue with RDOQ returning different coefficients for identical adjacent
frames.
- This actually probably worsens BD-rate a little for all frames except the
first one because we were using last frames final CABAC context for every
LCU and now we are using initialized CABAC contexts. The fix is to encode
the LCU before we start compressing the next LCU so we can update CABAC
contexts.
The coeff flags are no longer propagated upwards because encode_transform_tree
is being called from depth > 0. The fix is to initialize the whole coeff flag
array when the coeff flag is set.
