Commit graph

4467 commits

Author SHA1 Message Date
Joose Sainio c309b81532 [intra] Fix chroma bit counting 2022-06-28 15:26:41 +03:00
Joose Sainio aa2f412c20 [rd] Don't calculate cost for non-basic transforms that do not have coeffs 2022-06-28 15:26:36 +03:00
Joose Sainio 079d1e9aaf [debug] Disable this for now 2022-06-28 15:26:25 +03:00
Joose Sainio 03246c031e [cabac] Fix trskip writing 2022-06-28 15:26:19 +03:00
Joose Sainio c11e30fdb4 [mts] get mts constraints for bit counting 2022-06-28 15:26:11 +03:00
Joose Sainio 81df3ef2e0 [mts] Reset mts constraints 2022-06-28 15:25:21 +03:00
Joose Sainio 675eb5b508 [debug] remove debug prints 2022-06-28 15:25:15 +03:00
Joose Sainio 2f5fd05f88 [intra] fix warnings 2022-06-28 15:25:09 +03:00
Joose Sainio 418c65cbf1 [intra] count mts and tr_skip bits 2022-06-28 15:25:01 +03:00
Joose Sainio 804bf3afcb [intra] Limit rough search 2022-06-28 15:24:39 +03:00
Joose Sainio 4c66561ebe [debug] Temporarily move split flag counting 2022-06-28 15:24:32 +03:00
Joose Sainio 8c939c1405 [intra] Fix chroma cbf counting for 4x4 2022-06-28 15:24:25 +03:00
Joose Sainio 2feb3daaaf [intra] Count chroma mode bits 2022-06-28 15:24:18 +03:00
Joose Sainio dfbe9dad87 [intra] intra dist = MIN(satd_dist, sad_dist*2) 2022-06-28 15:24:12 +03:00
Joose Sainio 09832c3c85 [intra] fix various things 2022-06-28 15:24:02 +03:00
Joose Sainio 6fcec1fdd3 [mrl] [mip] initial rd improvement 2022-06-28 15:23:56 +03:00
Joose Sainio 1b3885d5be [intra] Chroma working 2022-06-28 15:23:36 +03:00
Joose Sainio 0575c9c592 [intra] Fix warnings 2022-06-28 15:23:25 +03:00
Joose Sainio 80ab347d93 [intra] Luma working 2022-06-28 15:22:47 +03:00
Joose Sainio 23fda23322 [intra] Count fractional bits with get_coeff_cabac_cost 2022-06-28 15:22:33 +03:00
Joose Sainio 6d080b215c [intra] WIP: improve search 2022-06-28 15:22:20 +03:00
Joose Sainio c16f42e9df [satd] DC scaling 2022-06-28 15:20:56 +03:00
Joose Sainio 480a907c07 [lfnst] change kvz_ to uvg_ 2022-06-28 15:20:44 +03:00
siivonek 7abbf14d50 [lfnst] Fix lfnst constraint calculation check. Constraints will now be calculated regardless of tr_idx value. 2022-06-28 15:20:44 +03:00
siivonek d15fb473a2 [lfnst] Fix error in mts skip check. Skip value was derived from tr_skip instead of correct tr_idx. 2022-06-28 15:20:44 +03:00
siivonek 4d3688cf17 [lfnst] Fix error in luma tr_skip check. All cbf was checked when only luma cbf needed to be checked. 2022-06-28 15:20:43 +03:00
siivonek 9171e03a53 [lfnst] Add cu intra check to lfnst calls. 2022-06-28 15:20:43 +03:00
siivonek 4fee153318 [lfnst] Fix assert. No need for >= 0 check for unsigned integer. 2022-06-28 15:20:43 +03:00
siivonek b836c6f529 [lfnst] Fix error in is_lfnst_allowed. Error caused cabac to fail on depth 4 chroma blocks. 2022-06-28 15:20:43 +03:00
siivonek cf4343989f [lfnst] Add early termination to mts/lfnst search if MTS_SKIP is selected. Resolve TODOs. 2022-06-28 15:20:42 +03:00
siivonek e056e87ca0 [lfnst] Fix help text missing line break. Fix luma transform skip check. 2022-06-28 15:20:42 +03:00
siivonek 7551952eeb [lfnst] Fix separate tree type checks. Fix lfnst search. Lfnst is now done in outer loop. 2022-06-28 15:20:42 +03:00
siivonek 896bc093d3 [lfnst] Remove unused stuff from lfnst constraint check. Use temp values for storing lfnst constraints during search instead of updating values inside pred cu. 2022-06-28 15:20:41 +03:00
siivonek 7b6b6586c6 [lfnst] Implement lfnst constraint check during search. 2022-06-28 15:20:41 +03:00
siivonek 64c5cfb127 [lfnst] Fix error in scan table selection. 2022-06-28 15:20:41 +03:00
siivonek 1c5f7f9dfc [lfnst] Fix errors in quantization code. Lfnst was called in wrong place. Fix error in lfnst algorithm, 16 bit integer overflow issue. 2022-06-28 15:20:41 +03:00
siivonek 2975b1fef0 [lfnst] WIP Implement lfnst search. Resolve some TODOs. 2022-06-28 15:20:40 +03:00
siivonek de18eef956 [lfnst] Chroma transform data is no longer stored in top left block. Remove -4x-4 coordinate offset. 2022-06-28 15:20:40 +03:00
siivonek 76947c4c41 [lfnst] Fix rebase errors. 2022-06-28 15:20:40 +03:00
siivonek b9c71b7f01 [lfnst] Fix cabac write issues. Lfnst constraints needed to be reset after encode_transform_coeff call. 2022-06-28 15:20:39 +03:00
siivonek ca2890d7e9 [lfnst] Fix error in inv lfnst call in quant generic. Add lfnst calls to quant avx2. Improve lfnst flag sps write. 2022-06-28 15:20:39 +03:00
siivonek 7928149d05 [lfnst] Fix CI errors. Had to remove const modifier from function call. 2022-06-28 15:20:39 +03:00
siivonek c1190e47d6 [lfnst] Fix cabac write issues. Remove is_luma from encode coeff function call, that information is already passed with type. Change parameter name 'type' -> 'color' to better reflect its purpose. 2022-06-28 15:20:38 +03:00
siivonek e3d92a5296 [lfnst] Add separate function to check if lfnst can be used with ISP. 2022-06-28 15:20:38 +03:00
siivonek 7a3cb69f45 [lfnst] Add CI test. 2022-06-28 15:20:38 +03:00
siivonek acfc28d7ce [lfnst] Fix CI errors. Lfnst tables didn't have kvz prefix. 2022-06-28 15:20:37 +03:00
siivonek ee9258b2d9 [lfnst] WIP Fix errors in lfnst cabac write. 2022-06-28 15:20:37 +03:00
siivonek 560be8c5b4 [lfnst] Fix writing sps lfnst flag. 2022-06-28 15:20:37 +03:00
siivonek 835f576eef [lfnst] Fix error in writing lfnst constraints. Added check to test if CU data exists before referencing it. 2022-06-28 15:20:37 +03:00
siivonek e11fd3154d [lfnst] Implement cabac write. 2022-06-28 15:20:36 +03:00