Commit graph

4647 commits

Author SHA1 Message Date
Joose Sainio 536c0ff2ef [quant] fix fast coeff cost 2023-08-15 13:01:54 +03:00
Joose Sainio cf5f7658a0 [mtt] fix 2023-08-15 13:01:53 +03:00
Joose Sainio 1668b65f3f [mtt] fix 2023-08-15 13:01:53 +03:00
Joose Sainio e931c096db [mtt] fix 2023-08-15 13:01:52 +03:00
Joose Sainio c590e5ec73 [mtt] also copy top right CU 2023-08-15 13:01:51 +03:00
Joose Sainio a1e7664db3 [mtt] temporarily disable zero coeff rdo 2023-08-15 13:01:50 +03:00
Joose Sainio 239ee88306 [mtt] fix 2023-08-15 13:01:49 +03:00
Joose Sainio 1cf1501542 [mtt] fix 2023-08-15 13:01:48 +03:00
Joose Sainio 924a93b60e [mtt] Only initialize higher depth ctus partially 2023-08-15 13:01:48 +03:00
Joose Sainio 274e71dff6 [transform] Simplify chroma transform search a bit 2023-08-15 13:01:47 +03:00
Joose Sainio 58c6af8c87 [mtt] Add function for easily getting all split cu_locs 2023-08-15 13:01:46 +03:00
Joose Sainio cfc6aebe3c [mtt] Remove depth from cu_info_t 2023-08-15 13:01:45 +03:00
Joose Sainio b14f6f98ec [mtt] Completely remove tr_depth 2023-08-15 13:00:53 +03:00
Joose Sainio 9a29d9ded3 [mtt] remove depth from cbf 2023-08-15 13:00:51 +03:00
Joose Sainio e3dbeda7f7 [mtt] remove dependency to depth from deblock 2023-08-15 13:00:50 +03:00
Joose Sainio 89af7bda8e [mtt] remove unnecessary depth dependency from split flag 2023-08-15 13:00:49 +03:00
Joose Sainio 0b6f666a1b [mtt] remove lfnst dependency to depth 2023-08-15 13:00:08 +03:00
Joose Sainio 790b1fad48 wip 2023-08-15 13:00:07 +03:00
Joose Sainio 6a0864839c [mtt] Actually remove the last width dependency to depth 2023-08-15 13:00:06 +03:00
Joose Sainio dcf879e5ed [mtt] remove all rest usages of deriving width and height from depth 2023-08-15 12:59:39 +03:00
Joose Sainio 26dcadc149 [mtt] change most if not all of search hierarchy to use cu_loc_t 2023-08-15 12:47:11 +03:00
siivonek 0ec16967a1 [isp] Fix reference building. When ISP was in use, not enough samples were generated. Uninitialized memory was referenced. Fix some typos. 2023-08-14 12:21:30 +03:00
siivonek b16c404362 [isp] Remove some obsolete TODOs and old commented out code. 2023-08-14 12:21:30 +03:00
siivonek 4633cb33b5 [isp] Fix error in mts tests. CU isp mode was not nullified before testing. 2023-08-14 12:21:29 +03:00
siivonek 95d73116f9 [isp] Fix some CI errors. Some const modifiers were discarded. 2023-08-14 12:21:29 +03:00
siivonek 90e2a17759 [lfnst] Fix LFNST error when MIP enabled. 2023-08-14 12:21:28 +03:00
siivonek 7005d222d5 [isp] Fix lfnst constraint check when ISP is used. Remove some obsolete comments. 2023-08-14 12:21:28 +03:00
siivonek 3c861e4c02 [isp] Fix search. Best LFNST and MTS modes were not selected correctly for ISP modes. 2023-08-14 12:21:27 +03:00
siivonek b4cc321349 [isp] Fix transform selection when MTS & ISP is used. Wrong transform was selected. Change mts parameter name to better reflect its purpose. 2023-08-14 12:21:26 +03:00
siivonek 85f6b00394 [isp] Add lfnst asserts. Fix error in MTS search. Fix chroma lfnst index when no coefficients present. 2023-08-14 12:21:26 +03:00
siivonek b9822398a0 [isp] Fix lfnst constraint checks when ISP is in use. Add some asserts. 2023-08-14 12:21:25 +03:00
siivonek 701257cdd2 [isp] Remove unnecessary code from forward dct 32. 2023-08-14 12:21:25 +03:00
siivonek eb6771321f [isp] Disable fast residual cost calculation from ISP related CI tests. 2023-08-14 12:21:24 +03:00
siivonek 89db34d4e0 [isp] Use TR_MAX_WIDTH in ISP checks instead of parameter. 2023-08-14 12:21:24 +03:00
siivonek c4bc2d6b10 [isp] Limit ISP search to block size 32. Size 64 is not allowed. 2023-08-14 12:21:23 +03:00
siivonek 8ae285322e [isp] Add CI tests. 2023-08-14 12:21:23 +03:00
siivonek 5713fbff1a [isp] Add ISP checks to search. LFNST can be used with ISP for larger blocks. Transform skip cannot be used with ISP. 2023-08-14 12:21:22 +03:00
siivonek 7282534879 [isp] Fix CI errors. 2023-08-14 12:21:22 +03:00
siivonek 01c4d1ddb0 [isp] Fix cabac issues. There are always four transform blocks even if there are only two ISP splits. Fix prediction issues. PDPC filter was applied when it should be disabled. Fix reference building issues. Left reference was built incorrectly for blocks with height 2. 2023-08-14 12:21:21 +03:00
siivonek b8e36bbc4a [isp] Fix storing cbfs for small ISP splits. Fix pdpc filtering. Cannot be used if width or height is less than 4. Fix dct related CI errors. 2023-08-14 12:21:20 +03:00
siivonek 99495c331b [isp] Fix some asserts to allow log2_dim 1 block sizes. Fix coefficient group scan order for small dimensions. 2023-08-14 12:21:19 +03:00
siivonek d39fddf0d8 [isp] Implement DCT for small blocks. 2023-08-14 12:21:19 +03:00
siivonek 910501012f [isp] Fix referene building for depth 2 blocks. Flip horizontal mode dimensions during prediction. Fix reference length during prediction when ISP enabled. 2023-08-14 12:21:18 +03:00
siivonek 7ba557af6b [isp] Fix luma cbf writing for ISP splits. Do not write luma cbf if first three splits had luma cbf 0. 2023-08-14 12:21:17 +03:00
siivonek a28e61eff7 [isp] Fix CI errors. 2023-08-14 12:21:17 +03:00
siivonek 4794104ecc [isp] Fix errors in reference building. Use cubic filter during prediction if ISP enabled. 2023-08-14 12:21:16 +03:00
Joose Sainio 662f31d61d [isp] Use correct coordinates for depth 4 chroma tu cost calculation 2023-08-14 12:17:36 +03:00
Joose Sainio 08942a5394 [tr-skip] fix transform skip flag writing 2023-08-14 12:17:35 +03:00
siivonek a261d4c5b3 [isp] WIP 2023-08-14 12:17:34 +03:00
siivonek 6340dfe4ce [isp] Fix mistake in pu_loc argument passing, was not used after passing. 2023-08-14 12:17:33 +03:00