Commit graph

4572 commits

Author SHA1 Message Date
siivonek 6975febccf [isp] Fix lfnst constraint checks when ISP is in use. Add some asserts. 2022-09-23 15:41:50 +03:00
siivonek 85029fc2ca [isp] Remove unnecessary code from forward dct 32. 2022-09-22 16:49:09 +03:00
siivonek 9671c4551e [isp] Disable fast residual cost calculation from ISP related CI tests. 2022-09-22 15:07:16 +03:00
siivonek 785ae95302 [isp] Use TR_MAX_WIDTH in ISP checks instead of parameter. 2022-09-22 14:28:40 +03:00
siivonek 18fca3798b [isp] Limit ISP search to block size 32. Size 64 is not allowed. 2022-09-21 12:55:07 +03:00
siivonek d059db1f65 [isp] Add CI tests. 2022-09-20 18:01:49 +03:00
siivonek e9c3351460 [isp] Add ISP checks to search. LFNST can be used with ISP for larger blocks. Transform skip cannot be used with ISP. 2022-09-20 17:44:10 +03:00
siivonek d553bbba8a [isp] Fix CI errors. 2022-09-20 16:15:23 +03:00
siivonek 80945c985a [isp] Fix cabac issues. There are always four transform blocks even if there are only two ISP splits. Fix prediction issues. PDPC filter was applied when it should be disabled. Fix reference building issues. Left reference was built incorrectly for blocks with height 2. 2022-09-20 15:16:14 +03:00
siivonek e8779fc175 [isp] Fix storing cbfs for small ISP splits. Fix pdpc filtering. Cannot be used if width or height is less than 4. Fix dct related CI errors. 2022-09-16 15:46:31 +03:00
siivonek c46abbdaa1 [isp] Fix some asserts to allow log2_dim 1 block sizes. Fix coefficient group scan order for small dimensions. 2022-09-16 10:37:51 +03:00
siivonek 19106bfced [isp] Implement DCT for small blocks. 2022-09-15 13:45:45 +03:00
siivonek c894e9c6b9 [isp] Fix referene building for depth 2 blocks. Flip horizontal mode dimensions during prediction. Fix reference length during prediction when ISP enabled. 2022-09-14 15:48:49 +03:00
siivonek 124cdd4e4a [isp] Fix luma cbf writing for ISP splits. Do not write luma cbf if first three splits had luma cbf 0. 2022-09-13 13:27:16 +03:00
siivonek 75d4de20e3 [isp] Fix CI errors. 2022-09-12 15:29:53 +03:00
siivonek dda36aef4c [isp] Fix errors in reference building. Use cubic filter during prediction if ISP enabled. 2022-09-12 14:41:44 +03:00
Joose Sainio 1bd3a4cb55 [isp] Use correct coordinates for depth 4 chroma tu cost calculation 2022-09-06 09:30:46 +03:00
Joose Sainio 68263f428b [tr-skip] fix transform skip flag writing 2022-09-06 08:49:22 +03:00
siivonek a7b26f329e [isp] WIP 2022-09-05 18:14:59 +03:00
siivonek 790deb2504 [isp] Fix mistake in pu_loc argument passing, was not used after passing. 2022-09-05 16:57:07 +03:00
Joose Sainio 60ccdbf853 [lfnst] Fix lfnst constraint checking for the new coeff order 2022-09-05 16:51:23 +03:00
Joose Sainio 17abc7a050 [lfnst] lfnst is not allowed for transform split 2022-09-05 12:04:21 +03:00
Joose Sainio bbae4e2b07 [jccr] Fix jccr coefficient copying 2022-09-05 11:20:09 +03:00
Joose Sainio 95b6ddbe67 [cabac] fix cbf_y context for tr splits 2022-09-05 11:10:55 +03:00
siivonek 42bea563e4 [isp] Add extra logic to reference building to accommodate ISP. Remove some asserts which were invalidated by ISP. 2022-09-02 16:10:42 +03:00
siivonek b49a4d0a57 [isp] Fix chroma coeff writing for ISP. 2022-08-31 14:54:30 +03:00
siivonek 043e502a4f [isp] Fix jccr coeffs. 2022-08-31 13:57:00 +03:00
siivonek 9557a20c7f [isp] Fix coeff cost calculation. Coeff arrays were indexed wrongly. 2022-08-31 13:53:17 +03:00
siivonek def855a027 [isp] Fix error in last sig coeff function call. Height was not used. Fix cbf writing. Fix transform skip flag writing. 2022-08-31 12:28:57 +03:00
siivonek a7be4496d5 [isp] Fix mistake in isp cbf writing. Loop index was increased twice. 2022-08-30 12:06:43 +03:00
siivonek c9291ec5a2 [isp] Fix mistake in function declaration. 2022-08-29 12:37:23 +03:00
siivonek 3c5479f99b [isp] Convert functions to handle new coeff array order. Add function for getting coeff array subset. Fix assert. 2022-08-26 16:39:57 +03:00
siivonek 125a3106d6 [isp] Use temporary coeff array when quantizing coeffs. After deriving coeffs, copy temp coeffs from linear order to correct arrays with cu order. 2022-08-24 18:45:13 +03:00
siivonek e245c2ac6e [isp] Fix assert. 2022-08-24 17:12:50 +03:00
siivonek efc0f13501 [isp] Redo call hierarchy to include x, y coordinates. 2022-08-24 17:12:36 +03:00
siivonek 7d7e22fbfb [isp] Keep lfnst constraint up to date during search. 2022-08-24 16:00:53 +03:00
siivonek 4ffbbee606 [isp] Fix chroma width error when writing cu loc. Remove redundant IPS mode checks. 2022-08-24 15:40:14 +03:00
siivonek 1c989906e5 [isp] Fix mistake in setting cbfs. Skip stting if ISP is not used. 2022-08-24 15:11:01 +03:00
siivonek bc3bf8a08c [isp] Set cbfs for isp splits after search. Add helper function for isp split number. 2022-08-24 13:05:59 +03:00
siivonek 35886671df [isp] Fix mistake in isp cabac write. Intra luma mpm flag bit was checking isp when it did not need to. 2022-08-23 13:20:57 +03:00
siivonek 1125248046 [isp] Fix mistake in cost calculation. Remove some commented out code blocks. 2022-08-22 14:39:36 +03:00
siivonek 25e0c48633 [isp] Modify encode transform coeff func to handle non-square blocks, use cu_loc_t where possible. Fix mistake in mts idct generic. 2022-08-21 12:46:07 +03:00
siivonek 4c5ca63632 [isp] Resolve TODOs. Make scan order tables const. 2022-08-19 15:05:22 +03:00
siivonek 4ae6b780b3 [isp] Write isp config bit to sps. 2022-08-18 15:55:23 +03:00
siivonek f0ff249035 [isp] Remove old_scan tables and related asserts. Fix coefficient group indexing. 2022-08-18 15:40:26 +03:00
siivonek fae833c4d2 [isp] Fix assert. Implement coef cost calculation for isp splits. 2022-08-18 15:22:17 +03:00
siivonek 943ae415db [isp] Add non-square block handling to functions. 2022-08-18 15:07:22 +03:00
siivonek eb5e24c15d [isp] Count isp cbfs. 2022-08-18 14:03:53 +03:00
siivonek d9b03ab399 [isp] Fix isp search. 2022-08-18 13:33:29 +03:00
siivonek 6a9ca94d25 [isp] Implement coefficient encoding for isp splits. Make get_split_dim non static, it was needed elsewhere after all. 2022-08-17 16:42:22 +03:00