4611 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
siivonek	5713fbff1a	[isp] Add ISP checks to search. LFNST can be used with ISP for larger blocks. Transform skip cannot be used with ISP.	2023-08-14 12:21:22 +03:00
siivonek	7282534879	[isp] Fix CI errors.	2023-08-14 12:21:22 +03:00
siivonek	01c4d1ddb0	[isp] Fix cabac issues. There are always four transform blocks even if there are only two ISP splits. Fix prediction issues. PDPC filter was applied when it should be disabled. Fix reference building issues. Left reference was built incorrectly for blocks with height 2.	2023-08-14 12:21:21 +03:00
siivonek	b8e36bbc4a	[isp] Fix storing cbfs for small ISP splits. Fix pdpc filtering. Cannot be used if width or height is less than 4. Fix dct related CI errors.	2023-08-14 12:21:20 +03:00
siivonek	99495c331b	[isp] Fix some asserts to allow log2_dim 1 block sizes. Fix coefficient group scan order for small dimensions.	2023-08-14 12:21:19 +03:00
siivonek	d39fddf0d8	[isp] Implement DCT for small blocks.	2023-08-14 12:21:19 +03:00
siivonek	910501012f	[isp] Fix referene building for depth 2 blocks. Flip horizontal mode dimensions during prediction. Fix reference length during prediction when ISP enabled.	2023-08-14 12:21:18 +03:00
siivonek	7ba557af6b	[isp] Fix luma cbf writing for ISP splits. Do not write luma cbf if first three splits had luma cbf 0.	2023-08-14 12:21:17 +03:00
siivonek	a28e61eff7	[isp] Fix CI errors.	2023-08-14 12:21:17 +03:00
siivonek	4794104ecc	[isp] Fix errors in reference building. Use cubic filter during prediction if ISP enabled.	2023-08-14 12:21:16 +03:00
Joose Sainio	662f31d61d	[isp] Use correct coordinates for depth 4 chroma tu cost calculation	2023-08-14 12:17:36 +03:00
Joose Sainio	08942a5394	[tr-skip] fix transform skip flag writing	2023-08-14 12:17:35 +03:00
siivonek	a261d4c5b3	[isp] WIP	2023-08-14 12:17:34 +03:00
siivonek	6340dfe4ce	[isp] Fix mistake in pu_loc argument passing, was not used after passing.	2023-08-14 12:17:33 +03:00
Joose Sainio	88c33c0489	[lfnst] Fix lfnst constraint checking for the new coeff order	2023-08-14 12:17:33 +03:00
Joose Sainio	e0e96068cc	[lfnst] lfnst is not allowed for transform split	2023-08-14 12:17:32 +03:00
Joose Sainio	cb7f9919e3	[jccr] Fix jccr coefficient copying	2023-08-14 12:17:32 +03:00
Joose Sainio	3e23fd0601	[cabac] fix cbf_y context for tr splits	2023-08-14 12:17:31 +03:00
siivonek	59292d8808	[isp] Add extra logic to reference building to accommodate ISP. Remove some asserts which were invalidated by ISP.	2023-08-14 12:17:31 +03:00
siivonek	33cd44f11b	[isp] Fix chroma coeff writing for ISP.	2023-08-14 12:17:30 +03:00
siivonek	d8d206365c	[isp] Fix jccr coeffs.	2023-08-14 12:17:29 +03:00
siivonek	7398e58431	[isp] Fix coeff cost calculation. Coeff arrays were indexed wrongly.	2023-08-14 12:17:29 +03:00
siivonek	d050efcb87	[isp] Fix error in last sig coeff function call. Height was not used. Fix cbf writing. Fix transform skip flag writing.	2023-08-14 12:17:28 +03:00
siivonek	33ae02aae0	[isp] Fix mistake in isp cbf writing. Loop index was increased twice.	2023-08-14 12:17:28 +03:00
siivonek	4a21039e23	[isp] Fix mistake in function declaration.	2023-08-14 12:17:27 +03:00
siivonek	b8506c757c	[isp] Convert functions to handle new coeff array order. Add function for getting coeff array subset. Fix assert.	2023-08-14 12:17:26 +03:00
siivonek	69dcb04c99	[isp] Use temporary coeff array when quantizing coeffs. After deriving coeffs, copy temp coeffs from linear order to correct arrays with cu order.	2023-08-14 12:17:26 +03:00
siivonek	0ae71feae4	[isp] Fix assert.	2023-08-14 12:17:25 +03:00
siivonek	2e8f008de4	[isp] Redo call hierarchy to include x, y coordinates.	2023-08-14 12:17:24 +03:00
siivonek	10f9b2be26	[isp] Keep lfnst constraint up to date during search.	2023-08-14 12:16:42 +03:00
siivonek	39f30563c5	[isp] Fix chroma width error when writing cu loc. Remove redundant IPS mode checks.	2023-08-14 12:16:42 +03:00
siivonek	b53308f258	[isp] Fix mistake in setting cbfs. Skip stting if ISP is not used.	2023-08-14 12:16:41 +03:00
siivonek	56ebea7358	[isp] Set cbfs for isp splits after search. Add helper function for isp split number.	2023-08-14 12:16:40 +03:00
siivonek	510798cb3d	[isp] Fix mistake in isp cabac write. Intra luma mpm flag bit was checking isp when it did not need to.	2023-08-14 12:16:40 +03:00
siivonek	f86dc29ce7	[isp] Fix mistake in cost calculation. Remove some commented out code blocks.	2023-08-14 12:16:39 +03:00
siivonek	bbb8faea98	[isp] Modify encode transform coeff func to handle non-square blocks, use cu_loc_t where possible. Fix mistake in mts idct generic.	2023-08-14 12:16:38 +03:00
siivonek	7062697beb	[isp] Resolve TODOs. Make scan order tables const.	2023-08-14 12:16:37 +03:00
siivonek	93317cafa4	[isp] Write isp config bit to sps.	2023-08-14 12:16:37 +03:00
siivonek	182d0f4e66	[isp] Remove old_scan tables and related asserts. Fix coefficient group indexing.	2023-08-14 12:16:36 +03:00
siivonek	f8641f7436	[isp] Fix assert. Implement coef cost calculation for isp splits.	2023-08-14 12:16:35 +03:00
siivonek	ae0336fdfc	[isp] Add non-square block handling to functions.	2023-08-14 12:16:34 +03:00
siivonek	031a758d6c	[isp] Count isp cbfs.	2023-08-14 12:16:01 +03:00
siivonek	75175ee2e2	[isp] Fix isp search.	2023-08-14 12:16:00 +03:00
siivonek	8d914ce849	[isp] Implement coefficient encoding for isp splits. Make get_split_dim non static, it was needed elsewhere after all.	2023-08-14 12:15:59 +03:00
siivonek	573ecf80e3	[isp] Move can_use_lfnst_with_isp to intra.c. Remove duplicate functions. Move isp related functions from search to intra. Make isp_split_dim static. Move isp related defines from search to intra.	2023-08-14 12:15:58 +03:00
siivonek	bcbd952dfd	[isp] Add height handling to avx2 reconstruction.	2023-08-14 12:15:57 +03:00
siivonek	7c340fd92b	[isp] Add height to inverse transform skip.	2023-08-14 12:15:56 +03:00
siivonek	318d925028	[isp] Add new convert_to_log2 table. Change all instances which used old convert_to_bit table to change dimensions into log2.	2023-08-14 12:15:55 +03:00
siivonek	6922157ed3	[isp] Fix quantization function calls. Some were not getting height as input.	2023-08-14 12:15:54 +03:00
siivonek	50ad91a94e	[isp] Modify quantization functions to work with non-square blocks.	2023-08-14 12:15:53 +03:00