4620 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
siivonek	3c861e4c02	[isp] Fix search. Best LFNST and MTS modes were not selected correctly for ISP modes.	2023-08-14 12:21:27 +03:00
siivonek	b4cc321349	[isp] Fix transform selection when MTS & ISP is used. Wrong transform was selected. Change mts parameter name to better reflect its purpose.	2023-08-14 12:21:26 +03:00
siivonek	85f6b00394	[isp] Add lfnst asserts. Fix error in MTS search. Fix chroma lfnst index when no coefficients present.	2023-08-14 12:21:26 +03:00
siivonek	b9822398a0	[isp] Fix lfnst constraint checks when ISP is in use. Add some asserts.	2023-08-14 12:21:25 +03:00
siivonek	701257cdd2	[isp] Remove unnecessary code from forward dct 32.	2023-08-14 12:21:25 +03:00
siivonek	eb6771321f	[isp] Disable fast residual cost calculation from ISP related CI tests.	2023-08-14 12:21:24 +03:00
siivonek	89db34d4e0	[isp] Use TR_MAX_WIDTH in ISP checks instead of parameter.	2023-08-14 12:21:24 +03:00
siivonek	c4bc2d6b10	[isp] Limit ISP search to block size 32. Size 64 is not allowed.	2023-08-14 12:21:23 +03:00
siivonek	8ae285322e	[isp] Add CI tests.	2023-08-14 12:21:23 +03:00
siivonek	5713fbff1a	[isp] Add ISP checks to search. LFNST can be used with ISP for larger blocks. Transform skip cannot be used with ISP.	2023-08-14 12:21:22 +03:00
siivonek	7282534879	[isp] Fix CI errors.	2023-08-14 12:21:22 +03:00
siivonek	01c4d1ddb0	[isp] Fix cabac issues. There are always four transform blocks even if there are only two ISP splits. Fix prediction issues. PDPC filter was applied when it should be disabled. Fix reference building issues. Left reference was built incorrectly for blocks with height 2.	2023-08-14 12:21:21 +03:00
siivonek	b8e36bbc4a	[isp] Fix storing cbfs for small ISP splits. Fix pdpc filtering. Cannot be used if width or height is less than 4. Fix dct related CI errors.	2023-08-14 12:21:20 +03:00
siivonek	99495c331b	[isp] Fix some asserts to allow log2_dim 1 block sizes. Fix coefficient group scan order for small dimensions.	2023-08-14 12:21:19 +03:00
siivonek	d39fddf0d8	[isp] Implement DCT for small blocks.	2023-08-14 12:21:19 +03:00
siivonek	910501012f	[isp] Fix referene building for depth 2 blocks. Flip horizontal mode dimensions during prediction. Fix reference length during prediction when ISP enabled.	2023-08-14 12:21:18 +03:00
siivonek	7ba557af6b	[isp] Fix luma cbf writing for ISP splits. Do not write luma cbf if first three splits had luma cbf 0.	2023-08-14 12:21:17 +03:00
siivonek	a28e61eff7	[isp] Fix CI errors.	2023-08-14 12:21:17 +03:00
siivonek	4794104ecc	[isp] Fix errors in reference building. Use cubic filter during prediction if ISP enabled.	2023-08-14 12:21:16 +03:00
Joose Sainio	662f31d61d	[isp] Use correct coordinates for depth 4 chroma tu cost calculation	2023-08-14 12:17:36 +03:00
Joose Sainio	08942a5394	[tr-skip] fix transform skip flag writing	2023-08-14 12:17:35 +03:00
siivonek	a261d4c5b3	[isp] WIP	2023-08-14 12:17:34 +03:00
siivonek	6340dfe4ce	[isp] Fix mistake in pu_loc argument passing, was not used after passing.	2023-08-14 12:17:33 +03:00
Joose Sainio	88c33c0489	[lfnst] Fix lfnst constraint checking for the new coeff order	2023-08-14 12:17:33 +03:00
Joose Sainio	e0e96068cc	[lfnst] lfnst is not allowed for transform split	2023-08-14 12:17:32 +03:00
Joose Sainio	cb7f9919e3	[jccr] Fix jccr coefficient copying	2023-08-14 12:17:32 +03:00
Joose Sainio	3e23fd0601	[cabac] fix cbf_y context for tr splits	2023-08-14 12:17:31 +03:00
siivonek	59292d8808	[isp] Add extra logic to reference building to accommodate ISP. Remove some asserts which were invalidated by ISP.	2023-08-14 12:17:31 +03:00
siivonek	33cd44f11b	[isp] Fix chroma coeff writing for ISP.	2023-08-14 12:17:30 +03:00
siivonek	d8d206365c	[isp] Fix jccr coeffs.	2023-08-14 12:17:29 +03:00
siivonek	7398e58431	[isp] Fix coeff cost calculation. Coeff arrays were indexed wrongly.	2023-08-14 12:17:29 +03:00
siivonek	d050efcb87	[isp] Fix error in last sig coeff function call. Height was not used. Fix cbf writing. Fix transform skip flag writing.	2023-08-14 12:17:28 +03:00
siivonek	33ae02aae0	[isp] Fix mistake in isp cbf writing. Loop index was increased twice.	2023-08-14 12:17:28 +03:00
siivonek	4a21039e23	[isp] Fix mistake in function declaration.	2023-08-14 12:17:27 +03:00
siivonek	b8506c757c	[isp] Convert functions to handle new coeff array order. Add function for getting coeff array subset. Fix assert.	2023-08-14 12:17:26 +03:00
siivonek	69dcb04c99	[isp] Use temporary coeff array when quantizing coeffs. After deriving coeffs, copy temp coeffs from linear order to correct arrays with cu order.	2023-08-14 12:17:26 +03:00
siivonek	0ae71feae4	[isp] Fix assert.	2023-08-14 12:17:25 +03:00
siivonek	2e8f008de4	[isp] Redo call hierarchy to include x, y coordinates.	2023-08-14 12:17:24 +03:00
siivonek	10f9b2be26	[isp] Keep lfnst constraint up to date during search.	2023-08-14 12:16:42 +03:00
siivonek	39f30563c5	[isp] Fix chroma width error when writing cu loc. Remove redundant IPS mode checks.	2023-08-14 12:16:42 +03:00
siivonek	b53308f258	[isp] Fix mistake in setting cbfs. Skip stting if ISP is not used.	2023-08-14 12:16:41 +03:00
siivonek	56ebea7358	[isp] Set cbfs for isp splits after search. Add helper function for isp split number.	2023-08-14 12:16:40 +03:00
siivonek	510798cb3d	[isp] Fix mistake in isp cabac write. Intra luma mpm flag bit was checking isp when it did not need to.	2023-08-14 12:16:40 +03:00
siivonek	f86dc29ce7	[isp] Fix mistake in cost calculation. Remove some commented out code blocks.	2023-08-14 12:16:39 +03:00
siivonek	bbb8faea98	[isp] Modify encode transform coeff func to handle non-square blocks, use cu_loc_t where possible. Fix mistake in mts idct generic.	2023-08-14 12:16:38 +03:00
siivonek	7062697beb	[isp] Resolve TODOs. Make scan order tables const.	2023-08-14 12:16:37 +03:00
siivonek	93317cafa4	[isp] Write isp config bit to sps.	2023-08-14 12:16:37 +03:00
siivonek	182d0f4e66	[isp] Remove old_scan tables and related asserts. Fix coefficient group indexing.	2023-08-14 12:16:36 +03:00
siivonek	f8641f7436	[isp] Fix assert. Implement coef cost calculation for isp splits.	2023-08-14 12:16:35 +03:00
siivonek	ae0336fdfc	[isp] Add non-square block handling to functions.	2023-08-14 12:16:34 +03:00