256 commits

Author	SHA1	Message	Date
siivonek	bd2e588a22	[isp] Add lfnst asserts. Fix error in MTS search. Fix chroma lfnst index when no coefficients present.	2022-09-26 14:44:51 +03:00
siivonek	6975febccf	[isp] Fix lfnst constraint checks when ISP is in use. Add some asserts.	2022-09-23 15:41:50 +03:00
siivonek	785ae95302	[isp] Use TR_MAX_WIDTH in ISP checks instead of parameter.	2022-09-22 14:28:40 +03:00
siivonek	80945c985a	[isp] Fix cabac issues. There are always four transform blocks even if there are only two ISP splits. Fix prediction issues. PDPC filter was applied when it should be disabled. Fix reference building issues. Left reference was built incorrectly for blocks with height 2.	2022-09-20 15:16:14 +03:00
siivonek	e8779fc175	[isp] Fix storing cbfs for small ISP splits. Fix pdpc filtering. Cannot be used if width or height is less than 4. Fix dct related CI errors.	2022-09-16 15:46:31 +03:00
siivonek	124cdd4e4a	[isp] Fix luma cbf writing for ISP splits. Do not write luma cbf if first three splits had luma cbf 0.	2022-09-13 13:27:16 +03:00
siivonek	75d4de20e3	[isp] Fix CI errors.	2022-09-12 15:29:53 +03:00
Joose Sainio	68263f428b	[tr-skip] fix transform skip flag writing	2022-09-06 08:49:22 +03:00
Joose Sainio	17abc7a050	[lfnst] lfnst is not allowed for transform split	2022-09-05 12:04:21 +03:00
Joose Sainio	95b6ddbe67	[cabac] fix cbf_y context for tr splits	2022-09-05 11:10:55 +03:00
siivonek	b49a4d0a57	[isp] Fix chroma coeff writing for ISP.	2022-08-31 14:54:30 +03:00
siivonek	def855a027	[isp] Fix error in last sig coeff function call. Height was not used. Fix cbf writing. Fix transform skip flag writing.	2022-08-31 12:28:57 +03:00
siivonek	3c5479f99b	[isp] Convert functions to handle new coeff array order. Add function for getting coeff array subset. Fix assert.	2022-08-26 16:39:57 +03:00
siivonek	efc0f13501	[isp] Redo call hierarchy to include x, y coordinates.	2022-08-24 17:12:36 +03:00
siivonek	4ffbbee606	[isp] Fix chroma width error when writing cu loc. Remove redundant IPS mode checks.	2022-08-24 15:40:14 +03:00
siivonek	bc3bf8a08c	[isp] Set cbfs for isp splits after search. Add helper function for isp split number.	2022-08-24 13:05:59 +03:00
siivonek	35886671df	[isp] Fix mistake in isp cabac write. Intra luma mpm flag bit was checking isp when it did not need to.	2022-08-23 13:20:57 +03:00
siivonek	1125248046	[isp] Fix mistake in cost calculation. Remove some commented out code blocks.	2022-08-22 14:39:36 +03:00
siivonek	25e0c48633	[isp] Modify encode transform coeff func to handle non-square blocks, use cu_loc_t where possible. Fix mistake in mts idct generic.	2022-08-21 12:46:07 +03:00
siivonek	4c5ca63632	[isp] Resolve TODOs. Make scan order tables const.	2022-08-19 15:05:22 +03:00
siivonek	f0ff249035	[isp] Remove old_scan tables and related asserts. Fix coefficient group indexing.	2022-08-18 15:40:26 +03:00
siivonek	943ae415db	[isp] Add non-square block handling to functions.	2022-08-18 15:07:22 +03:00
siivonek	6a9ca94d25	[isp] Implement coefficient encoding for isp splits. Make get_split_dim non static, it was needed elsewhere after all.	2022-08-17 16:42:22 +03:00
siivonek	27a735ab5a	[isp] Move can_use_lfnst_with_isp to intra.c. Remove duplicate functions. Move isp related functions from search to intra. Make isp_split_dim static. Move isp related defines from search to intra.	2022-08-17 15:33:03 +03:00
siivonek	4a57dc8cc7	[isp] Add new convert_to_log2 table. Change all instances which used old convert_to_bit table to change dimensions into log2.	2022-08-16 15:00:15 +03:00
siivonek	83a216d5e6	[isp] Uncomment old scan order code to test against new one. Add assert to ensure old and new tables are the same.	2022-08-11 12:36:49 +03:00
siivonek	1fe45f719e	[isp] Modify existing scan table calls to use new getter. Add safety assert to getter.	2022-08-10 19:35:15 +03:00
siivonek	35f1ecbe9d	[isp] Add scan order getter. Add bookmark comments to scan order buffer.	2022-08-09 14:02:30 +03:00
Joose Sainio	ea32ef33ac	[lfnst] handle transform skip correctly during search	2022-07-08 10:57:26 +03:00
Joose Sainio	03fb6ce92e	[lfnst] Fix lfnst+tr_skip for dual tree	2022-07-08 10:57:26 +03:00
Joose Sainio	f9212b4e44	[mts] Don't do tr-skip when tr-skip is disabled	2022-07-06 15:15:28 +03:00
Joose Sainio	dc7c8eeb41	[tr-skip] fix uvg_encode_ts_residual	2022-07-06 10:51:01 +03:00
Joose Sainio	b35a75b2eb	[lfnst] Fix lfnst with rdoq	2022-06-29 16:25:25 +03:00
Joose Sainio	6ef532775b	[intra] Fix various issues with cclm, mip, dual-tree, and lfnst	2022-06-29 15:09:34 +03:00
Joose Sainio	68243e284f	[cleanup] fix warnings	2022-06-28 16:02:22 +03:00
Joose Sainio	e25ea52f6f	[lfnst] Fix mistakes	2022-06-28 15:32:33 +03:00
Joose Sainio	37590add20	[lfnst] [dual-tree] LFNST should work with dual tree	2022-06-28 15:32:31 +03:00
Joose Sainio	6c7dc9004c	[dual-tree] Fix split context state updating	2022-06-28 15:32:31 +03:00
Joose Sainio	b0d616b03c	[dual-tree][tests] Fix some issues and enable cabac state test to test for dual tree	2022-06-28 15:32:30 +03:00
Joose Sainio	345c50ecee	[dual-tree] rename kvz_ to uvg_	2022-06-28 15:32:30 +03:00
Joose Sainio	cf144e2724	[dual-tree] Works for all depths with basic tools	2022-06-28 15:32:29 +03:00
Joose Sainio	1c313e9c19	[dual-tree] works for depths 1 and 2	2022-06-28 15:32:29 +03:00
Joose Sainio	abd00d04a1	[dual-tree] Still not working but bitstream valid	2022-06-28 15:32:28 +03:00
Joose Sainio	0adb0846d2	[dual-tree] Bitstream valid, hash missmatches	2022-06-28 15:32:28 +03:00
Joose Sainio	be2ef18fea	[dual-tree] Not working	2022-06-28 15:32:27 +03:00
Joose Sainio	8fba042e02	[dual-tree]preliminary preparation for dual tree	2022-06-28 15:32:27 +03:00
Joose Sainio	20010cf759	[lfnst] Fix hash mismatches for depth 4 chroma	2022-06-28 15:32:10 +03:00
Joose Sainio	a6d79407ab	[lfnst] various small fixes	2022-06-28 15:30:28 +03:00
Joose Sainio	36137e2bee	[lfnst] WIP	2022-06-28 15:29:50 +03:00
Joose Sainio	55a0e35892	[lfnst] Count lfsnt bits	2022-06-28 15:29:41 +03:00