If we remove a master device and specify which other master devices
attached slaves should be returned to, enforce that those two are
indeeed a pointer and a keyboard.
Otherwise we can try to attach the keyboards to pointers and vice versa,
leading to possible crashes later.