Not all possible combos are listed in the dropdown for modifier keys. Editing the XML seems to work partially but not 100%...
From Discord user: Accordingly (accordingly9482)

I might be missing something, but is it not possible to do a 3-key Layer Modifier combo such as RWin+Alt+Ctrl? It is not in the drop down, and trying to enter the 3rd key in the None box doesn't seem to stick. I am on 2.21 Beta 59 (Portable). I would also be fine with manually editing the XMBCSettings.xml  if I can get some guidance on what to put (and if the changes will stick, not get overwritten by the software).

I figured out (courtesy ChatGPT) that <LayerKey4 Key="0" Modifier="22" /> does work for enabling the intended layer via the desired hotkeys, but it causes some weird side-effects where that layer stays activated constantly, and I have difficulty accessing and interacting with other windows. My guess is one or more keys is not being released properly (because XMBC isn't looking for all of the keys in the combo I'm trying to use, or something to that effect, since I'm "forcing" it via XML).

I may have solved things on my end for the time being by switching around some of the modifiers I'm using such that maybe I don't need the above combo after all (though I need to do some more testing to be sure I'm not going to run into some conflicts with other software, etc).

So far my alternate modifiers are working well, and have some other benefits too.