You cannot currently remove Layer Groups from View Filter Manager - if you remove one Layer Group it treats it as all Layers (and layer groups) - I have a request in to be able to manage Layer Group in View Filters - while the User Interface looks like this is possible, that only manages the major Group items like Flags, Photogrammetry, CAD, Georeferenced Images, and Layers (as a whole). The Layer Groups was added and they never extended this control capability to Layer Groups.
Alan