Dream with Our Eyes Open

I can load sets of brushes in Photoshop and append them to the end of the current in-use set, but of course, that means the brushes window takes longer and longer to open.

But I don't want to reset the brushes window because I'm still using some of the brushes I loaded.

How do unload just one set at a time? Is it possible?

I've started working on Yuletide icons, so I have a pile of new border brushes that I'm trying. I want to keep the ones that have turned out to be useful (as well as the Easy Elements Snowflake brushes, which I love .... although I have to set the size way down to make them work on usericons).