...I know you can wipe out beneficial bacteria by doing too many water changes...
You will never "wipe out" bacteria by changing water, because the bacteria don't live suspended in the water. They are on the surface of filter media, substrate, etc. and are unaffected by water changes.
I suppose it is theoretically possible to suppress the population of nitrifying bacteria by changing so much water...we're probably talking multiple 100% changes
per day...that you minimize the amount of ammonia that is present for them to utilize. This is academic; trust me, with the stocking density you have, a shortage of ammonia is never going to happen.
Don't worry about changing
too much water; rather, be aware that in this case it is virtually impossible to change
enough.
... I was going to use part of the filter media from the current set up...
Again, don't do half measures here. Use
every bit of the mature media from this tank when you move the fish to the new one, not just "some". The total bacterial population of all your current media might be enough to nitrify the amount of ammonia your stock produces, but it isn't any more than just barely enough. You just can't afford to lose any of it, so you shouldn't discard any of that media.