Changing your gravel out for something else is a huge change in any aquarium. I've done it in the past and never dreamt for one minute that it was a task that could be done without stressing the fish out, no matter what you leave in there for them to hide in.
I've always transferred the fish to a large covered tub. Nice and dark and without the constant noise of messing around with gravel, and the water would be proper manky too once you start, ughh, poor fish!
If the fish aren't in the tank you can really crack on and get it done quick. If you left them in there you'd be tippy toeing about so as not to stress them any further, which would just drag the job out.
It's a no brainer for me. Take them out, minimal stress, crack on with the job, put them back in. Job jobbed!