I may take a good amount, out to facilitate easier cleaning. Just leave a small layer.Just a heads up.
I used pea gravel many years ago, and found the large interstitial spaces allowed detritus to slip between and underneath the gravel, creating anoxic pockets of grunge. These can be hard to vacuum out, and can produce hydrogen sulfide (which can make the tank smell like rotten eggs when they release), and can contribute to bacterial blooms even after the tank is cycled.
I prefer PFS that does trap fish feces, old food and other detritus, and is easier to vacuum off its surface when water changes are done.