Agreed
for these reasons
1. If its sand it will be full of dust that will cloud your water when ever it gets disturbed
2. If its gravel thats "fake" like colored gravel it may have some paint dust I have found.
3. as already stated you never know what could be on or in the substrate and you run the risk of harming or even killing your fish.