All pool filter sand that I have available to me in my area contains silica.
And after some research on whether or not it will cause stress I found this:
"Silica sand and crushed or fragmented shell have a high soluble phosphate content. Further, silica sand will compact, creating anaerobic zones. Neither should be used in a marine aquarium"
So if that is true, then yes it will stress your fish. Phosphates can cause your pH to swing up, blocks calcification which is bad for invets that require shell growth, and is also a nutrient for cyanobacteria. It says it is soluble, which would mean to me that it would soak it up and then leach it back into the water once it reaches a certain saturation point.