It's going to be alot of work. removing the shell, boiling or atleast pooring gallons of boiling water over/thru the gravel to get rid of pollutants found at the shore line (oils/fuels/pesticides), Digging deeper (1-3ft) into the shore will get you into the anerobic area with worse things in the gravel (find a part of the shore that is either protected or has faster moving waters).
Driftwood will need scrubbing and alot of hot water treatment (re-desolving the salts after they have been absorbed into the fibers is more difficult). If your fish are salt tolerant, then just getting the accumulated polutions out/off of the wood will do.
In either the gravel or driftwood look for an oily slick on the hot water.
A rule of thumb to consider...If your not willing to
TASTE it don't consider putting it into the fishes tank. That will usually make a person rethink if it is 'clean enough'
I'm not trying to scare you out of it, just follow thru with all the work, no short cuts.
If the gravel has nice color & shape and you deem it worthwhile go for it (oh, make sure it's legal, here in Florida it's illegal to remove quanities of beach sand from public lands for private use).
If you decide to do it, take pix of the cleaning process as you go thru it and let us see how it goes.
Good Luck,
Dr Joe
.