I definitely prefer sand or fine grained gravel, anything 3mm or smaller. I found large grained gravel can be a real pain to plant things in and keep them planted, while with sand, planting anything no matter how big or small is really easy.
When I used plants with gravel I'd just gravel vac the areas that weren't planted and try not to disturb the plants. With sand or fine gravel I just wave the gravel vac over the surface to pick up whatever crud is sitting on it. Since sand and fine gravel compacts better than larger grained gravel, fish poop and debris don't sink through it much and sit on the top.
