That is true. I use mostly pool filter sand with a fairly uniform grain size and it takes a lot more to blow it around than the fine stuff.
I don't think I have ever had to vacuum my 200g display tank. Not even once. About the only thing I do is remove the small branches that fall off my wood once in a while.
I've used pool filter sand as well and that I have as much of a trouble with it. I ended up switching to moonlight Sand by caribsea. I literally use a air line hose to vacuum up the detritus sitting on the surface of the sand. My Royal pleco likes to chomp away at the bogwood which leaves little specks all over the place as well as fish waste. Just takes a few extra minutes while doing my water change to clean it up, to me it's worth it.