If you are already sure, why ask? You already say you have used it for fish. If you aren't going to consider the advice of others, why ask?
Like others said, polution is not the only danger in a lake. If there are fish, then there should be fish diseases native to the lake. Ancient people had to drink from lakes and streams. Many survived, but I'm sure there were some that died of sickness.
The only water I've drank raw was during a cave tour as a kid. The tour guide claimed we could and that it was the same water that was pumped up to the gift shop. There was some sort of cave crustacean(not a crayfish) that lived in the water. Probably not the smartest thing I've done, but I never got sick.