As far as I know phosphates aren't bad for the fish (maybe in extreme cases), but they do promote the grown of some algaes, which may in turn be bad for the fish.
if your tank is planted HIGH phosphates can actually kill your plants. there has to be a balance of phosphate to nitrate in order for your plants to utilize it the best. you do want to keep your phosphate around 1ppm ideal for planted in most cases.