That's likely your issue. I add tap directly as well. Don't worry about the fish being harmed immediately from the tap. I usually add 1/2 the dose of prime I'm using in the beginning of adding the water, then 1/2 the dose right at the end before I turn my filters back on.
Yeah I add tap treat the entire gallon size of the tank with prime after it's filled I rum my power heads for about 10 min to mix it all before turning my filter's on