Fish and people are very different ... so I guess there's an extremely slight chance of a fish being affected by human diseases. The only thing that could go wrong is if you had a significant amount of chemical(s) on your hands when dealing with the food.
(I voted no...)
P.s. But in your other poll going I said you could take care of my fish