feeder goldfish (called Comets) produce a lot of waste and ammonia. even 1 of these fish isn't appropriate for a 55 gallon tank due to their maximum size.
fish poop gets stuck in the gravel or sand on the bottom and breaks down slowly to become ammonia also. if you don't siphon the crap out it builds and builds up until the bacteria in the filter can't cope with the amount of ammonia being produced by the amount of crap in the gravel and in the filter's mechanical section / sponges.
i can't imagine never taking my trash to the curb.. and just keeping it all in my house, that is essentially what you have done with your tank by not siphoning the gravel or changing the water
