I'm fairly confident it's that sponge. Trust me, when you give it a few good wacks, the sludge will AMAZE YOU! It may "look" clean on the outside, but unless you use physical force (not just squishing it or swishing it) the stuff just doesn't come out.
Continue with your gravel vacuuming, and start really beating the sponges. Cutting them in half will also help.
As for Prime, I use it because I have many large fish tanks, and do many large water changes every week. If I used Amquel+ or NovAqua, I'd be forced to close down some of my tanks, simply because I couldn't afford dechlorinator! I buy a 2 liter bottle online and it treats 20,000 gallons. I am also a stickler for being efficient, and spending more money for the same job, and using more (watered down) product, just doesn't sit right with me, but to each his own.
Oh, and when I had gravel (I have all bare-bottom or planted tanks, so don't have to gravel vac anymore) I took the siphon, stuck it in the gravel til it ran relatively clean, picked up the siphon, and did the next 3 inches, and so on and so forth. 1/2 of the tank one week, and 1/2 the tank the next week.