Also, if you don't clean out the "gunk" from your gravel or filters, even if you don't "overfeed" they will begin to increase. Check your nitrAtes as well, they are typically a sign of high nitrates.
This does NOT mean to clean the gunk using tap water! This will kill your beneficial bacteria. Swish the sponges/filter floss in used tank water, and give them some good WHACKs up against the side of the bucket a few times. Don't do ALL of your media at once, rotate them every water change. A good schedule is weekly water change of 50%, sometimes more if you are highly stocked or feeding heavily. A good way to judge is by testing your nitrAtes.