... what should I consider testing
As I have listed already:
ammonia
nitrite
nitrate
hardness
- all by a liquid test, not sticks.
Plus
pH
temp.
Again, describe the diet, please.
From what you describe, something is very wrong with your situation.
In such a small tank (for the size of the fish) your fish may easily be getting nitrite poisoning, aka methemoglobinemia or brown blood disease, blood not being able to carry oxygen anymore, because there isn't enough water for instant dilution of dangerous chemicals to a safe temporary level (which is below the level of detection by a reputable liquid home kit), because it takes time for the bacteria to process ammonia into nitrite, and nitrite into nitrate.
In other words, ideally, the water leaving your biofilter must have zero ppm ammonia and zero ppm nitrite, firmly, by a liquid test kit (API, Salifert, etc.), but with such a high bioload, your fish breath out so much ammonia especially after a feeding, that the biofilter fails to convert all ammonia quickly into nitrite, and then all the nitrite quickly enough into the nitrate, hence poisoning your fish temporarily (but killing permanently the blood cells carrying oxygen, while fish ain't got bone marrow to quickly replace the oxygen carrying hemoglobin in the bloodstream), especially right after the feedings, which may be why your RTCs upchuck their feed. Some fish can cope better with this and hold the feed down, other less so, it varies, but in any case, this situation in a fish tank is extremely dangerous and much, much sooner than later, they will all get sick and perish from it.
