All fish have Growth Inhiiting Hormone but it only affects conspecific species, not all fish species. So your largest goldfish would produce the most amount of GIH and stunt other goldfish like species. You should be worry about having too many ATF together.
Dont worry, even if you only kept ATF, if your water changing the tank regularly then the GIH is diluted enough anyway.