Water changes remove built up waste products such as nitrates that can stress the fish. They also remove biologicals such as excess bacteria and even hormones produced by fish. Some fish such as bluegills actually produce a growth suppressant hormone that accelerates maturity in crowded conditions, this is why so many ponds with large breeding populations of them have very few large individuals. Anything that acts as a long term stressor, even low grade, tends to slow groth and lessen color.