How do you know this? What species is this observation based on? Do you have negative controls?
I've picked pairs out from a group and spawned them when they were young. Compared to the rest of the group that didn't get to spawn, there was absolutely no difference in growth later. I've seen this with miltiple species. Given proper care, the effect of spawning activity on growth is neglectable.
That's the key phrase there. If you provide the pair with a large tank and good clean water then growth may not be affected. Most people probably do not provide an optimal environment for this. I know Mel's pair of freds spawned very young but if you look at his later vids the pair look just as big as any other pair of freds out there. Large tank + clean water = optimal growth.