genetics can play a big part, as can dominance. generally the smallest fish in a school are the lowest on the totem pole. the others may be eating all the food before the smaller guy can get his fill, or harassing it to the point it's barely getting enough to survive let alone metabolize and grow.