How did them not eating affect their growth? As GTs are reknowned for being slow growers as it is. Also, after they started to breed did you notice a slow in growth? I only ask as i have a similar incedent with my GTs.
I bought 5x fish from different bloodlines in the hope to guarantee a pair, all were roughly 1.5-2" and straight away there were two males who fought for dominance. The best looking one won (which pleased me) and has grown really fast. He is touching 5" and this is after about two months or so (really fast growth for GTs, huh?). The others are now varied in size, the smallest being a female around 2.5", then another at about 3" also female, the sub-dominant male at around 3" and a 3.5" male which is either a poor specimen or has drab colours to represent a female - and therefore reduce aggression from the doninant male.
Now, the dominant male tried in vain to mate with the larger female several times. But she kept rejecting him. So he decided to go with the smaller female. And in no time at all i had free swimming fry, both Ma and Pa did a great job of caring for the fry - the male was really involved which shocked me. Until, the larger female decided she was now ready to breed - so she sabotaged the smaller females brood. Constantly attacking her and eating fry etc. Whilst the male watched. Once the littler female abandoned the area she moved in with the male and spawned. Then, they got into a similar situation just reversed. The smaller female, however, is MUCH more fiesty and reclaimed her teritory with ease.
Currently each female has their own side of the tank (the smaller females area being much larger however), and the male patrols both sides flirting with the females, both of which are intensely coloured and are ready to spawn. However, i dont wish to end up in a similar situation to you so i am going to remover the larger less aggressive female and the other two fish to a spare 55. See if a can create another pair and sell them on. I believe the dominant male and the larger female are from the same bloodline anyway so this is probably the best option for me.