I agree with the comments on the necessity of culling. But the only time it should be considered IMO is if you're considering either breeding the fish or selling it to someone else down the line (as in not intending to keep it for life). If you're just looking for a "wet pet," and whatever defect the fish has doesn't affect its quality of life, by all means keep it!
If you consider yourself a "serious" fish keeper, and are keeping species that are rare and facing potential extinction in their natural habitat sometime in the future (like lake Victorian africans or some SA/CA cichlids, some chinese species like hillstream loaches, or some marine species), IMO your goal should always be to keep the gene pool strong and breed them if you have the means to and get them out into the hobby. Some fish in the hobby are actually extinct in the wild... Red tailed sharks come to mind. In cases like this it is especially important to consider this stuff.
If you consider yourself a "serious" fish keeper, and are keeping species that are rare and facing potential extinction in their natural habitat sometime in the future (like lake Victorian africans or some SA/CA cichlids, some chinese species like hillstream loaches, or some marine species), IMO your goal should always be to keep the gene pool strong and breed them if you have the means to and get them out into the hobby. Some fish in the hobby are actually extinct in the wild... Red tailed sharks come to mind. In cases like this it is especially important to consider this stuff.