It's not just potential physical damage to the fish, but if overused or abused it could also lead to stress-associated immunosuppression. Think of the use of mirrors as someone agitating a fish by constantly tapping on the glass. This can trigger a flight or flight response in a fish, and while it can assist in "pumping a fish up" causing increased coloration, etc, over time it can also potentially wear a fish down, weakening it's immune system. Long term release of substances such as cortisol etc can cause just as much harm to a fish, as they can to a human being.
The Role of Stress in Fish Disease
By R.W. Rottmann, R. Francis-Floyd1, and R. Durborow. Published by the Southern Regional Agricultural Center and the Texas Aquaculture Extension Service. Physiological stress and physical injury are the primary contributing factors of fish disease and mortality in aquaculture. Stress is defined as physical or chemical factors that cause bodily reactions that may contribute to disease and death.