It comes down to the end result to your tank and simple common sense. A small temperature drop of just a few degrees won't stress most fish, for some it can even be stimulating, but a large, sudden temperature drop can be stressful-- or worse-- for some species more than others. In other words you don't want to do a 90% water change with 40 degree water and suddenly bring an 80 degree tropical tank down to 50. An extreme example, maybe, but the point is, whatever the combination of water volume changed and temperature of new water, for most fish it should add up to a relatively mild temperature change.
In my case I generally add water a few degrees cooler than the tank and change warmer tanks first, then cooler tanks. In other words by the time my water heater is running low I'm on a cooler tank anyway.