Its probably better to be safe than sorry based on the amount needed to kill someone, but i would imagine that would only happen from handling a puffer IF you had a minor cut on your hand somewhere and just so happen to have got the TTX on that area which led into your blood stream (in other words, quite unlikely but possible)...
Another good example is to look at the Japanese chefs that prepare fugu for their clients daily, none of them wear gloves and they literally cut into the whole fish and touch some of the organs with the highest concentration TTX and show no ill effects that can be seen...
Hopefully that puts your mind to ease a little