This is why you attract attention. This kind of flighty, weak aura sends the message that you are vulnerable. Men will either smell the opportunity to prey on you, or will feel compelled to protect you. Neither case is what you want. You need to walk at a healthy but relaxed pace, move like you're made of iron, and look every person you see square in the eyes without emotion. Remember that it's YOUR sidewalk and you're letting them share it with you, if they behave. Own it.
But I do understand now that he was a creep who wasn't letting up, and that is scary. My wife won't leave the house for the same reason. Our alley is a primary pathway for the convicts released from the prison and they often camp out in it.