I would sit back and watch your fish for a while. I'm sure you can work out what is spooking them as fish rely heavily on body and behavioural communication.
Could be as simple as vibration on wooden floor when someone walks past.
Could be as simple as vibration on wooden floor when someone walks past.