You can just leave it there since it isnt harming your fish or you can use a toothbrush dipped in Hydrogen Peroxide to scrub it off. H2O2 - dunno why its not called Hydrogen Dioxide - which is a slightly acidic bleaching oxidiser so it will kill mould but is quite mild so it wont harm your fish in small diluted quantities. I actually use it as a fish anti-fungal and to help hatch fish eggs. And to temporarily increase oxygen levels but I dont recommend anyone use it for these methods without some experience.
You could try just scrubbing it with salt first but the H2O2 will disinfect and lightly bleach which may be necessary if your silicon is discoloured. Chemist will have it (Pharmacy?), Pre-mixed 3% solution is perfect for your needs.