Males or females can both be confrontational but it is usually the male that causes problems, mainly the fight other males, and sometimes females of their ownspecies. Also fish of the same general appearence. Put the mollies in first and givew them a day or two to settle in, if the smaller molly is at least 1/2 the size of the betta and already established in the tank the betta should not mess with it. It helps if you have a few clumps of tall plants like elodea to break visual lines and allow definition of territories (even in a small tank).