U could keep the betta like that. He will eat the shrimp in my experience, the fancier the betta (those or red ones) the more aggressive the betta. What would be much better is a small air powered filter (ten bucks at wally world) and just once a week 20 percent water changes.
To many water changes could stress the betta, they can live in aquaria in excess of 7what years. My wife has knew in a five gallon with 3 plants, deco, and small air powered filter. They don't like too much current in there tank with those giant fins they have. My boy had one in a 3 gallon with air powered unddrgravel and one plant, but he gave that to my friends daughter. They are smart fish and u will probably find yourself getting him a bigger tank later on.