Yeah, I agree that he'd eat the pygmys.
I'll have to take a video of my checkered madtom eating, madtoms have fairly large mouths, they're like a vaccuum, like any other catfish, they just don't get as big.
I've also kept pygmy sunfish, and I'll just say that they're not very bright...I don't even think they'd try to stay away from your madtom, they'd probably swim right into his mouth. Basically all pygmy sunfish do is look for something to eat and breed. Ours started breeding an hour after we put them in the tank. Actually the male started trying to spawn with the females about 5 mins after we put him in the tank, he just wasn't successful for awhile.
If you wanna keep pygmys I would set up a seperate tank for them. It doesn't even have to be that complicated, in fact you could keep them in a tub outside with plants if you wanted to.
We kept ours in a 10g filled with hornwort, with a small compact flourescent grow bulb, and an uncycled sponge filter. I did water changes periodically and no ammonia ever showed up. They'll breed constantly and lay their eggs in the plants, but they don't eat their eggs or fry so be careful if there are no other fish in there 'cause you might end up with a whole ton of them. We had mosquitofish in there with them, so the eggs got eaten whenever they were layed.
Make sure you keep a male with at least 2 females, 'cause like guppies he will harrass them and try to breed CONSTANTLY. Heavy planting also helps because the females won't be visible at all times for him to harrass, and besides, they like to lay their eggs on plants.