It's not unheard of to take a few weeks to settle, albeit often as you say it is days, not weeks.
Large catfish can fast for several months without any problem or notable weight loss.
It's not a timid fish and it is smart and interactive with the keeper. It sounds like it behaves itself as expected too other than not feeding much. I don't see any reason for concern yet.
I'd not get it accustomed and spoiled on the worms. I suggest offering it what you want it to feed on and if uneaten, take out. Offer again in a few days. Repeat. If it doesn't accept the feed in 1-2 months, it'd be then warranted to troubleshoot the situation.
I am saying all this assuming that your water is not the source of the problem, that there is firmly zero ppm ammonia and nitrite by a liquid (not dip sticks) test kit, there is proper aeration with a good margin of safety, the temp, pH, hardness, TDS are not out of whack / range for this fish, etc. That is the basics all check out.
If you haven't checked and tested all this, I'd say you must.