Something is stressing him, e.g., water parameters, tank mates, low DO, lights, etc.
They are not finicky eaters except for dry foods. It is not easy to get them to take pellets and take enough of them. Mine always went for anything fleshy - thawed whole fish and pieces, shrimp, sea foods, etc. Earthworms is a great diet. If not only he has never taken to them well, but now refuses them, something is very wrong, as stated above.
What was it fed before? If only live feeders, that would explain a lot but the time is running out to wean it over onto prepped foods.