It can and has been done. It requires a few layers of other substrate above it to help keep water clear. I had a planted tank with soil/peat/sand/then rock on the top layer, and the plants loved it.
I have used dirt, mud, and even silt, it just takes a long time to settle, dirt especially has a lot of floating debris.
Getting it from the park can get you arrested.