You'd have to define "dirt" first. By doing so, you'd have to indicate its chemical makeup. Some "dirts" are more organic than others. Sands and gravels tend to be inert, or at least inorganic.
In most cases, you would just introduce a lot of organics that would need to be broken down (burdening your BB). That said, a heavily planted tank may actually require "dirt" to provide a suitable growth base.
It's not like introducing "dirt" would be equal to introducing poison, but there would likely be the organic load issue to deal with..........but define "dirt!" ; )