All depends where you're at. I forget where but I was with my family and we decided to try and take some rocks or something (I know, not plants) and ended up getting in trouble 
You should be fine, though. Unless it's like a national park or something, it shouldn't cause a fuss.

You should be fine, though. Unless it's like a national park or something, it shouldn't cause a fuss.