You can ride MTBs in whatever feels comfortable, but eventually you'll probably agree that cycle specific clothing is best.
Clothing should meet certain criteria, then it's usually down to personal taste. Clothing must be fairly tight fitting without pinching, yet sufficiently "flexible" to allow freedom of movement. It must also absorb and wick away perspiration and should be free from bulky seams.
Riding off road requires slightly stronger materials as one has a tendency of hitting the ground more frequently; so do not skimp on your clothing.
On the subject of "clothing", mitts or gloves are almost essential to take some of the sting out of the trail bumps you'll be hitting all the time.