Should I go Mag or Phys in SMTV?

Nocturne and SMTIV I played Mag, but IVA I played Phys and also had a lot of fun with that build. Have also heard of an ailment build which may be interesting, although gimpy compared to the other two

I haven't found a way to get ailments to consistently work yet. You can easily clean up random encounters in a turn or two, which makes setting up status conditions too slow, and most bosses resist them causing you to waste multiple attacks before it actually sticks.

I went with Phys personally, and it's a lot of fun.

Phys is a bit stronger in the end and requires less glory points for less efficiency miracles to make a optimal build. Phys is eventually more efficient at dealing damage to a single target while Mag is better at dealing damage to multiple targets throughout the game. Not counting uniques, more Phys skills are multihit against a single target, but more importantly Phys skills always have a chance of being critical on their own as long as the enemy doesn't resist them, where as Mag can't critical on their own and require you to hit their weakness, so you'll have to enable them to critical using another skill if the enemy doesn't have a weakness to exploit. Single target Almighty Phys skills are some of the most powerful, efficient skills in the game too. Another reason why a varied endgame Mag isn't as good as endgame Phys is that it requires too many skill slots if you want a versatile Mag build and the more elements you use the more you'll have to sacrifice damage output skills.