Worse Than Latin
Yet if you see the methological section of Caplan's essay, he doesn't really defend it, as he says that "The empirical evidence on their contribution is decidedly negative" and also says that it makes it more difficult to understand and it is a "waste of time". His disagreement with Austrians on this subject is that this is the established "language" of modern economics, and though there are no rational reasons for this, we should just accept it, similar to how people within the Catholic church must accept the use of Latin, which also has no rational basis. In other words, "yes, it's stupid,just like the use of Latin within the Catholic church, but we should accept it anyway because that's just the way it is".
Caplan is right in the sense that if you want an academic economic position right now, you need to learn mathematical economic modeling, no matter how much you dislike it because it is mandatory for anyone pursuing such a career. However, that doesn't mean we should stop arguing against it, just like the fact that just because it is mandatory for anyone working outside of the "underground economy" to pay high taxes doesn't mean that we shouldn't argue against high taxes.
Furthermore, Caplan's likening of mathematical economics with latin in the Catholic church understates how bad it is. As far as I know, the use of the otherwise extinct Latin language instead of the native languages of the different Catholic countries doesn't really mean that its beliefs are altered. In this case, the problem is indeed only that it is a waste of time to learn it. That's bad, but perhaps not that important.
What's worse is that the mathematical modeling means that the theories are adapted to fit mathematical equations, instead of reality. That is why for example entrepreneurship is disregarded in economic models, despite being an important part of the real life economy, since it is incompatible with the equilibrium required by some of the mathematical functions use. The problem is thus not just that it's a waste of time, it also makes economic theories more unrealistic.