It's even worse than I thought. You can give *any phone number you like*. As long as you can receive SMS / answer calls at it.
So I logged in with my username and password, and then gave a phone number. It called me, I pressed #, and it authenticated me.
But next time I can give a different phone number.
What is the point of this?!