You got it. You authenticate with your biometrics. The server sends you a challenge. Your device decrypts that challenge locally and sends the server the response back. As for the mathematics behind it all, I am not the person to ask. I just know that it does work, and that quantum computers could be a serious problem in the future. But for now, they're not.

Also no expert about the math behind it but this is how I learned asymmetric encryption with public and private key on a very high level of abstraction:

To encrypt a message you basically take the unencrypted message to the power of the private key (lets call that d) and divide it by the result of a multiplication of two large prime numbers (lets call that N). Now you take the rest of that division, so basically modulo if you are familiar with that.

To decrypt, you take the encrypted message to the power of the public key (e) and divide it by N. The rest of that operation is again the unencrypted message.

So the private and public key are the pairs (d, N) and (e, N) respectively. The great thing about this is that you can post the public key publicly without care and only you, in possession of the private key, can decrypt messages encrypted with your public key. So its easy to verify if you got the key but hard to brute force due to modulo being a one way function. However, this basic RSA approach as described here is not used on its own anymore but rather in combination with other techniques to prevent guessing on very short messages or keys.