I don't know the details. But basically, a Tor relay provides some token to the onion server. The onion server signs that token with its private key, and returns the output to the Tor relay. Then the Tor relay verifies that the token was signed by the private key corresponding to the onion hostname.
