Ok, so your server thinks that it is not running on the hostname that the browser is requesting. The security key / passkey generation procedure has to ensure that the the hostname of the browser matches that of the server (keys are generated per hostname).
Can you log into your Rails console and check what the output of Discourse.current_hostname is? If it doesn’t match the URL you use to access the site, that’s the problem.