This might be a little tricky because I think the hostname is stored inside the public key inside the security key table (it has been a while since I worked on this so I could be wrong). Will require a little finagling to raise this issue to the UI to disable the button and show the message. Also this would only show if all the registered security keys are the wrong hostname – if one matches the user is fine.
Kind of related, I also have to fix 2fa security key breaks when migrating to custom domain. I will assign this topic to me as well, because I think when we change hostnames we should probably just disable all the existing security keys because they become effectively useless.