I think I have a fix for this now in FIX: Case when user has disabled auto-switching by pmusaraj · Pull Request #24 · discourse/discourse-color-scheme-toggle · GitHub, can you give that a try @jordan.vidrine.
To be clear, the PR addresses this use case:
When the user has disabled auto-switching, the component will now make a request for the site’s default dark color scheme. If there is one, it will load it, and then it will be able to toggle it.