That looks like expected behavior… The color slot takes any value that you can use as background-color in CSS. So when you give a hex value, it will stay the same color in different modes and themes.
If you actually don’t want a background color, you should state none or transparent.
If you want a background color that changes with modes, you’d need to state it with a CSS custom property: you could use one of Discourse’s pre-defined ones like var(--secondary-high). Or define some yourself, as explained in