Insecure favicon with mixed content warnings

The favicon and other site icons get loaded over HTTP while the site is HTTPS. Chrome is able to upgrade these requests to HTTPS but the favicon stays HTTP and I get an error in console.

EDIT: Force HTTPS was disabled in settings, enabling solved the problem.