The favicon and other site icons get loaded over HTTP while the site is HTTPS. Chrome is able to upgrade these requests to HTTPS but the favicon stays HTTP and I get an error in console.
EDIT: Force HTTPS was disabled in settings, enabling solved the problem.