This is a bit embarrassing - but I have found out why this was.
Once everything was working without cloudflare, I added the nginx pagespeed module to my nginx config (pagespeed on;)
This was the reason for this error 
I found this out browsing through the console within DevTools.
I apologise for wasting your time.
Just to add: I had also asked pagespeed to honourCSP, but this creates the same problems. At this moment, I have removed pagsepeed and everything works great.