Besides adding and removing backends in Haproxy what have you done to test it?
Does your Haproxy config do random backends or affinity? Is the 404 logged by Haproxy? Is the 404 logged by nginx? Is it the first or second backend that errors? Does your site require login? Can you reproduce the errors when connecting to nginx directly from behind the proxy server? Do the files being requested exist on disk? Is it all JS or just specific ones?