In my case, I would opt for that as we make sure that images added to the topics we Tweet on our account have large enough images. AFAIK Twitter automatically falls back to the small version, if you don’t provide a large enough image even when summary_large_image is provided.
Even better would be if images sizes are vetted by core and depending on the minimum sizes required use summary or summary_large_image (min 300x157) automatically.
If that is not possible at all, I would certainly be happy to at least have the option to use summary_large_image instead of summary.