If you add swap, you’ll avoid the OOMs. You don’t want OOMs, and you don’t want a lot of paging activity, otherwise the memory usage bump might or might not be important. (It might be interesting, but that’s a different thing.)
I’d recommend running
vmstat 5 5
or similar to see what the paging activity looks like.
Also,
free -h
is more useful than
free -mh
because it does matter how much swap you have left.