You absolutely want swap. It is super cheap insurance. Even if you never use it, it’s easily worth the minor cost in disk space.
Unless, of course, you enjoy processes being randomly terminated when you go exactly one byte over the total system memory at any given time…