Run sar -W 1 in a console then go play with the site. If you come back and there’s any sustained values other than 0.00, you’re swapping and need to reassess your memory usage. Based on your description
I’m pretty sure you’re going to find that you’re swapping, and from there I’d bet good money that’s the cause of your performance problems.