Looks like it was just an overdue reboot that was causing this. Managed to reboot the server and the numbers look much much better as of now
root@discourse:~# free
total used free shared buff/cache available
Mem: 3927308 977624 2246208 42880 703476 2646836
Swap: 2097148 0 2097148