Ideally you shouldn’t be using any “panel” on the server you use to host discourse.
Your VPS provider is likely over provisioning their resources and that is the likely reason for the slow load, another contributor could be slow or congested network at your provider’s end or ISP’s end.
Subsequent loads are fast because the resources have already been downloaded, the app only needs small payloads to refresh the latest data. This is the beauty of discourse.
Some mitigations may include the following:
- Try a different VPS provider
- Consider adding CDN Acceleration to discourse