Problems with Patreon Login, Force HTTPS, and S3 CDN (three) Issues

installation
1

I may need to split this into three separate posts, but they are related, so I’ll start with one.

A few days ago, I used this tutorial (How to Scale a Discourse Deployment with a Load Balancer and Managed Database Cluster | DigitalOcean) pretty much to the word and migrated my standalone Discourse Droplet on Digital Ocean to two Droplets inside a Load-Balancer, so far so good.

I then went through this tutorial (Configure an S3 compatible object storage provider for uploads) but after rebuilding discourse from the command line my Discourse site was only displaying a blank screen. I looked in Inspector in the browser to find that the browser was blocking all of my content because it was being served from HTTP and not HTTPS. This is probably because the load balancer is SSL Terminated, so everything external is HTTPS, but the servers themselves are running on HTTP.

At this point, I completely broke my servers again, trying to make them work with HTTPS inside the Load-Balancer, but it was simply not possible. I could not get the Digital Ocean Space/CDN to work with S3/CDN according to this tutorial (Configure an S3 compatible object storage provider for uploads). I went through it with a fine-toothed comb and inspected every aspect multiple times, but it would not work. The only way I could get Discourse to rebuild was to remove the DISCOURSE_S3_ENDPOINT: https://sfo3.digitaloceanspaces.com parameter from app.yml, but then even though it had built, I couldn’t get the server to respond. I got either a 503 server not responding error or a regular browser server not responding or server disconnected error. It differed based on the Load-Balancer and DO Space/CDN settings I was trying. I tried every possible combination of settings and nothing would enable me to serve a page.

When I left the DISCOURSE_S3_ENDPOINT parameter in place I got the following error during the Discourse rebuild, but this disappeared when I comment out the S3_ENDPOINT parameter.
Aws::S3::Errors::InvalidAccessKeyId: Aws::S3::Errors::InvalidAccessKeyId

All of my files were synced to S3, so I think it’s safe to assume the Access Key was fine, and the problem was caused by the S3_ENDPOINT parameter somehow.

Today, I gave up trying to get the previous attempt working, and restored a backup of my Droplets that were just Load-Balancing with just HTTP and finally did get it working again by doing this tutorial (Set up file and image uploads to S3) but this time I edited the S3 settings via the Discourse Admin panel rather than editing the app.yml with the settings in the recommended tutorial. It finally worked, but the important difference is that I deliberately left out the S3 CDN settings. I have confirmed that images uploaded to posts are being stored on S3 and I can backup Discourse directly to S3, and this is really all I want, but, I now have three issues that are hounding me, one is critical, and two are ignorable, although I’d like to confirm that here if possible.

So, the critical issue is that users can no longer login using the Patreon login button on the Discourse login page. This message is displayed:
Sorry, there was an error authorizing your account. Please try again.

The URL is this:
https://mbp.community/auth/failure?message=invalid_credentials&origin=https%3A%2F%2Fmbp.community%2Flogin&strategy=patreon

I would really appreciate some advice on what I could possibly try to get this working, but again, I’m wondering if this is because internally the servers are not running HTTPS. As you can see from the URL, externally they are on HTTPS, so it’s hard to know for sure. I guess I’m hoping someone here has some experience with the Digital Ocean Load-Balancing etc. with Discourse.

The other two issues are being called out now in the Admin Console as below:

Some advice based on your current site settings

  • Your website is using SSL. But [force_https](https://mbp.community/admin/site_settings/category/all_results?filter=force_https) is not yet enabled in your site settings.
  • The server is configured to upload files to S3, but there is no S3 CDN configured. This can lead to expensive S3 costs and slower site performance. See “Using Object Storage for Uploads” to learn more.

So, I don’t mind trying to turn on force_https, but I’m worried that it will lock me out of my server because internally the load-balanced servers are not running on HTTPS and due to the problems I had yesterday, I am reluctant to spend another twelve hours banging my head against a brick wall watching countless 15-minute rebuilds of Discourse only to get nowhere. Again, if anyone knows that it’s safe to turn on force_https with my configurations, please let me know.

And the second issue, again, did not go well via the parameters added to the app.yml file, so I’m reluctant to try this again as well. Can you confirm that this would essentially do the same as parameters added to the app.yml file? If so, I’ll just ignore this second message. Conversely, if it is for some reason safe to try, let me know and I’ll make a backup and give it a pop.

Sorry for the long post. I hope you can figure out what I’m trying to get advice on.

1 Like
2

Then you really need to get help there because all that’s really supported here is a standard install.

Are you expecting more than 200K page views/day? If not, a single 8GB droplet with CDN will be much easier to manage and probably cheaper. And from what I can tell, there are at least a couple of ways those instructions are likely not to work for anyone.

First, did set you up an external redis as described in step 5? If not, I’d expect things to be at least a little bit broken. They imply that using sticky will “fix” it, but it really won’t. So you can expect hard-to-diagnose spurious errors. And they don’t specify a way to make sure that all of your instances are running exactly the same version of Discourse, so that too is likely to make things broken.

You really wanted to do that first, otherwise, actually, that setup can’t work, because some uploads will be on one server and some on another, and those instructions make no mention of the word “upload”, so I expect that if you’ve been using this for more than testing you’ve got something of a mess on your hands and you’ll need to sync up the uploads between your multiple droplets.

It specifically states that the Digital Ocean CDN doesn’t work with Discourse.

Did you use a different CDN as it recommends? Bunny.net is pretty easy and simple to set up.

2 Likes
3

Oh my, I don’t know who wrote that guide, but for sure it wasn’t someone very familiar with Discourse at scale.

The last step tells you can add more backends to the load balancer by using the droplet snapshot feature, but since the guide doesn’t mention using Object Storage for the uploads, doing that will completely break user uploads. Also if you follow that guide, updates will become non-trivial.

My advice to @Martin_Bailey is that you should not follow anything there. It will only result in a broken install, as you have found.

Please follow our official install guide if you want a quick and sane install of Discourse. Going off that path can quickly become a full time work.

3 Likes
4

Funny. I left a comment there outlining some of the issues and linking to Rafael’s post, but it was deleted.

3 Likes
5

Wow! OK, so I thought it had gone well although there were a few things I did notice that were incorrect, but I do have a load balanced installation now. Of course, the first thing I found was that image uploads were broken, which is why I used S3 to store images.

As it stands, it will take another truck load of work to go back to a single server, so is there anything I can do about the Patreon login issue? I will ignore the other two issues.

Thanks for your help either way. You guys do such a good job here.

6

Hi Jay, thanks for your help. In reply to your questions…

I’m not expecting a lot of users, as it’s a closed Patreon Community. My main goal was to be able to update one server without it taking the site down. I’ve actually confirmed that this is possible, so I was happy with the setup. Yes, I did do step five, so state is being stored on an external Redis droplet.

The other thing that I had to figure out, which held me back for a while, was that I also needed to add the below parameter to app.yml, otherwise, the rebuild kept failing because it was trying to connect to Postgres on the default port, despite having the actual port in the DISCOURSE_DB parameter.

DISCOURSE_DB_BACKUP_PORT: 25060

I hadn’t thought about uploads until after I’d got everything working based on the first tutorial, and initially it did break everything when I tried to set up S3, but that was because the DO Space CDN settings that you guys here provide don’t work.

It specifically states that the Digital Ocean CDN doesn’t work with Discourse.

I know, but then the tutorial has us add this:
DISCOURSE_S3_ENDPOINT: https://sfo3.digitaloceanspaces.com

Which comes from the DO Space, right? I have no idea based on all that I’ve read in these tutorials how I would work with a different CDN, but I’m not concerned at this point, as I’ll cover in a moment.

No, I didn’t use a different CDN. I’m actually fine not using a CDN. I will leave the CDN settings empty. As a further update, I based on the advice you’ve all kindly provided so far, I was just going to revert to my backup from last week, but I figured that I’d try enabling the force_https option first, and enabling that fixed the Patreon login issue, as I’d thought it might. Nothing was changed on the server(s) so the Patreon login issue was probably caused by some internal Discourse logic, although again, I realize (now) that I’m doing something that you don’t recommend or support.

So, at this point, my set up is pretty much as the first tutorial recommends, but images and backups are all going to S3, without a CDN in place. It’s working really well. I appreciate that you are recommending I just use the standalone install, but taking the site down for 15 minutes every time an update comes out is really painful. Just yesterday I did find your references to data.yml and web_only.yml for a mulitserver setup, but I couldn’t figure out what I was supposed to do, so I gave up on that.

I’m going to run with what I have for now. Thanks for your help, and for all that you guys do.

7

OK guys, you win. :slight_smile:

I started to see more issues with images not loading again because they were sometimes being shared over http and not https. You’re right, it’s a mess.

I’ve just backed most of the way out of this, leaving the Postgres database in place but back to just one server droplet, no load-balancer and images/Redis stored locally. I’ve left S3 in place for the site backups though. I love how smoothly that works.

I’m back to 15-minute outages with each upgrade I guess, but I’ve spent a total of like five full days on this so far, and I can’t spend any more time on it, so I’m happy that you were all able to put me straight regarding that original tutorial I followed. It’s almost as though they just want people to pay for more Droplets. :slight_smile:

Thanks again for your help.

8

Actually, if I could just ask, is there a tutorial to help us to set up Discourse in a way that makes it possible to avoid that 15-minute outage with each update. I saw the note about the data.yml and web_only.yml but I haven’t a clue what I need to do to make that happen.

9

Having separate data and web_only ymls is from the two-container set-up:

1 Like
10

This will work and resolve some issues. And if you happen to get locked out for some (other) reason you can always launcher enter app your way back in and toggle the site setting back from the Rails console.

As others already said, it’s better to follow a different guide though.

1 Like
11

Hi guys,

I wrote that DigitalOcean article, it looks like I made a few mistakes, my bad!

I’d like to update the article to correct the issues.

So I’d just like to ask a few questions to make sure I get things right with the corrected version.

In the article I said you could optionally use a Managed Redis instance, my thinking at the time was that using sticky sessions would allow you to use the builtin Redis in the Discourse image. Is this correct? Maybe this isn’t sufficient and an external Redis instance should be a compulsory requirement.

I agree with @Falco 's comment about Object Storage, that’s an oversight from me. I can update the article to include instructions for using DigitalOcean Spaces to handle image uploads.

I suspect removing all state from the Droplets is the answer to solve the installation problems, I had hoped using an external Redis was optional due to sticky sessions but I could be mistaken.

I agree also that this makes your Discourse installation more complicated to upgrade, I think though if you remove all state from the Droplets, you should be able to just upgrade one Droplet, then snapshot it and just replace the other Droplets with new ones created from the snapshots (A bit like Kubernetes Deployments, except you do the re-deployment manually).

I agree also that there probably are better ways to scale Discourse, I’d still like to correct the article though so people can follow it if they’d like to.

Thanks,
Francis

12

I am a happy Digital Ocean customer and have a dashboard that my clients enter API keys and a hostname and thenit will automatically create a droplet, configure mailgun, wait for DNS records to be set up, and then install Discourse.

It doesn’t work at all as you suggest. I contacted digital ocean on your forum (I couldn’t find another way) to let you know and my message was deleted. Then, 9 months later you reply here.

Doing it correctly is quite a complicated feat, and the cases that it would be useful are fairly far-fetched. I have a site with 100K pageviews per day on an 8GB droplet. Who do you think is the target audience for this guide?

Yes, you need external redis, postgres, and S3 buckets with a CDN, and the Digital Ocean CDN doesn’t work, so your guide would need to walk them through setting up some other company’s CDN. I don’t think you want to do that. And that’s just getting it set up. If they then want to do an upgrade, it would be a whole other set of procedures that no one else on the planet would know how to help with.

The best thing you could do would be to delete that guide altogether. If you want to be a real hero, you could fix the 1-click install so that it didn’t use your own proprietary script for setting up the email and so on, so that it would actually be a standard install. It’s pretty confusing to have to control-c to be able to get to where Discourse is, and since people who used the one-click didn’t use the standard Discourse tools, they don’t know how to use them when it comes time for a command-line upgrade. It would be really, really great if you could do that.

Here you can see people who used it and were having trouble: Search results for 'digital ocean one-click' - Discourse Meta

1 Like
13

No, as Redis isn’t a simple cache, but handles a lot of counting, rate limits, background jobs, pub sub. Having multiple Redii will result in corrupted counting and broken behavior.

That would do it, but adding a managed Redis, managed PostgreSQL, managed Load Balancer, Object Storage, Container Registry will also be more expensive than just paying for our standard hosting and a lot more complicated to maintain.

At that point the intersection between people who want to pay hundreds of dollars per month and don’t care if their service go down because of a SPOF is quite small, and it becomes more a hobbyist setup than what we would recommend for forum admins.

1 Like