Installed OK. Working fine for one day. Suddenly stopped working

s-moon · June 19, 2018, 11:24pm

Hi All,

I’ve recently moved the discourse forum from one server to another - an old Ubuntu (13?) to version 16.06. To the best of my understanding, it is running in a docker container with an apache/nginx front-end and is using version: 2.1.0.beta1 +282.

To accomplish this move, I upgraded to the latest version on the existing server, took a full backup, then installed the latest version on the target and imported the data. All seemed fine for one day.

Today, I was able to access the server (I was still logged in) but other people couldn’t. Then, I lost my access, meaning I got a blank screen, which is what other people are experiencing. One user did say that it worked on her phone, but not her main PC, but I suspect it is not working on either now.

Looking in Chrome dev tools, I have a host of errors of the form:

(index):32 Uncaught SyntaxError: Invalid or unexpected token
_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59785 Uncaught Error: Could not find module `preload-store` imported from `discourse/models/topic`
    at o (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59785)
    at i (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59763)
    at _ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59820
    at e (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59687)
    at require (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59819)
    at _application-fad5e5f87c66e66d6c184629209e8cc05bc33467507b76e7013a3ff871328b38.js:12551
_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59791 Uncaught Error: Could not find module preload-store
    at s (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59791)
    at require (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59806)
    at (index):307
    at (index):307
_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59791 **Uncaught Error: Could not find module preload-store**
    at s (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59791)
    at require (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:59806)
    at (index):313
    at (index):313
**(index):317 Uncaught SyntaxError: Invalid or unexpected token**
_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:29446 **TypeError: Cannot read property 'startRouting' of undefined**
    at startRouting (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:15321)
    at s.didBecomeReady (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:16290)
    at invokeWithOnError (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12554)
    at a.flush (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12613)
    at u.flush (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12737)
    at p.end (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12807)
    at p.run (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12921)
    at p.join (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12952)
    at Function.u.join (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:34030)
    at HTMLDocument.<anonymous> (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:34093)
u @ _ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:29446
_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:29446 TypeError: Cannot read property 'startRouting' of undefined
    at startRouting (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:15321)
    at s.didBecomeReady (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:16290)
    at invokeWithOnError (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12554)
    at a.flush (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12613)
    at u.flush (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12737)
    at p.end (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12807)
    at p.run (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12921)
    at p.join (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:12952)
    at Function.u.join (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:34030)
    at HTMLDocument.<anonymous> (_ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:34093)
u @ _ember_jquery-27e777857b8c0730dacfe09cb11711365d21a5db4f9ee0b85d494e4259cf6cda.js:29446

Things I have tried include:

restarting Apache
restarting NGINX
stopping the app in the container/docker
restarting the app in the container/docker
I’ve rebuilt the app.
I’ve read this: How do I manually update Discourse and Docker image to latest?
– as a consequence, I did the git pull, rebuild step - I already had the latest code.
– I also tried the safe mode - that seemed to get me in once or twice, but now no longer works.

Looking at the code, I see that it is unhappy with this on the main page (view > source)

<script>//<![CDATA[
e��ø�[�)q�h��i�$�5�D4�����z��J�;cmঀq`����\z��,��[��Ы��m��>J���!%>��R	�R�<i���L��(䔽�$��b\1�_�kH�#�BF;*�>7�\V��T�K��D���B�bK�*���aޢ-��u7�Q��e��b�fۺ�QC��V�]���a�3U��0�.k��LD��U�0˙�e.=1�.�>�H:��������*���(=i�)%kk^9d>�p���,���8��yH	:�Y3����G�/���kQ����:p���G+�1XF���}] �ʘ���x���
//]]></script>

And looking through the code in the docker container, I can see where that is inserted (in file: /app/views/layouts/application.html.erb):

<%= preload_script "ember_jquery" %>
------>    <%= preload_script "preload-store" %>
<%= preload_script "vendor" %>

Does anyone have any ideas as to why this might have occurred and how I could resolve it?

Thanks, Steve

itsbhanusharma · June 19, 2018, 11:43am

If I’m not going miserably off topic, You shouldn’t have either of those installed for the vanilla 30 minutes docker install. Could you please do the following steps:

sudo apt-get purge apache2
sudo apt-get purge nginx
sudo apt-get autoclean
cd /var/discourse
git pull
./launcher rebuild app

s-moon · June 19, 2018, 11:55am

Hi @itsbhanusharma

Thanks for the reply. This forum is running on a server which hosts other applications - WordPress, api etc. I may not need the NGINX (my ignorance of that doesn’t help) but I am pretty sure I don’t want to remove Apache.

I can post the config files I am using for Apache if that is helpful plus my app.yml file. As I mentioned, why this would work for a day and then suddenly stop is what is even more intriguing. Is it cache related?

itsbhanusharma · June 19, 2018, 11:59am

If You’re pretty sure that you can’t remove apache2, my simplest recommendation will be to move to a server dedicated to your forum. However, if you have to have forum & other applications on the same server then please search this forum for topics about configuring access to discourse with apache2 There have been some success cases but IMO those don’t really work well.

There is a lot that can happen and a lot more that may have been borked with time. You took a very nuclear decision and should have done some research before attempting such a complicated setup.

_vincent · June 19, 2018, 12:20pm

This is not a complicated setup… but it requires some work and documentation.

itsbhanusharma · June 19, 2018, 12:31pm

first, some research should have been done.
Second, He’s talking of apache2 if I’m not mistaken. Article linked above contains nginx which plays very well with discourse.
Also, not to forget, there are 3 services fighting for port 80 & 443 above.

EDIT:
Found a #howto though

Reason why it doesn’t works well is well documented by sam in comment below that tutorial. Please read it.

_vincent · June 19, 2018, 12:32pm

I would recommend OP to drop Apache totally and use nginx outside of the Discourse container instead.

itsbhanusharma · June 19, 2018, 12:33pm

They have their API and website already running on apache2 dependency thing maybe? He mentioned that he can’t get rid of apache!

s-moon · June 19, 2018, 12:44pm

Hi @_Vincent and @itsbhanusharma

We’re already running some web apps under Apache so ideally, I would like to leave that as is - but I note your suggestions above @_Vincent and will definitely consider those (am also reading them now, too).

My app.yml file asks the container to listen on port 3000 and fwd to 80.

templates:
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
  #- "templates/web.ssl.template.yml"
  #- "templates/web.letsencrypt.ssl.template.yml"

## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
expose:
  - "3000:80"   # http
#  - "443:443" # https

I’ve also included the templates being used as they were mentioned in the other posts.

My Apache config for the site is as follows:

<IfModule mod_ssl.c>
<VirtualHost _default_:443>
        ServerAdmin me@server-name-we-use.org
        ServerName server-name-we-use.org

        ProxyPreserveHost On
        ProxyRequests Off
        ProxyPass / http://localhost:3000/
        ProxyPassReverse / http://localhost:3000/
        RequestHeader set X-Forwarded-Proto "https"
        Header always set Strict-Transport-Security "max-age=31536000"

[chopped for brevity]

As you can see, there is a SSL certificate in place, too.

Assuming I have replicated the setup and not made a mistake somewhere, this was working on another server as-is so I know it ought to be possible. I didn’t set this up originally though hence my gaps in understanding.

The original docs I have did say about installing NGINX but I can’t see how that was hooked in and it seems to be routing traffic in part.

Why would this work for a day though? Any ideas what might be the reason for that? Nothing changed (by me) in between.

Thank you for your assistance, Steve

itsbhanusharma · June 19, 2018, 12:51pm

It can be either luck or something that self updated. bothered digging logs to find out what was working and had since stopped working?

There are some problems with apache that will not let long polling function as expected. This heavily limits the functioning of discourse. Hence, it is not a recommended setup.

One more thing I would recommend though is looking at the rest of your apache config so to be sure that there are no default or global parameters that may be interfering with discourse.

s-moon · June 20, 2018, 8:30am

I have a bit more information.

Somehow, suddenly, the site started working yesterday. I am not sure why and it was after a small while, so not obviously immediately after I had done something.

However, it stayed up for about 18 hours. I logged in this morning and all was well. I replied to a post and saw that some other people had successfully logged on.

A little while later, it went down again.

I restarted Apache. No change.
I stopped and started the docker app. No change.
I rebuilt the app and watched it run. Still, no change.
I tried safe mode with all options, no change.

Looking at the production log file, I did see this which happened a few times before the site dropped.

Job exception: Cannot assign requested address - connect(2) for [::1]:6379

Job exception: Cannot assign requested address - connect(2) for [::1]:6379

Job exception: Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)

Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED) subscribe failed, reconnecting in 1 second. Call stack ["/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:344:in `rescue in establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:328:in `establish_connection'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:99:in `block in connect'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:291:in `with_reconnect'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:98:in `connect'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:363:in `ensure_connected'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:219:in `block in process'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:304:in `logging'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:218:in `process'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis/client.rb:118:in `call'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis.rb:889:in `block in get'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis.rb:45:in `block in synchronize'", "/usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis.rb:45:in `synchronize'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/redis-4.0.1/lib/redis.rb:888:in `get'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.1.5/lib/message_bus/backends/redis.rb:285:in `process_global_backlog'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.1.5/lib/message_bus/backends/redis.rb:321:in `block in global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.1.5/lib/message_bus/backends/redis.rb:334:in `global_subscribe'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.1.5/lib/message_bus.rb:530:in `global_subscribe_thread'", "/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/message_bus-2.1.5/lib/message_bus.rb:478:in `block in new_subscriber_thread'"]
Migrating to AllowNullIpSearchLog (20180521184439)

Could it be connected?

I’ve been checking the Apache configs as suggested and nothing seems an issue so far. Anything in particular you think I should look for/be worried about?

Thanks, Steve

itsbhanusharma · June 20, 2018, 8:08am

Seems like your redis is failing! perhaps a RAM issue?

s-moon · June 20, 2018, 1:43pm

I’ve ruled out RAM as being the cause. I’ve been monitoring the memory and doubled the VMs allocation to 16GB, so that is fine.

I now think this is related to SSL certificates because when I connect to the site without them through the Apache conf, I can get in.

I cannot log in however because in the prior site, it had HTTPS enabled. So, is there a way to disable that without logging in as an admin (because I can’t for the reason just mentioned).

Thanks for any assistance you can give,

Steve

ps - I’m going through this site and have found a few posts about how to enable SSL in NGINX (in the container) which I am attempting, but that isn’t working out just yet, so need a plan B.

itsbhanusharma · June 20, 2018, 1:46pm

You can change that site setting from rails console, Query can be found somewhere on this forum.

s-moon · June 20, 2018, 1:56pm

@itsbhanusharma that’s a big help - found a post here: How to change a site setting via the Console – see the last entry.

Thank you,

Right - now onto fixing the SSL