Timeouts via MiniRacer::ScriptTerminatedError


(Andrew Waugh) #1

I am trying to get familiar with some of the inner workings of Discourse (specifically how merging users works). To this end I have created an AWS instance, installed discourse, and run the setup, then modified the app.yml to disable email, then imported an old backup from our live site, so I have some data to work with.

The problem is this: Our live site has 1.8 Mio Posts, and about 34K user accounts. Restore ran fine on a t2.micro with 1GB Ram. Everything is there, and the site works, if a bit slow.

But…

Because the instance is memory bound, and there is so much to crawl through, things like users:change_post_ownership and rebake exhaust the RAM and I get “MiniRacer::ScriptTerminatedError: JavaScript was terminated (either by timeout or explicitly)”

I’ve tried deleting a bunch of categories to reduce the sheer number of items, but the size of the database stays the same.

Is there some (easy) way to really drop some of the stuff I restored from the backup, or do I need to export a couple of categories from our live site, create a new discourse install and import just the exported categories? (As I understand it, exporting a category includes the category, the users, and the posts, but not the uploads.)


(Sam Saffron) #2

This worries me a bit, I think you have some rogue posts there, you are going to have to isolate the post that causes this and send us the “body” so we can fix whatever bug there is.


(Andrew Waugh) #3

I’m not sure if it is just the sheer number of posts, or we have some Out of bound issues in posts. I’ve tried the dataexplorer queries mentioned in (Finding and eliminating a DatetimeFieldOverflow post) but they just timeout.

It’s a throwaway installation, so I can run any commands you can give me which would help me figure out if it is a rogue post or just the sheer size.

I can also give you ssh access, but you might have to explain how I do that with an AWS instance.


(Sam Saffron) #4

MiniRacer::ScriptTerminatedError would totally be a single post being rebaked. Somehow, someone, managed to author raw markdown that is so hard for our parser to parse that it times out.


(Andrew Waugh) #5

I follow that logic on the rebake, but I am also getting crashes when I do change_ownership - Isn’t change ownership just a database pointer change sort of operation, so the content of the post itself is irrelevant?

Of course, my dog might have tics and fleas…


(Sam Saffron) #6

I think it rebakes certain posts to fix up quotes and so on.


(Andrew Waugh) #7

Is there some trace I can run during a rebake to tell me which post the action is failing on? Or more specifically why?

Here is some output from the logs:

> root@ip-172-31-6-182:~# docker logs a8e|grep Ong_Ham
>         <p>In a message dated 6/20/2006 9:04:57 AM Pacific  Standard Time,<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a> writes:<br>Mike, the word "steps" evaded me a  while ago -- I am interested in the <br>steps<br>to reassembly, in fairly general  terms.  At this specific time, I am <br>needing<br>to move past the  wipers/squireter area and don''t know if I should start <br>with<br>the frame  assembly now, or work on the steering wheel/dash area of the<br>console.... or  some other area that is going to be a problem later if I<br>don''t do it  now.<br>Thanks,<br>Bruce</p>
> 2017-02-01 12:30:13 UTC [2281-11] discourse@discourse LOG:  duration: 837.359 ms  execute <unnamed>: UPDATE "posts" SET "cooked" = '<p>Anyone have a good lower wishbone for passenger side of an S1 roadster for<br>sale?  If so, please contact me offline. Thanks<br>Bruce<br>64ots<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a></p>
> 2017-02-01 12:30:28 UTC [11234-3] discourse@discourse LOG:  duration: 101.517 ms  execute <unnamed>: UPDATE "posts" SET "cooked" = '<p>Walt, I would love to have the alignment points document. Thanks very <br>much<br>Bruce Harless - 64 S1 3.8 ots<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a></p>
> 2017-02-01 12:30:29 UTC [11234-4] discourse@discourse LOG:  duration: 156.720 ms  execute <unnamed>: UPDATE "posts" SET "cooked" = '<p>Paul, I have my 64 3.8 S-1 apart, down to the firewall  on both sides -- is<br>there a picture I can take and send you that would help?<br>Bruce Harless<br>VaBeach<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a></p>
> 2017-02-01 12:30:34 UTC [11295-6] discourse@discourse LOG:  duration: 169.066 ms  execute <unnamed>: UPDATE "posts" SET "cooked" = '<p>Dick, I''d like to buy one of the CD''s for my own use. Can we mail you a<br>check for the CD and postage and you send one via USPS?<br>Bruce Harless<br>Va Beach<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a></p>
>         <p>Alan, 1/2 mile west and I''d be in Chesapeake -- down Indian River Rd and<br>across Mil. Hiway. Send me a note offline and we''ll see about meeting each<br>other. Would be good to meet you.<br>Bruce<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a></p>
> 2017-02-01 12:30:56 UTC [1779-12] discourse@discourse LOG:  duration: 272.543 ms  execute <unnamed>: UPDATE "posts" SET "cooked" = '<p>Mike, would you mind sending me a picture or two of your tunnel mods? I <br>would appreciate it.<br>Thanks much<br>Bruce Harless<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a>----- Original Message ----- <br>From: <a href="mailto:mmoore8425@aol.com">mmoore8425@aol.com</a><br>To: <a href="mailto:e-type@jag-lovers.org">e-type@jag-lovers.org</a><br>Sent: Tuesday, August 02, 2005 5:47 PM<br>Subject: Re: [E-Type] Modification to tranny tunnel cover</p>
> 2017-02-01 12:32:35 UTC [11295-15] discourse@discourse LOG:  duration: 158.706 ms  parse <unnamed>: SELECT  1 AS one FROM "groups" WHERE "groups"."name" = 'Ong_Ham' LIMIT 1
>         <blockquote><blockquote><blockquote><blockquote><blockquote><blockquote><blockquote><blockquote><p>From: "Bruce" &lt;<a class="mention" href="/users/ong_ham">@Ong_Ham</a>&gt;<br>Subject: Re: [E-Type] Should I re-paint the car, touch-up or leave it alone?</p></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>
> 2017-02-01 12:33:31 UTC [2496-17] discourse@discourse LOG:  duration: 149.425 ms  parse <unnamed>: SELECT  1 AS one FROM "groups" WHERE "groups"."name" = 'Ong_Ham' LIMIT 1
>         <p>Two weeks ago I used a very good vendor for resleeving-- excellent service <br>and price, will be glad to share the information with whoever wants to <br>contact me offline. <a class="mention" href="/users/ong_ham">@Ong_Ham</a>.<br>Bruce Harless----- Original Message ----- <br>From: Len Wheeler<br>To: e-type@jag-lovers.org<br>Sent: Monday, September 11, 2006 8:57 PM<br>Subject: Re: [E-Type] Caliper cylinder brass sleeves</p>
> 2017-02-01 12:36:33 UTC [20422-4] discourse@discourse LOG:  duration: 127.824 ms  bind <unnamed>: SELECT  1 AS one FROM "groups" WHERE "groups"."name" = 'Ong_Ham' LIMIT 1
> 2017-02-01 12:36:36 UTC [20422-5] discourse@discourse LOG:  duration: 434.603 ms  execute <unnamed>: UPDATE "posts" SET "cooked" = '<p>Dave, if you do get a picture of the clamp for the overflow pipes, would you <br>please include me in the distribution? Thanks,<br>Bruce Harless<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a>----- Original Message ----- <br>From: "Dave K" <a href="mailto:david@kerr2209.fsnet.co.uk">david@kerr2209.fsnet.co.uk</a><br>To: <a href="mailto:e-type@jag-lovers.org">e-type@jag-lovers.org</a><br>Sent: Monday, January 15, 2007 5:53 PM<br>Subject: [E-Type] Re: SU CARB Overflow Pipes</p>
>         <p>-------------- Original message ----------------------<br>From: "ongham" &lt;<a class="mention" href="/users/ong_ham">@Ong_Ham</a>&gt;</p>
> 2017-02-01 12:37:13 UTC [20238-7] discourse@discourse LOG:  duration: 165.677 ms  parse <unnamed>: SELECT  1 AS one FROM "groups" WHERE "groups"."name" = 'Ong_Ham' LIMIT 1
>         <p>In a message dated 3/22/2007 3:55:37 A.M.  Pacific Daylight Time,<br><a class="mention" href="/users/ong_ham">@Ong_Ham</a> writes:<br>I think Stefan must be off on a  buying trip or something -- he seems to<br>disappear like this on a repeating  schedule. I contacted him about a <br>month<br>ago about some parts, then he went  missing and hasn''t resurfaced. But he<br>always does, so I figure he is just  out of  pocket.</p>
> 2017-02-01 12:40:01 UTC [1797-617] discourse@discourse LOG:  duration: 1033.677 ms  parse <unnamed>: SELECT  1 AS one FROM "groups" WHERE "groups"."name" = 'Ong_Ham' LIMIT 1
> 2017-02-01 12:40:03 UTC [1797-618] discourse@discourse LOG:  duration: 2010.679 ms  bind <unnamed>: SELECT  1 AS one FROM "groups" WHERE "groups"."name" = 'Ong_Ham' LIMIT 1
> 2017-02-01 12:40:04 UTC [1797-619] discourse@discourse LOG:  duration: 133.419 ms  execute <unnamed>: SELECT  1 AS one FROM "groups" WHERE "groups"."name" = 'Ong_Ham' LIMIT 1

(Andrew Waugh) #8

Shouldn’t the change ownership should rebake all the posts as it runs, the whole topic for that matter?

This would leave the changed posts and their containing threads ready for use, so the only missing bit would be any posts which mention the old username (I am talking specifically about user merge/rename actions here, which is about the only use for a global change ownership I can think of).


(Andrew Waugh) #9

Ok, I think I’ve narrowed it down to a single post by looking at /logs

There I see an event: Job exception: JavaScript was terminated (either by timeout or explicitly)
The env tab shows:

hostname	ip-172-31-6-182-app
process_id	[25394, 137]
application_version	0f2de4863b9b35289f0e37ff8ae0d52a5bc4bafc
current_db	default
current_hostname	discourse.waugh.net
job	Jobs::ProcessPost
problem_db	default
	
opts	null
post_id	455123
bypass_bump	false
invalidate_oneboxes	true
current_site_id	default

So, how do I then find post 455123? Searching for it in the GUI doesn’t work.


(Sam Saffron) #10
http://SITE_URL/raw/POST_ID

should do the trick!


(Andrew Waugh) #11

Oops! That page doesn’t exist or is private.

Can you pass me the larger hammer please?


(Gerhard Schlager) #12

Since you seem to have the post_id, try this:

http://SITE_URL/posts/POST_ID/raw


(Andrew Waugh) #13

With http://SITE_URL/posts/POST_ID/raw I was able to find the topic which it is in via the GUI search.

Thing is, it looks perfectly normal, I was able to rebuild the HTML in the GUI.

I’ve tagged the thread.

I wanted to rerun the change_ownership command which made it barf the minracer error, but I ran ./launcher rebuild app instead of enter app

I’m thinking if I can rebuild the html then the post content should be clean (I am presuming that rebuilding html is the same as rebake). Once the app rebuild is finished I’ll rerun the change_ownership command.


(Andrew Waugh) #14

Ok, The change_ownership still falls over and increments the job exception in /logs.

Here is the offending post:

http://35.164.82.91/posts/455123/raw

Surely if I can rebuild the html then it isn’t a problem with the content of the post?


(Sam Saffron) #15

Try doing a divide and concur on that post to find out what part of it causes the failure. (erase half, try again, add a quarter, try again, and so on)


(Andrew Waugh) #16

Ok.

So far I’ve had to do a reboot after each time it has failed, I have also been doing a rebuild. I realize the rebuild might be a bit OTT, but is there no way to avoid the reboot?

./launcher stop doesn’t work, neither does docker kill.


(Sam Saffron) #17

Very strange, are you running latest Docker?


(Andrew Waugh) #18

Yep, per the README.

The only odd thing about the instance I can think of is that I changed the swapfile to 4G when I thought it was running out of ram.

I think I’ll create another free tier instance and start again.


(Andrew Waugh) #19

Similar stuff on the new instance and a more recent backup from the live site.

I think I’m closing in on it:

The sidekiq log showed that the miniracer was happening a lot. Checked out some of the raw posts and they didn’t seem to have any weird characters or such in them, then I notice that they were pretty long… Sure enough, larger than “max post length” in /admin.

So… 2 questions:

  1. How do I find out what the longest post is?
  2. Is there a way to split such posts programmatically, or am I faced with a choice between setting “max post length” to the size of my largest historical post, or truncating/deleting those posts?

(Jeff Atwood) #20

How long are the posts and how much markup do they have in them?