I’m continuing this conversation from ‘ Rebuild HTML for entire topic ’ as my experiments are going into quite another direction and I thought there might be value in sharing my thoughts and results as I go along. My situation is the following: we’re on the brink of launching a new migrated forum wit…

We should normalize the rake task to go in descending ID order as well @techapj . Unless this is super hard, many hours of work, or something?

Agree, but it is a bit tricky cause we would need to carry a big list of ids in memory. I wonder if we should amend it so the rake task is resumable? Have rake posts:rebake reset version and just work through old posts using calls to rebake_old And add rake posts:rebake:resume that simply resumes …

[image] sam: a big list of ids in memory Is carrying a list of integer IDs in memory really that expensive?

[image] codinghorror: Is carrying a list of integer IDs in memory really that expensive? we can probably live with it to be honest … that retains the tasks working exactly as they do today (in reverse order). Though something in me wants these tasks to be resumable cause if you are working th…

Maybe V1 can be the simple version with a comment // TODO: make this resumable because carrying around 20 million ids in memory is not a great idea long term

Done via: FIX: rake task should rebake posts in descending ID order committed 06:48PM - 09 Apr 18 UTC (UTC) [image] arpitjalan +6 -3 …

I’ve used a script that was resumable at the topic level by using the custom fields. Here’s one that skips private messages (since my import had a LOT of them and they weren’t a priority): Topic.includes(:_custom_fields).where(archetype: Archetype.default).find_each do |t| unless t.custom_fields[…

[image] pfaffman: So rather than running rake posts:rebake, one should instead do Posts.all.update_all(‘baked_version: null’) and all posts will be rebaked in batches according to rebake_old_posts_count? Yes, and @bartv would be able to get his “rebuild for just one topic” by doing: Posts.wh…

What’s the frequency of these new batches, and how can you monitor the progress?

[image] sam: but it is a bit tricky cause we would need to carry a big list of ids in memory. I wonder if we should amend it so the rake task is resumable? This is now done via: PERF: do not carry post ids in memory when rebaking all posts …

My journey into a massive posts rebake job

Support Self-hosting

techAPJ (Arpit Jalan) April 9, 2018, 6:53pm 17

Done via:

4 Likes

Topic		Replies	Views
Rake posts:rebake_incremental (feature request) Feature	2	793	April 22, 2020
Rebake all posts? Self-hosting	16	9044	October 28, 2022
Slow rebake script to avoid overwhelming your server Migration	4	142	November 13, 2025
Rebake with rails command or rake task doesn't work, but rebuilding HTML does. Why? Self-hosting	11	1919	April 26, 2022
Rebuild HTML for entire topic Feature	6	3893	April 8, 2018

My journey into a massive posts rebake job

Related topics