Rebake with rails command or rake task doesn't work, but rebuilding HTML does. Why?

Canapin · March 25, 2022, 9:35pm

So, here’s where I am:

After trying to do a post.rebake!(invalidate_broken_images: true) on all my 40000 posts that contains the string [img], it worked for a lot of images… But far from all, despite being hosted on the same external image hosting service.
For example, I have thousands of “working” casimages links (that links to valid images, and show images in the composer preview on edit), broken in the cooked version of the posts, that were properly displayed and uploaded on the server thanks to my script, but I also have a lot of other ones where it simply didn’t, and I don’t know why.

Post.where('raw LIKE ?', '%[img]%').find_each do |p|
    p.rebake!(invalidate_broken_images: true)
end

I also have images links from other image hosting that were uploaded, and some on which it didn’t work.

I failed to see any difference between these posts and image links. They all had working images, and the fact that they used the same images hosting puzzled me.

I tried the operation multiple times and the results were inconsistent, regardless of the external hosting services… Some images were uploaded, some weren’t. It looked like of random.

It reminds me a bit of the issue that encountered @Amethi: Some linked images not displaying/show as broken - #8 by Amethi where it worked only on some images only without any explanation.

I’ll talk only about casimages here though my imported forum used various other image hosters.

So, I thought that maybe casimages temporarily blacklisted my IP if I tried to retrieve too many images from their servers. That could explain both the fact that it didn’t work for all images and the randomness of the success of uploading the images from my server.
There were even cases where the Rebuild HTML option worked -at first only-, the images were then displayed instead of showing a broken image icon, though there were still hosted on their external hosting service, but when the pull external image Sidekiq task was triggered it broke the images.
Same by using rail scripts with rebake!(invalidate_broken_images: true)

So, I’m currently trying a slower approach, where I wait 5 seconds between each of my rail rebake! commands:

total = Post.where('lower(raw) LIKE ?', '%[img]https:%').count
i = 0
Post.where('raw LIKE ?', '%[img]https:%').find_each do |p|
    p.rebake!(invalidate_broken_images: true)
    print "#{i}/#{total}"
    print "\r"
    i +=1
    sleep(5)
end

I’ll see in ~60 hours if it went better…

I’d like to understand the fundamentals of my issue here and why a “normal” rebake can’t upload an image on the server (if I’m not temporarily blacklisted by casimages).

Note that this time, the certificate of casimages’s server seems OK

I also don’t understand what invalidate_broken_images really does. I’m not very familiar with Discourse’s code.

I look at the code to see the occurrence of invalidage_broken_images and saw these files:

github.com

discourse/discourse/blob/b72688340ffdf0d7424e1914644ec93c800812f6/lib/tasks/posts.rake#L630

      
        
              end
              if RailsMultisite::ConnectionManagement.current_db != "default"
                recover_uploads_from_index(path)
              else
                RailsMultisite::ConnectionManagement.each_connection do
                  recover_uploads_from_index(path)
                end
              end
            end
            
            
desc 'invalidate broken images'
            task 'posts:invalidate_broken_images' => :environment do
              puts "Invalidating broken images.."
            
            
  posts = Post.where("raw like '%<img%'")
            
            
  rebaked = 0
              total = posts.count
            
            
  posts.find_each do |p|
                rebake_post(p, invalidate_broken_images: true)

Why it is searching specifically for the <img string? My posts are from an imported phpBB and the raw version contains only [img] bbCode, not <img> tags; so how it would have an effect (and it did, see my previous message) on my posts?

I also don’t really understand the difference between these two methods (?):

github.com

discourse/discourse/blob/20a954fa0f74b730996a36913588a9a1528c88df/app/models/post.rb#L702

      
        
                  else
                    post.custom_fields["rebake_attempts"] = attempts + 1
                    post.save_custom_fields
                  end
            
            
    end
              end
              problems
            end
            
            
def rebake!(invalidate_broken_images: false, invalidate_oneboxes: false, priority: nil)
              new_cooked = cook(raw, topic_id: topic_id, invalidate_oneboxes: invalidate_oneboxes)
              old_cooked = cooked
            
            
  update_columns(
                cooked: new_cooked,
                baked_at: Time.zone.now,
                baked_version: BAKED_VERSION
              )
            
            
  if is_first_post?

github.com

discourse/discourse/blob/20a954fa0f74b730996a36913588a9a1528c88df/app/controllers/posts_controller.rb#L566

      
        
              guardian.ensure_can_change_post_type!
              post = find_post_from_params
              params.require(:post_type)
              raise Discourse::InvalidParameters.new(:post_type) if Post.types[params[:post_type].to_i].blank?
            
            
  post.revise(current_user, post_type: params[:post_type].to_i)
            
            
  render body: nil
            end
            
            
def rebake
              guardian.ensure_can_rebake!
            
            
  post = find_post_from_params
              post.rebake!(invalidate_oneboxes: true, invalidate_broken_images: true)
            
            
  render body: nil
            end
            
            
def unhide
              post = find_post_from_params

It seems to tell that rebake set the default arguments to false, and that rebake! sets the default argument to true.

How are these two related (I’m aware of the purpose of the ! character in ruby by the way), and why are they in different files?

My goal is only to understand why my external images are sometimes uploaded, sometimes not, and if I can find a reliable way to upload them properly and automatically, even if it implies uploading an image every hour.
I’ve been almost two weeks on this and it’s driving me (and the people which I migrated their server for) crazy.

Also, there is nothing in Discourse’s log, instead of multiple Sidekiq is consuming too much memory (using: 592.25M). Note that I’m working on Ubuntu via WSL on Windows 10, but I intend to use a working solution (if I find one…) on our VPS.

Topic		Replies	Views
My journey into a massive posts rebake job Installation	35	4104	July 10, 2024
Old image uploads become broken images Support	64	8222	June 8, 2024
Some linked images not displaying/show as broken Support	9	1692	March 21, 2022
Rake uploads:migrate_from_s3 fails Installation s3	31	2787	April 24, 2021
What rebake is required for the new autocomplete styling? Support hashtag-autocomplete	20	1432	March 12, 2023

Rebake with rails command or rake task doesn't work, but rebuilding HTML does. Why?

Related topics