JS embed fails with "Job exception: invalid stored block lengths (Zlib::DataError)"

icaria36 · November 22, 2021, 5:30pm

Hi, we cannot get JS embed to work in our setup. I have read all the other similar topics and I have got this embedding to work in a total different project in the past. This one looks like something different…

On the external site, only our logo can be seen followed by “loading discussion…”

On Discourse, no topics are being created, even if the error log shows that the requests are coming.

We have reviewed the URLs. We have also tested adding a static URL (instead of a variable). Nothing.

Discourse error log has this Error:

Job exception: invalid stored block lengths (Zlib::DataError)

excon-0.88.0/lib/excon/middlewares/decompress.rb:23:in `inflate'

excon-0.88.0/lib/excon/middlewares/decompress.rb:23:in `response_call'

excon-0.88.0/lib/excon/middlewares/base.rb:26:in `response_call'

excon-0.88.0/lib/excon/middlewares/instrumentor.rb:44:in `response_call'

excon-0.88.0/lib/excon/middlewares/base.rb:26:in `response_call'

excon-0.88.0/lib/excon/middlewares/expects.rb:20:in `response_call'

excon-0.88.0/lib/excon/middlewares/response_parser.rb:12:in `response_call'

excon-0.88.0/lib/excon/connection.rb:451:in `response'

excon-0.88.0/lib/excon/connection.rb:282:in `request'

excon-0.88.0/lib/excon.rb:250:in `get'

/var/www/discourse/lib/final_destination.rb:206:in `public_send'

/var/www/discourse/lib/final_destination.rb:206:in `resolve'

/var/www/discourse/app/models/topic_embed.rb:120:in `find_remote'

/var/www/discourse/app/models/topic_embed.rb:192:in `import_remote'

/var/www/discourse/lib/topic_retriever.rb:52:in `fetch_http'

/var/www/discourse/lib/topic_retriever.rb:39:in `perform_retrieve'

/var/www/discourse/lib/topic_retriever.rb:12:in `retrieve'

/var/www/discourse/app/jobs/regular/retrieve_topic.rb:15:in `execute'

/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'

rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'

/var/www/discourse/app/jobs/base.rb:221:in `block in perform'

/var/www/discourse/app/jobs/base.rb:217:in `each'

/var/www/discourse/app/jobs/base.rb:217:in `perform'

sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'

sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'

/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:143:in `invoke'

sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'

sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'

sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'

sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'

sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'

sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'

sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'

sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'

sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'

sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'

sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'

sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'

sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'

sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'

sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'

Searching for this error message brings nothing here, neither on… the rest of the Internet?

Any hints?

Falco · November 22, 2021, 5:40pm

Looks like an error related to the page compression? Can you share a link the to the page you trying to embed?

icaria36 · November 22, 2021, 6:47pm

Thank you for looking at this!

This is the external page: Bitwig Preset: Phase-3 | Bitwiggers

And this is the category where topics should appear: Bitwiggers.com - Bitwish

(We have removed the template after the failed tests.)

Does this comment help? (this is out of my tech league):

The responses are successful 200s, and based on that discussion it seems like it’s failing to handle that response while scraping Bitwiggers, and it’s specifically failing when trying to handle the compression.

Based on inflate , it seems like it negotiated to use the deflate compression scheme, whereas I think the preferred would be gzip . Firefox’s requests indicate that it would handle both, but the server decides on gzip in that case.

Falco · November 22, 2021, 8:41pm

Looks like if I send a request to your webserver as

curl 'https://bitwiggers.com/presets/d1a7c2c7-848d-425c-9058-993317cbcc9c/' -H 'accept-encoding: deflate, gzip' -vv

I get back a deflate compressed response, where most webservers when given the two options would prefer using gzip. That’s weird.

I see that I can easily reproduce this error with

require 'excon'
response = Excon.get('https://bitwiggers.com/presets/d1a7c2c7-848d-425c-9058-993317cbcc9c/',
 headers: {'accept-encoding' => 'deflate, gzip'})

# fails, default in excon
Zlib::Inflate.new(-Zlib::MAX_WBITS).inflate(response.body)

#works
Zlib::Inflate.new(0).inflate(response.body)

This code is over 8 years old in Excon, so I’d guess your webserver is at fault here, but I reported it upstream anyway

github.com/excon/excon

Excon fails to inflate specific page

opened 08:40PM - 22 Nov 21 UTC

xfalcox

Just got a bug report on discourse about an error that I tracked back to excon. … ``` require 'excon' response = Excon.get('https://bitwiggers.com/presets/d1a7c2c7-848d-425c-9058-993317cbcc9c/', headers: {'accept-encoding' => 'deflate, gzip'}) # fails, default in excon at Zlib::Inflate.new(-Zlib::MAX_WBITS).inflate(response.body) #works Zlib::Inflate.new(0).inflate(response.body) ``` The default inflate is set at https://github.com/excon/excon/blob/bbb5bd791d0bb2251593b80e3bce98dbec6e8f24/lib/excon/middlewares/decompress.rb#L23 I don't know if the webserver is providing an invalid response or if excon is at fault here. Any ideas?

icaria36 · November 22, 2021, 9:17pm

Interesting, and thank you for filing the bug. Subscribed.

Trying to summarize our thoughts:

We have no control over that handling, but we don’t think it’s the webserver. What about the construction of the headers instead? If passed in as gzip, deflate , then the response should be gzipped. According to HTTP Specifications, the client should present the options, weighted or not, in order according to their preference. so if the request is preferring deflate over gzip, then gets a deflate. Isn’t then the problem in the code/request, because it received an encoding that it asked for and failed to handle it?

Maybe the problem is related to this?

github.com

discourse/discourse/blob/53abcd825d2abbd2065796c974314f0952b26041/lib/final_destination.rb#L419

    
      
            nil
          end
          
          
protected
          
          
def safe_get(uri)
            result = nil
            unsafe_close = false
          
          
  safe_session(uri) do |http|
              headers = request_headers.merge(
                'Accept-Encoding' => 'gzip',
                'Host' => uri.host
              )
          
          
    req = Net::HTTP::Get.new(uri.request_uri, headers)
          
          
    http.request(req) do |resp|
                if Net::HTTPRedirection === resp
                  result = :redirect, [resp['location'], resp['Set-Cookie']]
                end

There is a “merge” and what seems to be an expectation for gzip?

The admin of the other server is looking at how can gzip be forced on their end. Still, it would be interesting to know your thoughts about this.

(For reference, the server is AWS API Gateway; not something we have control over, aside from enabling/disabling compression.)

icaria36 · November 28, 2021, 2:28pm

The developer has responded: Excon fails to inflate specific page · Issue #768 · excon/excon · GitHub

icaria36 · December 2, 2021, 9:58pm

Could some help translating the response from the Excon developer? I am not sure where is the problem and who could fix it.

I also wonder whether we could apply a local patch to our Discourse to circumvent this gzip/deflate problem. Local patches are painful, but it is even more painful to have an integration of two sites with hundreds of pages to be embedded with Discourse discussions pending because of this problem.

icaria36 · December 5, 2021, 7:24pm

The Excon maintainer has committed a patch that in theory solves this problem! A new Excon release should come soon, he says.

icaria36 · December 8, 2021, 8:58am

And now the Excon maintainer has published a new version of the library including this patch.

Does this mean that we can update our instance right away, or is Discourse using their own staged repository for these libraries? Remembering the output of Discourse upgrades I believe that they are being pulled directly from upstream repositories, but I’d rather ask before touching our backend.

icaria36 · December 10, 2021, 9:56pm

I decided to ask Discourse directly by upgrading and checking the logs. After seeing this

Installing excon 0.89.0
www/discourse/vendor/bundle/ruby/2.7.0/specifications/excon-0.89.0.gemspec

we have tried again and NOW EMBEDDING WORKS!

Thank you so much @Falco for your report upstream. You were absolutely right, and I added some noise that perhaps wasn’t that useful. My apologies, and thank you again.

system · January 9, 2022, 9:56pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Comments not embedding, 500 Error feature pr-welcome	15	1917	October 3, 2017
Embedding error support	13	3810	November 21, 2020
"Error Embedding" comments support	16	2114	May 4, 2022
Embedding topics stopped working bug	2	518	August 7, 2019
Cannot get embedding to work support	11	2354	August 31, 2021

JS embed fails with "Job exception: invalid stored block lengths (Zlib::DataError)"

Related Topics