JS embed fails with "Job exception: invalid stored block lengths (Zlib::DataError)"

Hi, we cannot get JS embed to work in our setup. I have read all the other similar topics and I have got this embedding to work in a total different project in the past. This one looks like something different…

On the external site, only our logo can be seen followed by “loading discussion…”

On Discourse, no topics are being created, even if the error log shows that the requests are coming.

We have reviewed the URLs. We have also tested adding a static URL (instead of a variable). Nothing.

Discourse error log has this Error:

Job exception: invalid stored block lengths (Zlib::DataError)  
excon-0.88.0/lib/excon/middlewares/decompress.rb:23:in `inflate'

excon-0.88.0/lib/excon/middlewares/decompress.rb:23:in `response_call'

excon-0.88.0/lib/excon/middlewares/base.rb:26:in `response_call'

excon-0.88.0/lib/excon/middlewares/instrumentor.rb:44:in `response_call'

excon-0.88.0/lib/excon/middlewares/base.rb:26:in `response_call'

excon-0.88.0/lib/excon/middlewares/expects.rb:20:in `response_call'

excon-0.88.0/lib/excon/middlewares/response_parser.rb:12:in `response_call'

excon-0.88.0/lib/excon/connection.rb:451:in `response'

excon-0.88.0/lib/excon/connection.rb:282:in `request'

excon-0.88.0/lib/excon.rb:250:in `get'

/var/www/discourse/lib/final_destination.rb:206:in `public_send'

/var/www/discourse/lib/final_destination.rb:206:in `resolve'

/var/www/discourse/app/models/topic_embed.rb:120:in `find_remote'

/var/www/discourse/app/models/topic_embed.rb:192:in `import_remote'

/var/www/discourse/lib/topic_retriever.rb:52:in `fetch_http'

/var/www/discourse/lib/topic_retriever.rb:39:in `perform_retrieve'

/var/www/discourse/lib/topic_retriever.rb:12:in `retrieve'

/var/www/discourse/app/jobs/regular/retrieve_topic.rb:15:in `execute'

/var/www/discourse/app/jobs/base.rb:232:in `block (2 levels) in perform'

rails_multisite-4.0.0/lib/rails_multisite/connection_management.rb:80:in `with_connection'

/var/www/discourse/app/jobs/base.rb:221:in `block in perform'

/var/www/discourse/app/jobs/base.rb:217:in `each'

/var/www/discourse/app/jobs/base.rb:217:in `perform'

sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'

sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:138:in `block in invoke'

/var/www/discourse/lib/sidekiq/pausable.rb:138:in `call'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:140:in `block in invoke'

sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:143:in `invoke'

sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'

sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'

sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'

sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'

sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'

sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'

sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'

sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'

sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'

sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'

sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'

sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'

sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'

sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'

sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'

sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'

Searching for this error message brings nothing here, neither on… the rest of the Internet?

Any hints?

1 Like

Looks like an error related to the page compression? Can you share a link the to the page you trying to embed?

2 Likes

Thank you for looking at this!

This is the external page: Bitwig Preset: Phase-3 | Bitwiggers

And this is the category where topics should appear: Bitwiggers - Bitwish

(We have removed the template after the failed tests.)

Does this comment help? (this is out of my tech league):

The responses are successful 200s, and based on that discussion it seems like it’s failing to handle that response while scraping Bitwiggers, and it’s specifically failing when trying to handle the compression.

Based on inflate , it seems like it negotiated to use the deflate compression scheme, whereas I think the preferred would be gzip . Firefox’s requests indicate that it would handle both, but the server decides on gzip in that case.

1 Like

Looks like if I send a request to your webserver as

curl 'https://bitwiggers.com/presets/d1a7c2c7-848d-425c-9058-993317cbcc9c/' -H 'accept-encoding: deflate, gzip' -vv

I get back a deflate compressed response, where most webservers when given the two options would prefer using gzip. That’s weird.

I see that I can easily reproduce this error with

require 'excon'
response = Excon.get('https://bitwiggers.com/presets/d1a7c2c7-848d-425c-9058-993317cbcc9c/',
 headers: {'accept-encoding' => 'deflate, gzip'})

# fails, default in excon
Zlib::Inflate.new(-Zlib::MAX_WBITS).inflate(response.body)

#works
Zlib::Inflate.new(0).inflate(response.body)

This code is over 8 years old in Excon, so I’d guess your webserver is at fault here, but I reported it upstream anyway

2 Likes

Interesting, and thank you for filing the bug. Subscribed.

Trying to summarize our thoughts:

We have no control over that handling, but we don’t think it’s the webserver. What about the construction of the headers instead? If passed in as gzip, deflate , then the response should be gzipped. According to HTTP Specifications, the client should present the options, weighted or not, in order according to their preference. so if the request is preferring deflate over gzip, then gets a deflate. Isn’t then the problem in the code/request, because it received an encoding that it asked for and failed to handle it?

Maybe the problem is related to this?

There is a “merge” and what seems to be an expectation for gzip?

The admin of the other server is looking at how can gzip be forced on their end. Still, it would be interesting to know your thoughts about this.

(For reference, the server is AWS API Gateway; not something we have control over, aside from enabling/disabling compression.)

1 Like

The developer has responded: Excon fails to inflate specific page · Issue #768 · excon/excon · GitHub

1 Like

Could some help translating the response from the Excon developer? I am not sure where is the problem and who could fix it.

I also wonder whether we could apply a local patch to our Discourse to circumvent this gzip/deflate problem. Local patches are painful, but it is even more painful to have an integration of two sites with hundreds of pages to be embedded with Discourse discussions pending because of this problem.

The Excon maintainer has committed a patch that in theory solves this problem! :tada: A new Excon release should come soon, he says.

1 Like