Encoding conversion error from ASCII-8bit to UTF-8 in logs

RBoy · May 19, 2024, 1:27pm

I’m seeing this warning in the logs, not sure where it’s coming from. Never seen it before, started after upgrading from 3.2.0 to 3.3.0.

It’s the sole warning around the timestamps, no other errors/warnings (initially thought it was email related, but not seeing any errors or bounced emails around the time).

Installed

3.3.0.beta2-dev

(777b8f6d51)

Message (4 copies reported)

RequestTracker.get_data failed : Encoding::UndefinedConversionError : "\xA1" from ASCII-8BIT to UTF-8

Backtrace

/var/www/discourse/lib/middleware/request_tracker.rb:190:in `encode'
/var/www/discourse/lib/middleware/request_tracker.rb:190:in `get_data'
/var/www/discourse/lib/middleware/request_tracker.rb:207:in `log_request_info'
/var/www/discourse/lib/middleware/request_tracker.rb:320:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/engine.rb:530:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `public_send'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:634:in `process_client'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'

Arkshine · May 19, 2024, 3:13pm

There seems to be an issue with a specific web crawler; it should not affect you.

github.com

discourse/discourse/blob/main/lib/middleware/request_tracker.rb#L187-L194


      
          if h[:is_crawler]
            user_agent = env["HTTP_USER_AGENT"]
            if user_agent && (user_agent.encoding != Encoding::UTF_8)
              user_agent = user_agent.encode("utf-8")
              user_agent.scrub!
            end
            h[:user_agent] = user_agent
          end

Here, there is an issue with encoding the crawler user agent to UTF-8, as it contains an invalid byte.
scrub is supposed to do that. Maybe using :undef parameter can help here:
user_agent.encode("utf-8", :undef => :replace)

[51] pry(main)> string = "hello \xa1\x28world\x29".force_encoding("ASCII-8BIT")
=> "hello \xA1(world)"
[52] pry(main)> string.encode('utf-8')
Encoding::UndefinedConversionError: "\xA1" from ASCII-8BIT to UTF-8
from (pry):52:in `encode'
[53] pry(main)> string.encode('utf-8', :undef => :replace)
=> "hello �(world)"

sam · May 22, 2024, 5:00am

Looks good, a PR ?

Arkshine · June 11, 2024, 3:14pm

A fix has been merged.

github.com/discourse/discourse

FIX: Crawler requests exceptions for non UTF-8 user agents with invalid bytes

discourse:main ← Arkshine:fix-crowler-useragent-encoding

opened 04:29PM - 24 May 24 UTC

Arkshine

+86 -14

Meta: https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-…utf-8-in-logs/308603/2 Crawler requests for non-UTF-8 user agents that contain invalid bytes generate an exception at two places. See [`get_data()`](https://github.com/discourse/discourse/blob/main/lib/middleware/request_tracker.rb#L158-L193) function: * On `encode("utf-8")` that results either in the following error depending on the incoming encoding * `InvalidByteSequenceError` * `UndefinedConversionError` * On matching user-agent with invalid byte results to `ArgumentError`. _Called from `helper.is_crawler` and `helper.is_mobile`, part of the `AnonymousCache::Helper` class._ This PR does the following: * Handles `encode()` exceptions by relying on `undef` and `invalid` params to replace the faulty bytes instead of raising an exception. It moved into its own module. * Provides a safe user agent in `AnonymousCache::Helper.` The `anonymous_cache_spec.rb` tests are specifically for the methods: `blocked_crawler?`, `key_is_modern_mobile_device?`, and `key_is_old_browser?`. Hopefully, the implementation is okay.

Lilly · June 11, 2024, 3:27pm

Nice work!

JammyDodger · June 14, 2024, 3:45pm

This topic was automatically closed after 3 days. New replies are no longer allowed.

Topic		Replies	Views
Octal string exception while importing database Support	3	566	June 19, 2020
Incoming email details 500 error Bug	4	837	January 31, 2018
Embed poll_feed fail with UTF8 title Bug	6	1528	July 21, 2022
Facebook login not working? Bug	2	1932	March 28, 2017
Translation data errors (pending users template, other) Bug	3	1269	January 1, 2016

Encoding conversion error from ASCII-8bit to UTF-8 in logs

Installed

3.3.0.beta2-dev

Related topics