Erreur de conversion d'encodage de ASCII-8bit à UTF-8 dans les journaux

RBoy · Mai 19, 2024, 1:27

I’m seeing this warning in the logs, not sure where it’s coming from. Never seen it before, started after upgrading from 3.2.0 to 3.3.0.

It’s the sole warning around the timestamps, no other errors/warnings (initially thought it was email related, but not seeing any errors or bounced emails around the time).

Installed

3.3.0.beta2-dev

(777b8f6d51)

Message (4 copies reported)

RequestTracker.get_data failed : Encoding::UndefinedConversionError : "\xA1" from ASCII-8BIT to UTF-8

Backtrace

/var/www/discourse/lib/middleware/request_tracker.rb:190:in `encode'
/var/www/discourse/lib/middleware/request_tracker.rb:190:in `get_data'
/var/www/discourse/lib/middleware/request_tracker.rb:207:in `log_request_info'
/var/www/discourse/lib/middleware/request_tracker.rb:320:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/engine.rb:530:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `public_send'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:634:in `process_client'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'

Arkshine · Mai 19, 2024, 3:13

There seems to be an issue with a specific web crawler; it should not affect you.

github.com

discourse/discourse/blob/main/lib/middleware/request_tracker.rb#L187-L194


      
          if h[:is_crawler]
            user_agent = env["HTTP_USER_AGENT"]
            if user_agent && (user_agent.encoding != Encoding::UTF_8)
              user_agent = user_agent.encode("utf-8")
              user_agent.scrub!
            end
            h[:user_agent] = user_agent
          end

Here, there is an issue with encoding the crawler user agent to UTF-8, as it contains an invalid byte.
scrub is supposed to do that. Maybe using :undef parameter can help here:
user_agent.encode("utf-8", :undef => :replace)

[51] pry(main)> string = "hello \xa1\x28world\x29".force_encoding("ASCII-8BIT")
=> "hello \xA1(world)"
[52] pry(main)> string.encode('utf-8')
Encoding::UndefinedConversionError: "\xA1" from ASCII-8BIT to UTF-8
from (pry):52:in `encode'
[53] pry(main)> string.encode('utf-8', :undef => :replace)
=> "hello �(world)"

sam · Mai 22, 2024, 5:00

Looks good, a PR ?

Arkshine · Juin 11, 2024, 3:14

A fix has been merged.

github.com/discourse/discourse

FIX: Crawler requests exceptions for non UTF-8 user agents with invalid bytes

discourse:main ← Arkshine:fix-crowler-useragent-encoding

opened 04:29PM - 24 May 24 UTC

Arkshine

+86 -14

Meta: https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-…utf-8-in-logs/308603/2 Crawler requests for non-UTF-8 user agents that contain invalid bytes generate an exception at two places. See [`get_data()`](https://github.com/discourse/discourse/blob/main/lib/middleware/request_tracker.rb#L158-L193) function: * On `encode("utf-8")` that results either in the following error depending on the incoming encoding * `InvalidByteSequenceError` * `UndefinedConversionError` * On matching user-agent with invalid byte results to `ArgumentError`. _Called from `helper.is_crawler` and `helper.is_mobile`, part of the `AnonymousCache::Helper` class._ This PR does the following: * Handles `encode()` exceptions by relying on `undef` and `invalid` params to replace the faulty bytes instead of raising an exception. It moved into its own module. * Provides a safe user agent in `AnonymousCache::Helper.` The `anonymous_cache_spec.rb` tests are specifically for the methods: `blocked_crawler?`, `key_is_modern_mobile_device?`, and `key_is_old_browser?`. Hopefully, the implementation is okay.

Lilly · Juin 11, 2024, 3:27

Nice work!

JammyDodger · Juin 14, 2024, 3:45

This topic was automatically closed after 3 days. New replies are no longer allowed.

Sujet		Réponses	Vues
Weird encoding issue on categories page Support unsupported-install	15	163	Février 5, 2025
Error importing from vanilla: invalid byte sequence in UTF-8 Migration	25	2128	Octobre 18, 2023
Automatic encoding of parsed URL params Dev	43	8077	Mai 15, 2025
Tags does not work with Cyrillic Bug	10	2199	Août 1, 2017
"Sorry, an error has occurred" when saving user preferences Bug	12	1702	Février 5, 2015

Erreur de conversion d'encodage de ASCII-8bit à UTF-8 dans les journaux

Installed

3.3.0.beta2-dev

Sujets connexes