Erro de conversão de codificação de ASCII-8bit para UTF-8 nos logs

RBoy · Maio 19, 2024, 1:27pm

I’m seeing this warning in the logs, not sure where it’s coming from. Never seen it before, started after upgrading from 3.2.0 to 3.3.0.

It’s the sole warning around the timestamps, no other errors/warnings (initially thought it was email related, but not seeing any errors or bounced emails around the time).

Installed

3.3.0.beta2-dev

(777b8f6d51)

Message (4 copies reported)

RequestTracker.get_data failed : Encoding::UndefinedConversionError : "\xA1" from ASCII-8BIT to UTF-8

Backtrace

/var/www/discourse/lib/middleware/request_tracker.rb:190:in `encode'
/var/www/discourse/lib/middleware/request_tracker.rb:190:in `get_data'
/var/www/discourse/lib/middleware/request_tracker.rb:207:in `log_request_info'
/var/www/discourse/lib/middleware/request_tracker.rb:320:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/engine.rb:530:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `public_send'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:634:in `process_client'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'

Arkshine · Maio 19, 2024, 3:13pm

There seems to be an issue with a specific web crawler; it should not affect you.

github.com

discourse/discourse/blob/main/lib/middleware/request_tracker.rb#L187-L194


      
          if h[:is_crawler]
            user_agent = env["HTTP_USER_AGENT"]
            if user_agent && (user_agent.encoding != Encoding::UTF_8)
              user_agent = user_agent.encode("utf-8")
              user_agent.scrub!
            end
            h[:user_agent] = user_agent
          end

Here, there is an issue with encoding the crawler user agent to UTF-8, as it contains an invalid byte.
scrub is supposed to do that. Maybe using :undef parameter can help here:
user_agent.encode("utf-8", :undef => :replace)

[51] pry(main)> string = "hello \xa1\x28world\x29".force_encoding("ASCII-8BIT")
=> "hello \xA1(world)"
[52] pry(main)> string.encode('utf-8')
Encoding::UndefinedConversionError: "\xA1" from ASCII-8BIT to UTF-8
from (pry):52:in `encode'
[53] pry(main)> string.encode('utf-8', :undef => :replace)
=> "hello �(world)"

sam · Maio 22, 2024, 5:00am

Looks good, a PR ?

Arkshine · Junho 11, 2024, 3:14pm

A fix has been merged.

github.com/discourse/discourse

FIX: Crawler requests exceptions for non UTF-8 user agents with invalid bytes

discourse:main ← Arkshine:fix-crowler-useragent-encoding

opened 04:29PM - 24 May 24 UTC

Arkshine

+86 -14

Meta: https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-…utf-8-in-logs/308603/2 Crawler requests for non-UTF-8 user agents that contain invalid bytes generate an exception at two places. See [`get_data()`](https://github.com/discourse/discourse/blob/main/lib/middleware/request_tracker.rb#L158-L193) function: * On `encode("utf-8")` that results either in the following error depending on the incoming encoding * `InvalidByteSequenceError` * `UndefinedConversionError` * On matching user-agent with invalid byte results to `ArgumentError`. _Called from `helper.is_crawler` and `helper.is_mobile`, part of the `AnonymousCache::Helper` class._ This PR does the following: * Handles `encode()` exceptions by relying on `undef` and `invalid` params to replace the faulty bytes instead of raising an exception. It moved into its own module. * Provides a safe user agent in `AnonymousCache::Helper.` The `anonymous_cache_spec.rb` tests are specifically for the methods: `blocked_crawler?`, `key_is_modern_mobile_device?`, and `key_is_old_browser?`. Hopefully, the implementation is okay.

Lilly · Junho 11, 2024, 3:27pm

Nice work!

JammyDodger · Junho 14, 2024, 3:45pm

This topic was automatically closed after 3 days. New replies are no longer allowed.

Tópico		Respostas	Visualizações
Weird encoding issue on categories page Support unsupported-install	15	164	5 de Fevereiro de 2025
Error importing from vanilla: invalid byte sequence in UTF-8 Migration	25	2128	18 de Outubro de 2023
Automatic encoding of parsed URL params Dev	43	8081	15 de Maio de 2025
Tags does not work with Cyrillic Bug	10	2199	1 de Agosto de 2017
"Sorry, an error has occurred" when saving user preferences Bug	12	1702	5 de Fevereiro de 2015

Erro de conversão de codificação de ASCII-8bit para UTF-8 nos logs

Installed

3.3.0.beta2-dev

Tópicos relacionados