Encoding conversion error from ASCII-8bit to UTF-8 in logs

Contribute Bug

AI-generated summary

A user reported an encoding error in the logs after upgrading Discourse from 3.2.0 to 3.3.0, specifically Encoding::UndefinedConversionError : "\xA1" from ASCII-8BIT to UTF-8 in RequestTracker.get_data. The issue was traced to a web crawler sending a malformed user agent string containing invalid bytes. Arkshine identified the root cause in request_tracker.rb and proposed using :undef => :replace in the .encode call to handle invalid characters gracefully. sam approved the fix, and Arkshine confirmed the fix was merged via PR #27181. Lilly praised the resolution. The bug was non-critical and only affected log output, not core functionality.

1

I’m seeing this warning in the logs, not sure where it’s coming from. Never seen it before, started after upgrading from 3.2.0 to 3.3.0.

It’s the sole warning around the timestamps, no other errors/warnings (initially thought it was email related, but not seeing any errors or bounced emails around the time).

Installed

3.3.0.beta2-dev

(777b8f6d51)

Message (4 copies reported)

RequestTracker.get_data failed : Encoding::UndefinedConversionError : "\xA1" from ASCII-8BIT to UTF-8

Backtrace

/var/www/discourse/lib/middleware/request_tracker.rb:190:in `encode'
/var/www/discourse/lib/middleware/request_tracker.rb:190:in `get_data'
/var/www/discourse/lib/middleware/request_tracker.rb:207:in `log_request_info'
/var/www/discourse/lib/middleware/request_tracker.rb:320:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/engine.rb:530:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `public_send'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:634:in `process_client'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'
2

There seems to be an issue with a specific web crawler; it should not affect you. :thinking:

Here, there is an issue with encoding the crawler user agent to UTF-8, as it contains an invalid byte.
scrub is supposed to do that. Maybe using :undef parameter can help here:
user_agent.encode("utf-8", :undef => :replace)

image

[51] pry(main)> string = "hello \xa1\x28world\x29".force_encoding("ASCII-8BIT")
=> "hello \xA1(world)"
[52] pry(main)> string.encode('utf-8')
Encoding::UndefinedConversionError: "\xA1" from ASCII-8BIT to UTF-8
from (pry):52:in `encode'
[53] pry(main)> string.encode('utf-8', :undef => :replace)
=> "hello �(world)"
4 Likes
3

Looks good, a PR ?

2 Likes
4

A fix has been merged. :+1:

https://github.com/discourse/discourse/pull/27181

4 Likes
5

Nice work! :clap: :smiley: :dragon:

2 Likes
8

This topic was automatically closed after 3 days. New replies are no longer allowed.