Encoding conversion error from ASCII-8bit to UTF-8 in logs

There seems to be an issue with a specific web crawler; it should not affect you. :thinking:

Here, there is an issue with encoding the crawler user agent to UTF-8, as it contains an invalid byte.
scrub is supposed to do that. Maybe using :undef parameter can help here:
user_agent.encode("utf-8", :undef => :replace)

[51] pry(main)> string = "hello \xa1\x28world\x29".force_encoding("ASCII-8BIT")
=> "hello \xA1(world)"
[52] pry(main)> string.encode('utf-8')
Encoding::UndefinedConversionError: "\xA1" from ASCII-8BIT to UTF-8
from (pry):52:in `encode'
[53] pry(main)> string.encode('utf-8', :undef => :replace)
=> "hello �(world)"
4 Likes