# Ошибка конвертации кодировки из ASCII-8bit в UTF-8 в логах

**URL:** <https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-utf-8-in-logs/308603>\
**Category:** Bug\
**Tags:** fixed\
**Created:** [19.Май.2024 13:27:23 UTC](https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-utf-8-in-logs/308603 "2024-05-19T13:27:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![RBoy](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@RBoy](https://meta.discourse.org/u/RBoy)\
**Post date:** [19.Май.2024 13:27:23 UTC](https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-utf-8-in-logs/308603/1 "2024-05-19T13:27:23Z")

</div>

В логах я вижу это предупреждение, но не уверен, откуда оно берется. Раньше такого не встречал, началось после обновления с версии 3.2.0 до 3.3.0.

Это единственное предупреждение, связанное с временными метками, других ошибок или предупреждений нет (сначала думал, что дело в почте, но в это время не было ни ошибок, ни отказов в доставке писем).

#### Установлено

### 3.3.0.beta2-dev

([777b8f6d51](https://github.com/discourse/discourse/commits/777b8f6d51b7a5a080f52f4fa0a1d7e487735e51))

```plaintext
Сообщение (4 сообщения)

Ошибка в RequestTracker.get_data: Encoding::UndefinedConversionError: "\xA1" из ASCII-8BIT в UTF-8

Трассировка стека

/var/www/discourse/lib/middleware/request_tracker.rb:190:in `encode'
/var/www/discourse/lib/middleware/request_tracker.rb:190:in `get_data'
/var/www/discourse/lib/middleware/request_tracker.rb:207:in `log_request_info'
/var/www/discourse/lib/middleware/request_tracker.rb:320:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/engine.rb:530:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `public_send'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/railties-7.0.8.1/lib/rails/railtie.rb:226:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:74:in `block in call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-2.2.9/lib/rack/urlmap.rb:58:in `call'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:634:in `process_client'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `load'
/var/www/discourse/vendor/bundle/ruby/3.2.0/bin/unicorn:25:in `<main>'

```

---

<div class="post-metadata">

**Author:** ![Arkshine](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/arkshine/32/298682_2.png) [@Arkshine](https://meta.discourse.org/u/Arkshine)\
**Post date:** [19.Май.2024 15:13:59 UTC](https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-utf-8-in-logs/308603/2 "2024-05-19T15:13:59Z")

</div>

Похоже, возникла проблема с конкретным веб-краулером; это не должно затронуть вас. 🤔

> <https://github.com/discourse/discourse/blob/main/lib/middleware/request_tracker.rb#L187-L194>

Здесь возникает проблема с кодированием строки user agent краулера в UTF-8, так как она содержит недопустимый байт.  
Метод `scrub` должен это исправить. Возможно, использование параметра [`:undef`](https://apidock.com/ruby/String/encode) поможет здесь:  
`user_agent.encode("utf-8", :undef => :replace)`

![image](https://cdck-file-uploads-global.s3.dualstack.us-west-2.amazonaws.com/meta/original/4X/a/b/2/ab29fd5f818f4849b0534bd1a8e753369d6bd52c.png)

```ruby
[51] pry(main)> string = "hello \xa1\x28world\x29".force_encoding("ASCII-8BIT")
=> "hello \xA1(world)"
[52] pry(main)> string.encode('utf-8')
Encoding::UndefinedConversionError: "\xA1" from ASCII-8BIT to UTF-8
from (pry):52:in `encode'
[53] pry(main)> string.encode('utf-8', :undef => :replace)
=> "hello �(world)"

```

---

<div class="post-metadata">

**Author:** ![sam](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/sam/32/102149_2.png) [@sam](https://meta.discourse.org/u/sam)\
**Post date:** [22.Май.2024 05:00:01 UTC](https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-utf-8-in-logs/308603/3 "2024-05-22T05:00:01Z")

</div>

Выглядит хорошо, есть PR?

---

<div class="post-metadata">

**Author:** ![Arkshine](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/arkshine/32/298682_2.png) [@Arkshine](https://meta.discourse.org/u/Arkshine)\
**Post date:** [11.Июнь.2024 15:14:09 UTC](https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-utf-8-in-logs/308603/4 "2024-06-11T15:14:09Z")

</div>

Исправление объединено. 👍

> <https://github.com/discourse/discourse/pull/27181>
>
> Meta: https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-…utf-8-in-logs/308603/2
> 
> Crawler requests for non-UTF-8 user agents that contain invalid bytes generate an exception at two places.
> See \[\`get\_data()\`\](https://github.com/discourse/discourse/blob/main/lib/middleware/request\_tracker.rb#L158-L193) function:
> \* On \`encode("utf-8")\` that results either in the following error depending on the incoming encoding
> \* \`InvalidByteSequenceError\`
> \* \`UndefinedConversionError\`
> \* On matching user-agent with invalid byte results to \`ArgumentError\`.
> \_Called from \`helper.is\_crawler\` and \`helper.is\_mobile\`, part of the \`AnonymousCache::Helper\` class.\_
> 
> This PR does the following:
> \* Handles \`encode()\` exceptions by relying on \`undef\` and \`invalid\` params to replace the faulty bytes instead of raising an exception. It moved into its own module.
> \* Provides a safe user agent in \`AnonymousCache::Helper.\`
> 
> The \`anonymous\_cache\_spec.rb\` tests are specifically for the methods: \`blocked\_crawler?\`, \`key\_is\_modern\_mobile\_device?\`, and \`key\_is\_old\_browser?\`.
> 
> Hopefully, the implementation is okay.

---

<div class="post-metadata">

**Author:** ![Lilly](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/lilly/32/575047_2.png) [@Lilly](https://meta.discourse.org/u/Lilly)\
**Post date:** [11.Июнь.2024 15:27:12 UTC](https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-utf-8-in-logs/308603/5 "2024-06-11T15:27:12Z")

</div>

Отличная работа! 👏 😃 🐉

---

<div class="post-metadata">

**Author:** ![JammyDodger](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/jammydodger/32/254611_2.png) [@JammyDodger](https://meta.discourse.org/u/JammyDodger)\
**Post date:** [14.Июнь.2024 15:45:54 UTC](https://meta.discourse.org/t/encoding-conversion-error-from-ascii-8bit-to-utf-8-in-logs/308603/8 "2024-06-14T15:45:54Z")

</div>

Эта тема была автоматически закрыта через 3 дня. Новые ответы больше не принимаются.