AI 임베딩 백필 레이크 중단됨

[Don]

안녕하세요

Discourse Ai - OpenAi API를 사용하기 시작했습니다. 처음에는 관련 주제(Related Topics)에 text-embedding-ada-002를 설정했습니다. 백필을 위한 rake 작업을 실행하려고 하니 다음과 같은 오류가 발생했습니다.

....rake aborted!
NoMethodError: undefined method `[]' for nil:NilClass (NoMethodError)

          response[:data].first[:embedding]
                  ^^^^^^^
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/text_embedding_ada_002.rb:37:in `vector_from'
/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:145:in `generate_representation_from'
/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:19:in `block (2 levels) in <main>'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:71:in `each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:71:in `block in find_each'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:229:in `loop'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:229:in `in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:70:in `find_each'
/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:17:in `block in <main>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
Tasks: TOP => ai:embeddings:backfill

또한 /logs에서 다음 내용도 확인할 수 있습니다.

도움 주셔서 감사합니다!

[piffy]

nil 응답을 받는 것 같습니다.

ai embeddings enabled를 확인하고 설정에 API 키를 추가해 보셨나요?

[Don]

네, 설정이 이렇게 보입니다.

Jobs::EmbeddingsBackfill은 실행 중인 것으로 보이지만, OP에서 언급한 오류로 인해 항상 실패합니다.

OpenAi와 연결되어 있는 것 같습니다.

[Don]

문제가 레이트 리밋(rate limit) 때문인 것 같습니다. OpenAI에서 티어를 업그레이드하고 모델을 text-embedding-3-large로 변경했습니다. 이제 rake 실행 중에 점(dot)이 많이 표시되고 있는데, 이는 좋은 징조입니다. 완료되면 이 게시물을 업데이트하겠습니다.

수정 1: 10시간 동안 점 표시가 계속되다가 rake가 중단(aborted)되었습니다

rake aborted!

Faraday::TimeoutError: Net::ReadTimeout with #<TCPSocket:(closed)> (Faraday::TimeoutError)

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:229:in `rbuf_fill'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:199:in `readuntil'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:209:in `readline'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http/response.rb:158:in `read_status_line'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http/response.rb:147:in `read_new'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:2342:in `block in transport_request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:2333:in `catch'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:2333:in `transport_request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:2306:in `request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.0/lib/patches/net_patches.rb:19:in `block in request_with_mini_profiler'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.0/lib/mini_profiler/profiling_methods.rb:50:in `step'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.0/lib/patches/net_patches.rb:18:in `request_with_mini_profiler'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:113:in `block in request_with_wrapped_block'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:1570:in `start'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:112:in `request_with_wrapped_block'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:102:in `perform_request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:66:in `block in call'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/adapter.rb:45:in `connection'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:65:in `call'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/request/url_encoded.rb:25:in `call'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/rack_builder.rb:152:in `build_response'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/connection.rb:444:in `run_request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/connection.rb:280:in `post'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday.rb:145:in `method_missing'

/var/www/discourse/plugins/discourse-ai/lib/inference/open_ai_embeddings.rb:18:in `perform!'

/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/text_embedding_3_large.rb:39:in `vector_from'

/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:147:in `generate_representation_from'

/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:30:in `block (2 levels) in <main>'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:71:in `each'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:71:in `block in find_each'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:229:in `loop'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:229:in `in_batches'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:70:in `find_each'

/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:28:in `block in <main>'

/usr/local/bin/bundle:25:in `load'

/usr/local/bin/bundle:25:in `<main>'

Caused by:

Net::ReadTimeout: Net::ReadTimeout with #<TCPSocket:(closed)> (Net::ReadTimeout)

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:229:in `rbuf_fill'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:199:in `readuntil'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-protocol-0.2.2/lib/net/protocol.rb:209:in `readline'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http/response.rb:158:in `read_status_line'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http/response.rb:147:in `read_new'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:2342:in `block in transport_request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:2333:in `catch'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:2333:in `transport_request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:2306:in `request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.0/lib/patches/net_patches.rb:19:in `block in request_with_mini_profiler'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.0/lib/mini_profiler/profiling_methods.rb:50:in `step'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rack-mini-profiler-3.3.0/lib/patches/net_patches.rb:18:in `request_with_mini_profiler'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:113:in `block in request_with_wrapped_block'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/net-http-0.4.1/lib/net/http.rb:1570:in `start'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:112:in `request_with_wrapped_block'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:102:in `perform_request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:66:in `block in call'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/adapter.rb:45:in `connection'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-net_http-3.1.0/lib/faraday/adapter/net_http.rb:65:in `call'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/request/url_encoded.rb:25:in `call'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/rack_builder.rb:152:in `build_response'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/connection.rb:444:in `run_request'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday/connection.rb:280:in `post'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/faraday-2.9.0/lib/faraday.rb:145:in `method_missing'

/var/www/discourse/plugins/discourse-ai/lib/inference/open_ai_embeddings.rb:18:in `perform!'

/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/text_embedding_3_large.rb:39:in `vector_from'

/var/www/discourse/plugins/discourse-ai/lib/embeddings/vector_representations/base.rb:147:in `generate_representation_from'

/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:30:in `block (2 levels) in <main>'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:71:in `each'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:71:in `block in find_each'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:245:in `block in in_batches'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:229:in `loop'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:229:in `in_batches'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:137:in `find_in_batches'

/var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active_record/relation/batches.rb:70:in `find_each'

/var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:28:in `block in <main>'

/usr/local/bin/bundle:25:in `load'

/usr/local/bin/bundle:25:in `<main>'

Tasks: TOP => ai:embeddings:backfill

(See full trace by running task with --trace)

rake를 다시 시작했습니다. 이번에는 처리되지 않은 주제만 처리할까요, 아니면 다시 모든 주제를 처리할까요?

수정 2: rake가 동일한 오류로 다시 중단되었습니다. 하지만 이 프로세스들이 대부분의 주제를 처리했을 것 같고, Sidekiq의 Jobs::EmbeddingsBackfill도 이제 정상적으로 실행되는 것 같습니다. 따라서 나머지 주제에 대해서는 이것만으로 충분할 것 같습니다.