# AI:embeddings:backfill - 处理 OpenAI 的 400 错误，因 Embeddings 中 Token 超限

**URL:** <https://meta.discourse.org/t/aibackfill-handling-openais-400-error-for-excessive-tokens-in-embeddings/292401>\
**Category:** Bug\
**Tags:** ai\
**Created:** [2024 年1 月 21 日 02:37 UTC](https://meta.discourse.org/t/aibackfill-handling-openais-400-error-for-excessive-tokens-in-embeddings/292401 "2024-01-21T02:37:55Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

作者： ![piffy](https://sea3.discourse-cdn.com/meta/user_avatar/meta.discourse.org/piffy/32/254198_2.png) [@piffy](https://meta.discourse.org/u/piffy)\
发布日期： [2024 年1 月 21 日 02:37 UTC](https://meta.discourse.org/t/aibackfill-handling-openais-400-error-for-excessive-tokens-in-embeddings/292401/1 "2024-01-21T02:37:55Z")

</div>

我正在尝试使用 OpenAI 嵌入来执行此 rake 任务：

> [@Discourse AI - Embeddings](https://meta.discourse.org/t/discourse-ai-embeddings/259603/8?u=piffy):
>
> You can also run rake ai:embeddings:backfill to generate embeddings for all topics eagerly.

> **我收到一条错误消息**
>
> [:/var/www/discourse# rake ai:embeddings:backfill --trace  
> \*\* Invoke ai:embeddings:backfill (first\_time)  
> \*\* Invoke environment (first\_time)  
> \*\* Execute environment  
> \*\* Execute ai:embeddings:backfill  
> .rake aborted!  
> Net::HTTPBadResponse: Net::HTTPBadResponse (Net::HTTPBadResponse)  
> /var/www/discourse/plugins/discourse-ai/lib/inference/open\_ai\_embeddings.rb:27:in `perform!’  
> /var/www/discourse/plugins/discourse-ai/lib/embeddings/vector\_representations/text\_embedding\_ada\_002.rb:36:in `vector\_from’  
> /var/www/discourse/plugins/discourse-ai/lib/embeddings/vector\_representations/base.rb:144:in `generate\_representation\_from’  
> /var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:19:in `block (2 levels) in ’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active\_record/relation/batches.rb:71:in `each’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active\_record/relation/batches.rb:71:in `block in find\_each’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active\_record/relation/batches.rb:138:in `block in find\_in\_batches’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active\_record/relation/batches.rb:245:in `block in in\_batches’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active\_record/relation/batches.rb:229:in `loop’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active\_record/relation/batches.rb:229:in `in\_batches’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active\_record/relation/batches.rb:137:in `find\_in\_batches’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/activerecord-7.0.8/lib/active\_record/relation/batches.rb:70:in `find\_each’  
> /var/www/discourse/plugins/discourse-ai/lib/tasks/modules/embeddings/database.rake:17:in `block in ’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `each’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:281:in `execute’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke\_with\_call\_chain’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke\_with\_call\_chain’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/task.rb:188:in `invoke’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:182:in `invoke\_task’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block (2 levels) in top\_level’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `each’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:138:in `block in top\_level’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:147:in `run\_with\_threads’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:132:in `top\_level’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:83:in `block in run’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:208:in `standard\_exception\_handling’  
> /var/www/discourse/vendor/bundle/ruby/3.2.0/gems/rake-13.1.0/lib/rake/application.rb:80:in `run’  
> bin/rake:13:in `\<top (required)\>’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `load’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `kernel\_load’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:23:in `run’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:451:in `exec’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke\_command’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:34:in `dispatch’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/cli.rb:28:in `start’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:28:in `block in \<top (required)\>’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/lib/bundler/friendly\_errors.rb:117:in `with\_friendly\_errors’  
> /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.5.3/exe/bundle:20:in `\<top (required)\>’  
> /usr/local/bin/bundle:25:in `load’  
> /usr/local/bin/bundle:25:in `’  
> Tasks: TOP =\> ai:embeddings:backfill

你不必阅读它，我已经弄清楚问题所在了。

`HTTPBadResponse` 是从下面这行抛出的：

> <https://github.com/discourse/discourse-ai/blob/4b4aedb50f7d0a3e2ee06c4781e7b52fceeba121/lib/inference/open_ai_embeddings.rb#L27>

`/logs` 页面显示这个：

```json
OpenAI Embeddings failed with status: 400 body: {
  "error": {
    "message": "This model's maximum context length is 8192 tokens, however you requested 8506 tokens (8506 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

```

基本上，似乎有什么东西的单词太多了。我不确定在这种情况下“prompt”和“completion”之间的区别。无论如何，这阻止了我进行回填。

我已经更改了站点设置中的最大帖子限制，所以也许是由一个非常长的帖子引起的？但如果是这样，我期望这个帖子的内容被截断，或者这个帖子应该被跳过？无论如何，它完全阻止了回填过程。

---

_[View the full topic](https://meta.discourse.org/t/aibackfill-handling-openais-400-error-for-excessive-tokens-in-embeddings/292401)._