### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 14, 2025, 3:51pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/1 "2025-01-14T15:51:39Z")
I’m trying to backfill the last 90 days. It successfully backfilled about 500 out of 2000 topics, but now progress is stalled because it keeps processing the same topic every time the job runs (every 5 minutes). No idea why- the topic already has a valid summary, and has no new posts in the past 12 days. AI Audit Logs table shows each request is successful. Sidekiq status is OK. Nothing relevant in /logs. How to debug this?
```plaintext
SELECT request_tokens,
response_tokens,
raw_response_payload,
topic_id,
created_at AS summary_created_at,
language_model
FROM ai_api_audit_logs
WHERE raw_request_payload LIKE '%Franklin Lexington%'
ORDER BY created_at DESC
LIMIT 40
```
All of the raw response payloads look valid. Each is slightly different as expected. Here’s an example (I’ve redacted most of the content text):
```plaintext
{
"id": "msg_016C2dHZik2Miwe16pRHFe9z",
"type": "message",
"role": "assistant",
"model": "claude-3-5-sonnet-20241022",
"content": [
{
"type": "text",
"text": "The Franklin Lexington Private Markets Fund (FLEX) is a new private equity secondaries fund... [REDACTED] "
}
],
"stop_reason": "end_turn",
"stop_sequence": null,
"usage": {
"input_tokens": 13848,
"cache_creation_input_tokens": 0,
"cache_read_input_tokens": 0,
"output_tokens": 416
}
}
```
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 14, 2025, 4:52pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/2 "2025-01-14T16:52:11Z")
I don’t know if this is a clue, but does the 103 mean it thinks it has only summarized through post 103, and there are 108 posts, so it needs a new summary?
Query generated by SQL Helper:
```plaintext
-- [params]
-- integer :summary_type = 0
-- integer :max_age_days = 30
-- integer :min_word_count = 100
WITH topic_candidates AS (
SELECT
t.id as topic_id,
t.title,
t.created_at,
t.word_count,
t.highest_post_number,
t.last_posted_at,
ais.id as summary_id,
ais.content_range,
ais.created_at as summary_created_at,
UPPER(ais.content_range) as content_range_upper
FROM topics t
LEFT OUTER JOIN ai_summaries ais ON
t.id = ais.target_id AND
ais.target_type = 'Topic' AND
ais.summary_type = :summary_type
WHERE
-- Word count threshold
t.word_count >= :min_word_count
-- Age restriction
AND t.updated_at > CURRENT_TIMESTAMP - (:max_age_days || ' DAYS')::INTERVAL
-- Either no summary exists OR summary needs updating
AND (
ais.id IS NULL OR
(
UPPER(ais.content_range) < t.highest_post_number + 1
AND ais.created_at < (CURRENT_TIMESTAMP - INTERVAL '5 minutes')
)
)
)
SELECT
topic_id,
title,
word_count,
highest_post_number,
created_at,
last_posted_at,
summary_id,
content_range,
summary_created_at,
content_range_upper
FROM topic_candidates
ORDER BY summary_created_at DESC NULLS FIRST, last_posted_at DESC
```
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 14, 2025, 6:47pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/3 "2025-01-14T18:47:20Z")
There are 5 hidden or deleted posts in this topic. 108-103=5. Is it possible it’s not handling hidden and deleted posts correctly? Yes 108 is the highest post, but only 103 are ever passed in, so it continually thinks there are 5 new posts to summarize.
@Falco @sam
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 14, 2025, 9:53pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/4 "2025-01-14T21:53:48Z")
I have another topic that also keeps regenerating the summary. That topic has no hidden or deleted posts, but is pinned, which creates a post “this topic was pinned globally…”. So max post is 8, but that last post isn’t really passed in?
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [January 14, 2025, 11:42pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/5 "2025-01-14T23:42:31Z")
> [@markschmucker](#):
>
> `UPPER(ais.content_range) < t.highest_post_number + 1`
Yeah I think you are on to something here @Roman
>
best\_replies is not guaranteed to include the last public post on a topic. We should add the last post unconditionally even when picking best\_replies.
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 15, 2025, 12:40am UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/6 "2025-01-15T00:40:34Z")
Thanks for looking into it. And just to clarify the impact, it’s not just the inefficiency of regenerating the same summary repeatedly- it completely stops the backfill. Say you have backfill set to 24 topics per hour. Every 5 minutes it tries 2 topics. Eventually both topics have an issue- deleted topics or a pinned topic or something else- and it keeps retrying the same 2 topics every 5 minutes. It doesn’t even try any other topics.
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 15, 2025, 3:00pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/8 "2025-01-15T15:00:33Z")
> [@sam](#):
>
> `.where("NOT hidden")`
Does it need `.where("NOT deleted")` also? I don’t understand the difference between hidden and deleted, but I have both hidden and deleted in my problem topic.
---
### Author:  [@Roman](https://meta.discourse.org/u/Roman)
#### Post date: [January 15, 2025, 3:30pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/9 "2025-01-15T15:30:12Z")
Hey Mark,
I’m currently looking into this and trying to figure out what’s going on. What @sam pointed out is true—`best_replies` not including the last post could cause the job to stall. I’m about to wrap up a fix for that and introduce a fail-safe to turn the job into a no-op and log an error if there’s a bug in the query. This is not ideal and it shouldn’t happen, but it’s better than regenerating the same summary over and over again.
On the other hand, I’m looking at the `Franklin Lexington...` example you shared, which I suspect is a different issue because:
1. It already has a summary (`149`).
2. `UPPER(ais.content_range) < t.highest_post_number + 1` should be `FALSE`. `UPPER` should return `109`, which equals `highest_post_number + 1`.
The end of the `content_range` interval would need to be lower for the job to stall due to that. We don’t really care about hidden or deleted posts when deciding if a summary is outdated; we only want to avoid summarizing those.
Do you have `ai_summary_gists_enabled` enabled? If so, can you confirm that you have two summaries for that topic, one with type `0` and one with type `1`? Also, in the `ai_api_audit_logs` query you shared, what is the value in the `feature_name` column?
I’ll keep you updated with what I find.
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 15, 2025, 5:45pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/10 "2025-01-15T17:45:27Z")
Thanks for looking into this. Happy to provide error logs when your fix is ready.
> [@Roman](#):
>
> Do you have `ai_summary_gists_enabled` enabled?
[https://meta.discourse.org/t/summarize-gists/340269/3](https://meta.discourse.org/t/summarize-gists/340269/3)
I don’t see how to enable it, so I don’t think it’s enabled.
> [@Roman](#):
>
> what is the value in the `feature_name` column
“summarize”
---
### Author:  [@Roman](https://meta.discourse.org/u/Roman)
#### Post date: [January 16, 2025, 3:46pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/11 "2025-01-16T15:46:46Z")
This will unblock the queue and make the job more resilient to these kind of problems:
[https://github.com/discourse/discourse-ai/pull/1071](https://github.com/discourse/discourse-ai/pull/1071)
Could please let me know how things look after updating your site?
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 17, 2025, 3:06pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/12 "2025-01-17T15:06:57Z")
It’s no longer regenerating summaries for topics that already had a summary, so that’s good.
But it’s not summarizing many topics, even when there are new posts, because of:
> [@How Does Regenerating Summaries Work?](https://meta.discourse.org/t/how-does-regenerating-summaries-work/333408/18):
>
> Found the problem. It’s comparing ai summary backfill topic max age days to topic.created\_at, not updated\_at. I think this should be changed to updated\_at- I have many very active topics created two years ago which still get new posts every week, but if I use a max age of 90 days or even a year, those topics will not get summarized. Please consider changing this. .where("topics.created\_at \> current\_timestamp - INTERVAL '#{max\_age\_days.to\_i} DAY'")
So backfill thinks it’s done, but maybe 75% of the Latest topics do not have a summary, even though they have new posts, **just because they happened to be created more than 90 days ago**. I’m sure that’s not the intent. Please change it or help me understand why.
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 18, 2025, 10:17pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/14 "2025-01-18T22:17:33Z")
> [@Roman](#):
>
> Could please let me know how things look after updating your site?
I forked the ai repo so I could change `created_at` to `updated_at` and move ahead. I’m pleased to report that it successfully summarized all of the 400+ topics I expected it to summarize. Some failed on the first backfill pass, possibly because we were over quota for that minute, but it successfully summarized those on the second pass.
In particular, the Franklin Lexington topic now has a summary.
Again, backfilling is not going to work well yet for others, because of the `created_at` issue.
---
### Author:  [@sam](https://meta.discourse.org/u/sam)
#### Post date: [January 20, 2025, 4:30am UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/15 "2025-01-20T04:30:20Z")
> [@markschmucker](#):
>
> Again, backfilling is not going to work well yet for others, because of the `created_at` issue.
Yeah I agree, we need to look at updated\_at or last\_posted\_at which is slightly less radical.
At the end of the day if we are backfilling we should base it on content change.
---
### Author:  [@markschmucker](https://meta.discourse.org/u/markschmucker)
#### Post date: [January 21, 2025, 5:29pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/16 "2025-01-21T17:29:23Z")
> [@sam](#):
>
> updated\_at or last\_posted\_at
If the difference is edits, I’d vote to regenerate on edits. We have a wiki as the first post of each topic which members wiki-edit with critical info that could affect the summary.
But if you choose not to do that, I can keep running my fork.
---
### Author:  [@Roman](https://meta.discourse.org/u/Roman)
#### Post date: [January 22, 2025, 6:05pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/17 "2025-01-22T18:05:17Z")
Sorry for the radio silence. I’ll try to make some time to work on this soon.
---
### Author:  [@Roman](https://meta.discourse.org/u/Roman)
#### Post date: [February 4, 2025, 2:51pm UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/18 "2025-02-04T14:51:10Z")
This got merged this morning:
[https://github.com/discourse/discourse-ai/pull/1108](https://github.com/discourse/discourse-ai/pull/1108)
The backfill job now uses `last_posted_at` instead of `created_at`. I also made changes to the logic to determine if a summary is outdated by checking if any of the posts’ `last_revised_at` is newer than the summary, to take edits into account.
---
### Author:  [@Falco](https://meta.discourse.org/u/Falco)
#### Post date: [February 8, 2025, 11:00am UTC](https://meta.discourse.org/t/ai-summarization-backfill-is-stuck-keeps-regenerating-the-same-topic/347088/19 "2025-02-08T11:00:42Z")
This topic was automatically closed after 3 days. New replies are no longer allowed.