Fix for “Record is too big” aborting the Algolia posts reindex
I encountered an issue where Algolia contained only a small portion of my forum’s posts. Running the posts reindex revealed the cause:
Algolia::AlgoliaHttpError:
Record at the position 258 objectID=1006 is too big:
size: 108426 byte(s), maximum allowed: 10000 byte(s).
In my installed plugin version, PostIndexer#to_object sends the entire post’s plain text as a single record without checking the serialized record size. An oversized record caused the reindex task to abort.
The following local workaround allowed the reindex to finish.
What this patch does
The patch limits each complete serialized post record to 9,500 bytes, leaving a margin below the 10,000-byte limit reported by Algolia.
It:
-
Includes metadata and JSON escaping in the size calculation.
-
Truncates only the content sent to Algolia.
-
Leaves the original Discourse post unchanged.
-
Keeps existing object IDs and the one-record-per-post structure.
The trade-off is that text omitted from the end of a long post will not be searchable through Algolia. This is truncation, not document chunking.
Patch
In:
plugins/discourse-algolia/lib/discourse_algolia/post_indexer.rb
Find the final return in to_object, immediately after the optional category metadata is added:
object
end
Replace that section with:
# Limit the complete serialized record, not just its content.
algolia_record_budget = 9_500
if object.to_json.bytesize > algolia_record_budget
full_content = object[:content]
object[:content] = ""
if object.to_json.bytesize > algolia_record_budget
raise "Algolia metadata alone exceeds the record budget for post #{post.id}"
end
low = 0
high = full_content.length
best_content = ""
while low <= high
midpoint = (low + high) / 2
object[:content] = full_content[0, midpoint]
if object.to_json.bytesize <= algolia_record_budget
best_content = object[:content]
low = midpoint + 1
else
high = midpoint - 1
end
end
object[:content] = best_content
end
object
end
The binary search finds a content prefix that fits the byte budget after JSON serialization. If metadata alone exceeds the budget, the patch raises an explicit error rather than silently discarding metadata.
Back up the file before editing. This patch targets the version where to_object returns a single hash containing content; review it against your installed version before applying it.
Validate and reindex
For a standard Docker installation, enter the container from the host:
cd /var/discourse
./launcher enter app
Switch to the Discourse user:
su - discourse
Once the discourse user’s prompt appears, run:
cd /var/www/discourse
ruby -c plugins/discourse-algolia/lib/discourse_algolia/post_indexer.rb
If that reports Syntax OK, run:
RAILS_ENV=production LOAD_PLUGINS=1 bundle exec rails algolia:reindex_posts
This task clears and repopulates the posts index. In my case, it completed successfully after applying the patch.
Already-running application/background-worker processes also need a restart to use the patched code for ongoing indexing.
A separate gotcha: the reported post count
The task reported:
Successfully pushed 723 posts to Algolia
But Algolia contained 272 records.
But Algolia contained 272 records.
In my installed version, the task increments its counter before calling the indexer’s visibility filter. A diagnostic using the same indexer confirmed:
Posts examined: 723
Eligible for indexing: 272
Excluded by visibility: 451
So the success message counts posts examined, not necessarily records uploaded. This discrepancy was separate from the oversized-record error.