PDF support in Discourse AI

Documentation Site Management
,
1

:bookmark: This guide explains how to implement and use PDF processing capabilities within discourse-ai, including both basic text extraction and enhanced processing with LLM assistance.

:person_raising_hand: Required user level: Administrator

Summary

The discourse-ai plugin supports PDF processing for RAG (Retrieval-Augmented Generation) in two distinct modes:

  1. Basic text extraction
  2. Enhanced processing with LLM analysis

Basic text extraction

This mode provides fundamental PDF processing capabilities:

  • Extracts text content using the pdf-reader gem
  • Supports files up to 100MB
  • Works immediately after plugin installation
  • Processes text-only content (ignores visual elements)

Enhanced processing with LLM improvements

:information_source: This mode requires specific configuration and provides more advanced capabilities.

Requirements:

  • Enterprise plan subscription or self-hosted Discourse
  • ImageMagick with Ghostscript support installed in container
  • ai_rag_images_enabled site setting enabled

Capabilities:

  • Interprets images, charts, and diagrams
  • Provides context from visual elements
  • Processes PDFs page by page
  • Maintains the 100MB file size limit

Implementation details

Processing specifications

  • Page processing resolution: 300 DPI
  • Maximum processing time: 600 seconds (10 minutes)
  • Automatic cleanup of temporary files
  • Full integration with RAG document embeddings

Processing workflow

  1. PDF upload and validation
  2. Content extraction (basic or enhanced mode)
  3. Text chunking with configurable overlap
  4. Chunk embedding and storage
  5. Progress tracking via MessageBus

Limitations

:warning: Be aware of these constraints when implementing PDF processing:

  • File size restrictions:
    • 100MB for existing PDF processing
    • 20MB for new admin interface uploads
  • Enhanced mode requires additional system resources
  • Complex PDF layouts may not be perfectly interpreted
  • Enhanced processing increases processing time significantly
11 Likes
2

This is really amazing news. Thanks team! Can’t wait for the enhanced processing to be finished. That’s gonna be critical for feeding LLMs research papers.

Also, is there any plan to allow doing RAG “chat-with-your-PDFs” by uploading PDFs in an AI bot PM or in a topic/post and mentioning the bot?

3

@sam Can you provide simple video to explain this great option, because what you mentioned not clear enough to implement

4

Where I can find this setting

5

It’s a hidden setting, you need to use the console, but you also need to configure the container, recommend you wait a few more weeks

3 Likes
6

Thank you, I appreciate your fantastic work

7

In my website (Arabic Forum) I did a test in Arabic by adding legislation in the first post “topic” and then I asked questions using AI, but the answers not accurate and I think this is because it is not Context Ragging

8

Sorry but this is not how it works, you need to define a persona or tool and then add the upload there.

There has been some discussion around supporting “upload and ask” here: Upload and discuss pdfs in composer but it is not supported yet.

9

First of all, really thank you for your great work. I really like it.

After playing around with the settings and changing the AI Model to Gemini-Flash-2.0, it worked great for me. Here’s the situation I have:

We are an Auditors, Accountants, and Tax Consultants community, and we needed a tool to share related laws and trigger discussions about them. This discussion should be very useful for visitors, as we are professionals in our field. We are targeting the AI Model to check and analyze legislation and answer our questions. The great experiment led to the conclusion that we can really discuss the context added in the first post, and if the AI model is smart enough, it will answer our questions with very high-quality output.
Really thank you again and looking forward to the PDF support as it will make Discourse best forum Sofware

3 Likes
10

Latest image of discourse supports the advanced mode if anyone feels like testing

2 Likes
11

Does it have to be enabled via console? Don’t see any advanced mode options via the UI.

Furthermore, I am getting an error when trying to upload this pdf. It is 34 MB but I have my max attachment size set to 100 MB (in both admin settings and app.yml). What’s strange is that I have a compressed version which is 16 MB and it uploads just fine. But perhaps the larger PDF is simply too complex for now? There are lots of images, equations, etc.

Screenshot 2025-02-28 at 13.07.43
Screenshot 2025-02-28 at 13.07.431648×324 13.7 KB

12

Yes, you need to SiteSetting.ai_rag_images_enabled = true in the Rails console to enable it.

1 Like
13

my guess here is that some nginx stuff needs to change in the container as well so it does not do the rejecting

1 Like