PDF support in Discourse AI

Documentation Site Management
,
1

:bookmark: This guide explains how to implement and use PDF processing capabilities within discourse-ai, including both basic text extraction and enhanced processing with LLM assistance.

:person_raising_hand: Required user level: Administrator

Summary

The discourse-ai plugin supports PDF processing for RAG (Retrieval-Augmented Generation) in two distinct modes:

  1. Basic text extraction
  2. Enhanced processing with LLM analysis

Basic text extraction

This mode provides fundamental PDF processing capabilities:

  • Extracts text content using the pdf-reader gem
  • Supports files up to 100MB
  • Works immediately after plugin installation
  • Processes text-only content (ignores visual elements)

Enhanced processing with LLM improvements

:information_source: This mode requires specific configuration and provides more advanced capabilities.

Requirements:

  • Enterprise plan subscription or self-hosted Discourse
  • ImageMagick with Ghostscript support installed in container
  • ai_rag_images_enabled site setting enabled

Capabilities:

  • Interprets images, charts, and diagrams
  • Provides context from visual elements
  • Processes PDFs page by page
  • Maintains the 100MB file size limit

Implementation details

Processing specifications

  • Page processing resolution: 300 DPI
  • Maximum processing time: 600 seconds (10 minutes)
  • Automatic cleanup of temporary files
  • Full integration with RAG document embeddings

Processing workflow

  1. PDF upload and validation
  2. Content extraction (basic or enhanced mode)
  3. Text chunking with configurable overlap
  4. Chunk embedding and storage
  5. Progress tracking via MessageBus

Limitations

:warning: Be aware of these constraints when implementing PDF processing:

  • File size restrictions:
    • 100MB for existing PDF processing
    • 20MB for new admin interface uploads
  • Enhanced mode requires additional system resources
  • Complex PDF layouts may not be perfectly interpreted
  • Enhanced processing increases processing time significantly
7 Likes
2

This is really amazing news. Thanks team! Can’t wait for the enhanced processing to be finished. That’s gonna be critical for feeding LLMs research papers.

Also, is there any plan to allow doing RAG “chat-with-your-PDFs” by uploading PDFs in an AI bot PM or in a topic/post and mentioning the bot?