Right now, file names are indexed for search but the contents are not. It would be nice if the contents of files were also indexed, at least for the most common text-based files types, e.g. txt, pdf, doc, xls, csv, etc.
Any plans for this?
8개의 좋아요
No plans for this at the moment.
“Me, too”
We’d like to have attachments (in our case, PDF’s) indexed for the search engine, too.
3개의 좋아요
This is very much an enterprise customer type feature. We don’t have concrete plans here with a timeline, I am uncertain what would happen to Postgres with huge PDF documents.
Certainly something we have thought about over the years and may get to over the next few years.
4개의 좋아요
Curious if Discourse added the capability to index and search PDF’s yet?
1개의 좋아요
Not yet, very feasible to build though in a plugin.
1개의 좋아요
When developing such a plugin: Where would you start? Being totally new to the discourse code I’d probably try to hook in UploadCreator, but that might be very wrong.
Developing a Discourse plugin that integrated with Paperless would be a good start.
Such a plugin would be involved to say the least, as stated before.
A plugin like this would require that the Discourse API allows for external handling of documents. Is that currently available?
This plugin would require integration with the search capabilities offered by the Discourse API. While this is not trivial, this has been done by several existing plugins, namely the discourse/discourse-algolia plugin.
Other Areas to Consider
- Backups
This would be something I would personally be interested in cutting my teeth on. I have started by looking at the Paperless API along with reverse engineering the discourse/discourse-algolia project… but there are others that integrate with search.
Any thoughts as to the choice of Paperless? I like how active the project is and the amount of issues they have closed in addition to the number of issues they have (currently 0).
2개의 좋아요
This would be a HUGE improvements especially for those forums that carry a lot of documentation via PDFs, scans, or images. If the scope of this limited to searching, then ideally you would just need to extract/create text and then store that text in a column in postgres. When you do that, then you can leverage the postgres full-text already in-place.
In linux for example, you could pdftotext to extract then text from a PDF and store in DB. Another (more expensive) idea is to use AI vision to explain/extract the pdf or image for storing in the DB.
Thoughts?
2개의 좋아요
그래서 이 문제를 조사해 보고, AI를 활용해 최적의 방법을 구현하기 위한 와이어프레임을 작성했습니다. 몇 가지 아이디어가 떠오르는데요. Apache TIKA를 사용하면 이미지 포함, 텍스트가 있는 거의 모든 파일 형식을 OCR로 처리할 수 있습니다. 이는 자체 호스팅 옵션이기도 합니다. 그리고/또는: Gemini Flash 1.5(예를 들어)를 사용해 OCR을 수행할 뿐만 아니라, 확인하고 분석된 이미지를 설명한 데이터를 PostgreSQL 테이블/열에 게시하여 검색할 수 있도록 할 수 있습니다. 물론 첨부파일/업로드가 있는 모든 게시물을 다시 처리하려면 상당한 양의 토큰이 초기에 필요하지만, 이는 가장 유용할 것입니다. 결국 가격만큼의 가치를 얻는 것일까요?
1개의 좋아요