Does Discourse support export conversations as an organized bulk of data?

Contribute Feature

AI-generated summary

The discussion revolves around the possibility of exporting conversations from Discourse as an organized bulk of data. Ivan_Rapekas initiated the conversation, explaining his use cases for such a feature, including extracting data for document preparation and organizing external product documentation. He currently uses Python to parse conversations and extract structured data.

Holger_Winkelmann suggested using TensorFlow to avoid unstructured data, but Ivan_Rapekas clarified that his goal is to collect data from different sources and find important highlights to create a structured dataset.

Ivan_Rapekas shared a Python template he uses for this purpose and expressed interest in having a similar data structure exported from Discourse, possibly through API.

Tobiaseigen suggested adding .json to Discourse URLs to view pages in a more portable format, providing an example URL. Ivan_Rapekas found this solution sufficient for his needs, thanking tobiaseigen for the suggestion.

1

Does Discourse support export conversations as an organized bulk of data that could be reused offline? Use cases in Slack:

  1. sometimes I extract some data from history to prepare documents for newcomers.
  2. the same approach to extract useful data from several discussions to organize external product documentaion.
  3. trying to parse a lot of bad-sorted topics/conversations using Python to get structured data with highlighted words - steps to define links between different conversations (some kind of data analyzing)

I hope I’ve been clear :grinning:

1 Like
Discourse for Teams is here!
Dump all conversations in a file and structured data
2

hhm, have you used tensorflow? :wink: You might should avoid many unstructured data in the first place. I ave the same problem here. But I try to get it somehow sorted at the source. I have a slack chat hell here I will get around by letting people structure the Data…

3

I’m not sure that AI is my case. I collect data from different sources and the main idea is to find really important highlights (words, urls, proofs etc) and create a structured data which could answer these questions:

  1. What was the proper order on the way of accepting the certain solution related to the some task. In order to reconstruct the actual picture of events.
  2. The algorithm should detect important mentions in a tons of poor-quality conversations (especially email with many levels of attachments and non-trusted web publications)
  3. Define valuable links between different actions, approximately such way: news → blog → public mood and needs → chat/email decision → used strategy → real actions → approved assumptions → related persons → explaination the result

So I use some python template for this:

PRODUCT_RELATIVE_SOURCES = {
    "websites": {
        "company1": [
            "blog",
            "vacancies",
            "news",
            "tags"
        ]
    },
    "social-networks": {
        "network1": [
            "feed",
            "story",
            "public",
            "direct",
            "tags"
        ]
    },
    "messengers": {
        "messenger1": [
            "chat1",
            "room1",
            "bot1",
            "direct",
            "tags"
        ]
    }
   "mailboxes": {
        "box1": [
            "subject",
            "body",
            "sender",
            "cc",
            "meta"
        ]
    }
}

EXCLUDE_SOURCES = {
    "main",
    "libs",
    "opt"
}

Probably I wonder to have similar data structure exported from Discourse (through API maybe). Initially I asked the question concerning Discourse for Teams, because I found many similar with Slack and our team is not satisfied with Slack. Payable function of history is almost useless.

1 Like
4

In this regard, everything you can do with Discourse you can do with Discourse for Teams. This is why I moved your post into its own new topic. Maybe others have suggestions for you.

Are you familiar with json? You can add .json to pretty much any URL in Discourse to see the page in a more portable format. Maybe that helps?

For example, this topic:

https://meta.discourse.org/t/does-discourse-support-export-conversations-as-an-organized-bulk-of-data/180537.json

5 Likes
5

Wow, it looks great, thank you very much, Tobias! I think it is enough for me :+1:

2 Likes