Data Explorer Plugin

Summary: The Data Explorer plugin allows you to make SQL queries against your live database, allowing for up-to-the-minute stats reporting.

:link: GitHub: https://github.com/discourse/discourse-data-explorer.git
:arrow_right: Install: Follow the plugin installation guide.

Features

After installing the plugin, head to /admin/plugins .

Click on the “Settings” button, and turn on data explorer enabled, then return and reload the page.
There should now be a tab in the stacked nav called “Data Explorer”. Select it or head to /admin/plugins/explorer , then use the + , Import or Run Query buttons to get started.

Configuration

Default Queries

On a fresh install, the data explorer now ships with a bunch of useful queries that can help you draw insights from your forum’s activity. Open any query and click Run Query to try it out.

Here's what the stock queries look like.

Note: Default queries are picked up from this file: https://github.com/discourse/discourse-data-explorer/blob/master/lib/queries.rb

If you have an awesome query that you want everyone to have, make a PR just like this one.

Default queries cannot be edited, feel free to copy the sql and paste it into a new query if you’d like to modify them.

Add your own queries

For an example, let’s import one that I wrote. Download this file, then open the import dialog and select the file to be uploaded or paste the query in the text box. Click Import to save it and then click Run Query to see the results.

most-common-likers.dcquery (1).json (442 Bytes)

Looks like all of the Discourse developers like each other quite a bit :heart:

I don’t know any SQL, where do I find more queries?

Our community has got you covered! :wink:
Import as many queries as you like from here: What cool data explorer queries have you come up with?

Writing Queries

When you click any of the Edit buttons on a query view, or start a new Explorer Query, you are presented with a screen like the one below:

The top two boxes are the name and description of the query. The name is what you’re selecting in the dropdown box, and the description should be used for things like explaining what to put in the query parameters or to describe what data is being queried.

The left pane is where you write the SQL query. Minor syntax highlighting and checking is provided.

Declaring Parameters in your query

Lets modify our example by using a parameter instead of hard-coding values. If you declare a variable as shown below, a text field will show up where you can input different values! Try it out!

de6


-- [params]
-- int :limit = 100
WITH pairs AS (
SELECT p.user_id liked, pa.user_id liker
FROM post_actions pa
LEFT JOIN posts p ON p.id = pa.post_id
WHERE post_action_type_id = 2
)
SELECT liker liker_user_id, liked liked_user_id, count(*)
FROM pairs
GROUP BY liked, liker
ORDER BY count DESC
LIMIT :limit

There are several available kinds of parameters to use in your query, some with specialized input. Any of these parameter types can be prefixed with null to allow an empty input.

  • int, bigint, double - Shows a number input, becomes a numeric value. int is restricted to 32-bit numbers, while bigint can be larger.
  • int_list - Enter comma-separated integers, becomes comma-separated integers in the query. (See List Parameters below.)
  • post_id, topic_id, category_id, group_id, badge_id - Numeric input; ensures the specified post, topic, category, group, or badge exists on the forum before running the query to prevent mistakes.
  • boolean - Shows a checkbox, or a dropdown if null boolean is used instead.
  • string - Freeform textbox, becomes a text value.
  • string_list - Comma-separated freeform text. (See List Parameters below.)
  • time, date, datetime - Textbox that is string-casted to the appropriate type in the query. No client-side validation (yet!).
  • user_id, user_list - Shows the Discourse user selector box, and becomes the numeric user ID or a comma-separated list of the numeric user IDs. (See List Parameters below.)

Using list parameters

Three of the above parameter types are “lists”. Special care must be taken when using these to avoid syntax errors. Here is an example of correctly using a list parameter:

-- [params]
-- user_list :the_user_ids
SELECT SUM(length(bio_raw))
FROM user_profiles
WHERE user_id IN (:the_user_ids)

Searching in your queries list

If you’ve accumulated too many awesome queries, no problem! The search bar can help you filter through titles and descriptions.

Allow moderators or custom groups to run a query

You can add groups (including moderators) to a query to grant them access to it:

After saving, you can find it from the Groups page for that particular group on the Reports tab:

Click any query to open it and then the Run button to see the results.

Limitations

  • If there are more than 1000 results, Run Query will return only the first 1000. Clicking Export will let you download all the results with a much higher limit.

CHANGELOG

TODO

84 Likes
Daily graph of user time online
What cool data explorer queries have you come up with?
Visual forum statistics
Network chart of your forum - data visualization
Extracting email addresses for a group
Visual map of users
Can I see in the statistics which post received the most likes
How do I download a record of top posts or most clicked links?
Beginners Guide to Install Discourse on Ubuntu for Development
Discourse Narrative Bot Data Explorer Queries :robot:
How to look up a user based on a user_id?
Discourse view file update does not reflect in browser
How to measure active users?
Where to find User ID information
Exporting all Topic titles and IDs?
External Data Analysis
Can staff be able to see real author of anonymous posts
Create report for amount of interactions per user
Tags: how can admin see who is following what tags?
CSV of all editeds post by moderator
[Paid] Data explorer queries for ticketing system use case
Retort - a reaction-style plugin for Discourse
Direct connect to the Discourse Database
How do I clear Warnings?
Can I make a plug-ins to view a certain group of people?
See what categories new users (& other trust levels) post in most
Can I access my Discourse forum database?
How can I extract the adminstrative records for a EU GDPR request?
Counting and reporting post views for commercial forum users (so we can charge them per-view)
Top user list public
Legal Tools Plugin
How do I audit votes for fraudulent behavior
How do I get a list of all users from the API?
Tracking referrals? Encouraging users to invite others?
Display Data Explorer query results in WordPress (with TwigAnything)
Database's Data Model?
How do I add users to a group?
Discourse disk space limits per user?
Masonry Image Gallery
Estimated timeline on a project?
Report on User Fields?
Dashboard - Segment number of Posts and Topics by team
Traditional multi level hierarchy vs flat discourse hierarchy
Is there a Trust Level Permissions Table?
Deleting posts by banned users with < 3 posts?
Is there a way to Search by Posts with No Staff replies on it
How to use SQL to query discourse database
Specify user by external id
Importing from phpBB3
Retrieving user information via REST API
How to check the uniqueness of users in "hot link" badge?
Active users per day
What cool data explorer queries have you come up with?
What cool data explorer queries have you come up with?
Questions about moving an existing forum to Discourse
Get a user object by username or id javascript
Best Practices on reply-by-email
Most liked reply
Likes on a post
Group forum members based on topics viewed?
The metrics on dashboard, do they include activities from administrators?
Can I access Discourse forum Database to use for PowerBI reporting?
Is there a log of user visits?
Which user has the most badges?
Find posts "solved" in specific month
Anonymous User Groups
Any way to NOT send emails when a topic category is changed?
Scan IP matched accounts
View/read by user or of user's posts?
Bug with [code]anything[/code] on a single line by itself
Logo is not appearing showing broken after upload
Users reporting lots of 502 errors when attempting to post due to "max consecutive replies" check
Query to find out who marked the solution as solved
What would you like to see on your Discourse 2.0 Admin Dashboard?
How to search/filter untagged topics
TeamSpeak Usergroups Sync
Use tags to duplicate images?
Database Access required for our forum
Automatic badge expiration
Looking to switch from Vanilla, and have a few questions
Comment on a question instead of answering it
Making Poll Results Visible After it has been closed
How to create custom reports
Deleted username but can not use that username in other user
Category Remains Muted after Category is available to Everyone
How do I pull a report for a single community member?
Report Links to New Topics by Date?
Excluding Suspended, Silenced and Suspect Users When Exporting
Theme-Component v Plugin: What's the difference
Export topics to Excel
Discourse Assign
Post notices (staff notices)
How to get reports for just a specific category?
Reports for group message statistics
TimeStamp of Tag
How we put people in a category?
Is it possible to see the dates of all scheduled topics?
Topic Count Per User for Custom Date?
Searching for unassigned topics
Reports by Discourse
How can I get the list of Discourse Topic IDs dynamically
How to Create Polls
How to edit sign up form?
Filter posts with images (gallery view)
Tracking unique contributors
New users who visited this month?
Making report on group membership?
Where to find post contents in the database
Searching All Messages as an Administrator
Are Staff and All Groups Included in Dashboard Reporting?
Deleted user with rails console u.delete -> email still "taken"
First Name, Last Name
Identify private message interactions between users (Data Explorer)
User Messages Inbox Error 500
Description of fields returned by Discourse API
Is there a document intruducing the database schema design?
Failed to bootstrap due to out of memory killer
What will happen to my forum if discourse suddenly stopped?
Why aren't warnings easily accessible like suspensions are?
Organization of community Data Explorer queries
Wildcard search
Possible to run a report for most posts within community?
Add an Admin subtab for Inactive users
Questions regarding Discourse features
How to query an email list of users that are not actived
Shopify Integration
How do I see Facebook signups?
Poll: Export poll results in CSV for quiz creator?
Discourse disk space limits per user?
Sorting all users by total posts?
SQL: The most N used words per user (speak their language!)
User Summary - Most Liked By user attributions are incorrect
Maximum Number of Search Results?
Get only top level posts in topic via discourse API
How to get all the deleted posts for a specific topic
Stats reports on a per-category basis?
Data analytics by complex networks and text mining
Error thrown when importing a query in Data Explorer
Discourse Solved (Accepted answer plugin)
Zero search results searching private messages for specific keyword
A way to see full list of users granted with a certain badge?
Who messaged the most?
Who messaged the most?
How to open hyperlinks in a new tab
Run report of digest Unsubscribes
Need log the who downloaded attachments
Redesign of use profile
Poll problem - exact vote numbers needed
Telegram Notifications Plugin (discourse-telegram-notifications)
Get last :heart: via API
Is it possible to keep the votes history of a poll?
How do I check a badge's status midway to it being awarded?
What plugins do you use - and why?
What cool badge queries have you come up with?
Export list of users based on date range
How to get all posts a user interacted with?
How can I email all recently active users?

I have just added documentation of the list of available query parameter types. #pr-welcome if anyone wants to change time, date, and datetime to use the in-browser calendar view.

9 Likes

If the column that is returned is called user_id or another name that ends with user_id, it will return a clickable avatar. For example, the following query will return user IDs instead of clickable avatars:

SELECT
id
FROM users LIMIT 5

To have clickable avatars returned, change the query to:

SELECT
id AS user_id
FROM users LIMIT 5
5 Likes

There is not. Data Explorer is very intentionally read-only. It is used to obtain data, never modify it.

You’d need to connect to the site via a rails or db console to make changes via SQL commands.

5 Likes

You can construct it with a admin_page_url column, or generally any result name ending in _url.

SELECT
  '/admin/users/' || users.id || '/' || users.username_lower
    AS admin_page_url
FROM users WHERE id = 172

Source: https://github.com/discourse/discourse-data-explorer/blob/master/assets/javascripts/discourse/components/query-row-content.js.es6#L86

Edit: _url has more hidden features. You can also do this:

SELECT
  'Admin Page,/admin/users/' || users.id || '/' || users.username_lower
    AS admin_page_url
FROM users WHERE id = 172
5 Likes

You can find instructions for that here: How to run Data Explorer queries with the Discourse API

The most likely problem is that you’re trying to pass the api_key in the URL. That’s no longer supported - you need to use HTTP headers to pass the credentials. More details on that can be found at docs.discourse.org

7 Likes

That’s the bit I missed: has to be headers now. Thanks!
dg

5 Likes

Thanks for the PR @Grayden_Shand - sorry we missed it for so long. We keep a very close eye on Discourse core PRs, but sometimes the plugins/themes are accidently missed.

We don’t require a post on Meta, but it can sometimes help to give context on a change. If I PR has been without activity for a very long time, do feel free to make a post here to remind us, as you’ve done :+1:

9 Likes