Twig Anything – Show any data from Discourse in Wordpress – an Open Source WordPress plugin

[meglio (Anton)]

Open-sourcing Twig Anything – a comprehensive WordPress plugin to retrieve and display JSON, CSV or MySQL data – from anywhere, including from Discourse Data Explorer.

Source Code

Fork me; PRs welcome:
https://bitbucket.org/meglio/wp-twig-anything

Related topics here at Meta (disregard pricing info, it’s now FREE and OPEN SOURCE):

• Announcing WordPress plugin to display any data from Discourse
• Display Data Explorer query results in WordPress (with TwigAnything)
• "Recently joined" widget for WordPress (Illustrated Tutorial)
• "Who's online" widget in Discourse - my version

WordPress install

1. Git clone the repository
2. Copy folder plugin_src to your WordPress plugin directory
3. Rename plugin_src to twig-anything’
4. Activate the plugin in your WordPress dahsboard

---

Features

• Understands JSON of any complexity
• Fetches JSON from URLs (demo) or any data from your blog’s MySQL database (demo, tutorial)
• Configurable CSV (comma-separated values) reader
• Caches retrieved data locally for better performance
• Uses expired values from cache as a fallback if data source is not available
• Simple Twig syntax to output data (with syntax highlighting)
• Embed it anywhere in WordPress with using shortcodes
• Compatible with Visual Composer (demo)
• Integrates with Discourse (demo, tutorial 1, tutorial 2)

---

Donate

Any donations to PayPal a.andriievskyi@gmail.com will be spent on:

• hosting home page https://twiganything.com/
• further development, PR review and non-priority support

Version changelog

The current version of Twig Anything is 1.6.5

v1.6.5 ^2017-12-05

• Update CodeMirror library to v. 5.32.0
  (the existing bundle stopped working
  for the latest Google Chrome browser version)

• Wrap CodeMirror library to prevent conflicts with CodeMirror instances
  loaded by other WordPress plugins or themes

• Toggle CodeMirror full screen mode by Ctrl-Enter rather than F11
  (F11 has a different purpose in MacOs)

• Use Clipboard.js library instead of unsupported and outdated ZeroClipboard

v1.6.3 ^2016-03-13

• Add GET/POST selector to the URL Data Source
• Update React.js library to v. 0.14.7

v1.6.2 ^2016-01-23

• Make get_post_meta() and get_current_post_meta() functions available in Twig Templates (example use-case)

v1.6.1 ^2015-08-14

• Add {{request('var-name')}} function to Twig templates
• Add “Empty” data source that would not fetch anything and always return empty data; useful for template inheritance and use by other templates.

v1.6 ^2015-08-03

• Make WordPress globals available in Twig Templates through {{ wp_globals }}
  Note. From now on, accessing wpdb is only possible as {{ wp_globals.wpdb }}
• Make WordPress conditionals available in Twig Templates
• Make wp_get_current_user() and get_current_user_meta() functions available
  in Twig Templates
• Make URL Data Source accept Twig syntax in URLs, just like Twig Templates do

v1.5 ^2015-08-02

• Add |json and |json_pretty_print filters
• More extensibility points for add-on developers:
  – Ability to load React.js library as a dependent script
  – Control removal of line breaks from templates
  – Override Twig configuration on template rendering

v1.4 ^2015-07-30

• Widgets support: add a custom widget named “Twig Template”

v1.3 ^2015-07-27

• Extensibility points for custom formats and data sources
• Update for php 5.3 compatibility (no warranties!)

v1.2 ^2015-07-26

• Improve the caching engine so that expired cached values can still be used as a fallback
• Add an option to configure how data errors are handled:
• Use cache or display nothing
• Use cache or display error
• Always display error
• Note. In the preview mode, errors are always displayed.
• Make the preview mode more intelligent

v1.1 ^2015-07-12

• Visual Composer support (demo)

v1.0 ^2015-07-01

• Initial release

[meglio (Anton)]

What if data cannot be fetched?

Sometimes, data cannot be retrieved and an error is thrown.

For example, JSON API is not accessible, or a file cannot be found to load CSV rows, or a remote database is not accessible because username/password do not match.

For such cases, Twig Anything lets you configure what to do if an error occurs during data retrieval:

The setting can be configured for every template individually. Possible options:

• Use cache or display nothing . If there is outdated data in cache, it is used, otherwise nothing is displayed
• Use cache or display error . The same as above, will try to use a cached value, even if it has expired; however, if nothing found in cache, an error is displayed.
• Always display error . This option will let the template always show the error message; might be useful while your WordPress website is still not so popular and is being actively developed.

Internals

The schema below demonstrates what happens behind the scenes every time a Twig template is processed:

As you can see, the logic is not trivial and lets the cache system play well with the error handling engine. However, you won’t have to think much about it as it all happens automatically. All you need is to choose one of the 3 options outlined above.

[meglio (Anton)]

FAQ

Functionality

Minimum Requirements

• WordPress version 3.6.1 or greater
• PHP version 5.4 or greater

How do I render data in my post / page / widget etc?

Once you created a Twig Template (just like you would do for posts and pages), use a shortcode wherever you want to embed your template:
[twig-anything slug="template slug"]
It will be replaced by the rendered output of the template. Template slug is the same kind of slug that you will find for posts and pages.

Is the data cached locally?

Yes, you can optionally configure each template separately to cache the data fetched, to avoid multiple HTTP requests and file reads. The caching uses WordPress transient API. This means that if you use any caching plugin to speed up your website, Twig Anything will benefit from it as well.

What is Twig?

Twig is a very simple syntax that allows you to embed any structured data in any text, HTML etc. For example:
{{ data.weather.temperature }}
will output the temperature field of the weather object found in the data fetched from your configured data source.

Twig can filter and transform arrays and strings on the go. Read more in the Twig reference

Support

How do I get support?

I do a non-priority support through Discourse Meta, please report any issues in this topic.

I found a bug. Will it be fixed?

Yes, I hate bugs!

Can you add another source of data?

I am open to your ideas, please let me know in the forum.
PR also are welcome.

Licensing

The plugin is Open Source.

[meglio (Anton)]

How cache works

This topic discusses all the aspects of the cache engine in Twig Anything.

What is cache

For every Twig Template, there is a setting:

If you leave it empty, the data source will be fetched every time it is used in your site.

For instance, if your template reads data from some API, a new HTTP call will be made each time the template is shown in a wordpress page or post.

If 10 users open your site, 10 HTTP calls will be made. This is not good and you would likely want to retrieve data once only and then cache it in WordPress locally and reuse it.

That is where Cache lifetime in seconds comes into play.

If you set it to 60 seconds (= 1 minute), data will be re-fetched not more often than once a minute. If 10 users open your site within a minute, the local cached version will be used.

When to use cache and when not to

Caching helps to considerably reduce data fetching. But is it always needed? Actually, no.

Set cache lifetime when reading data from:

• URLs
• medium and large local files
• MySQL database with heavy queries

If you don’t cache data that comes from a URL, your website might freeze as soon as it is opened by many users simultaneously. It will also let your server eat more traffic.

Do not set cache lifetime when reading data from:

• small files, for example a CSV file with 100 lines
• WordPress MySQL database with queries (except very heavy queries that run for a second or longer)

Where cache is saved

Twig Anything uses standard WordPress transient1 mechanism to store cache.

By default, it is saved in database. However, Transients are inherently sped up by caching plugins. For example, a memcached plugin would make WordPress store transient values in fast memory instead of in the database.

This means that if you use caching plugins with your site, Twig Anything will benefit from it as well.

[meglio (Anton)]

CSV Format add-on

Download from WordPress Plugin Directory

Introduction

With this add-on, you can now read CSV lines and display them in your WordPress site.

CSV means comma-separated values.
A typical CSV file looks like this:

Belgium,Brussels,EUR,Euro,BE,.be
Belize,Belmopan,BZD,Dollar,BZ,.bz
Benin,Porto-Novo,XOF,Franc,BJ,.bj
Bhutan,Thimphu,BTN,Ngultrum,BT,.bt

You usually obtain CSV data when you export it from Excel, Google Spreadsheets and many other applications.

Features

• Filter and slice rows by any column
• Only pick the columns you want
• Upload CSV files to WordPress Media and read from there
• Control the output with a simple syntax
• Configure the delimiter, enclosure and escape characters

Screenshots

Settings screen

Template preview

Simple template code

CSV Format add-on is free - grab your copy from WordPress now!

[meglio (Anton)]

API Endpoints add-on

Construct your own WordPress API endpoints !
This workhorse is simply unbeatable when it comes to building your custom WordPress API.

---

Download now9 and convert your WordPress into an API machine.

Features

This comprehensive add-on is loaded with all features you would ever need to manage your own API. Endless possibilities at your fingertips:

Customize API URLs

http://mywordpress.com/api-endpoint/tld - change api-endpoint and tld in the plugin settings screen:

Minimal capability permission per API endpoint

The built-in WordPress roles/capabilities system is used. If you wish to restrict access to the API, you would usually do this:

• Optionally, use user role editor1 WordPress plugin to create a special role/capability, for example “”
• Create a separate WordPress user to access API (if you are the only consumer of your own API), or assign api capability to the users you want to grant API access to
• Enter the capability for your endpoint (see screenshot above).

Customize HTTP headers

By default, a few HTTP headers are set for you by default. This means that they will be always sent with every API response, unless you override them:

Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: 0

The first header allows your API to be called with using Ajax from domains other than your WordPress site’s one. The other ones prevent API responses from being cached by the caller.

Good news is that you can easily add more or override existing HTTP Headers:

In this example we add a header that tells the API caller that the output is in JSON format.

Return any data from your WordPress site

Yes, in your API endpoints you can return pretty anything: posts/pages, custom posts, comments, settings, users, site options, post metadata, users, drafts, activity and much more.

This is possible because Twig Anything can read from the MySQL database of your WordPress. This simply means that you can query any information from your site and output it in you API.

Turn any local file into an API (CSV, JSON, XML, you name it)

Because Twig Anything can read from many various sources and understand a lot of different formats, you can turn all this data into API!

JSON, XML and MySQL is understood out of the box.

And there is a number of free add-ons that add even more sources to fetch data from and turn it into API: local and remote databases, CSV1.

Fetch any data from 3rd party APIs and turn into your own API

Twig Anything can do that as well - and you can turn any Twig Template into an API Endpoint.

Boost API speed with the caching system out of the box

API Endpoints naturally benefit from the intelligent caching system of Twig Anything. Read more about How cache works in a message above in this topic.

Output API in any format, including, but not limited to, JSON, XML, RSS/ATOM, YAML and HTML.

You have full control of what API returns through Twig Templates.

For instance, this template:

… makes up a JSON response:

Control success/error templates

In the settings screen, setup your desired templates for both successful responses and failures. You get some good JSON defaults for you out of the box, but feel free to change to any other format, i.e. XML, YAML etc.

P.S. An example of a failure is an authentication error:

Intelligent template editor with code highlighting and full screen mode

[meglio (Anton)]

Retrieve and display data from WordPress database

So, you want to fetch some data from your WordPress database and then display it somewhere on your blog? Easy!

All you need to know is some basics of the MySQL and Twig syntax, which you can learn instantly and effortlessly. The fastest way is to look at the examples found in Twig Reference.

In this tutorial, let’s go through a few steps to display post status statistics like this:

• Drafts: 3
• Published: 19

In other words, we want to display the number of published vs unpublished pages.

Step 1. Create a new Twig Template

Simply click on the “Add New” menu item under the “Twig Templates” menu to create a new Twig Template. Give it a name and choose MySQL settings as by the screenshot:

What does MySQL Result Type mean? It simply allows you to choose among the available native WordPressdata retrieval methods. In short:

• get_var – get a single value
• get_row – get a single row
• get_results – get all rows from the result of the MySQL query
• get_col – get multiple rows, every one consisting of a single column only

Step 2. Build a MySQL query

Next, enter your MySQL query into the MySQL Query area. The query is pretty simple – it looks like a standard MySQL query with just a few additions in the syntax:

SELECT post_status, count(post_status) as count
FROM {{ wp_globals.wpdb.posts }}
WHERE post_type = 'page'
GROUP BY post_status

If you didn’t know, in WordPress, table names are usually prefixed with wp_. For example, wp_posts, wp_postmeta etc. However, there is no guarantee that the prefix used in your WordPress instance is wp_ . What is more, in multisite setup, each blog will use its own tables, every one named differently. That is why WordPress makes table names available in a special global variable – $ wpdb . Twig Anything makes this variable available in your queries, so that you can use it to retrieve the correct table name, like this: {{ wp_globals.wpdb.posts }}

When rendering your template, this will be transformed to wp_posts or whatever is correct for your particular blog setup. I recommend you looking at the WordPress documentation for the complete list of the table namesavailable.

Actually, Twig Anything makes all WordPress globals available for you, so, for example, you can get the month of the post being displayed: {{ currentmonth }} .

For your convenience, there are more useful things available in both MySQL queries and Twig Templates, which is covered in the templates reference.

Step 3. Build a template

Now that we know which data is retrieved from our database, we want to display it. For this to happen, we have to enter a template, which is a simple HTML with Twig syntax enabled.

<ul>
{% for stat in data %}
  <li>
  {% if stat.post_status == 'draft' %}
    Drafts:
  {% elseif stat.post_status == 'publish' %}
    Published:
  {% else %}
    {{ stat.post_status }}:
  {% endif %}
  {{ stat.count }}
  </li>
{% endfor %}
</ul>

In your templates, you can access the MySQL result by a simple syntax:

{{ data.field1.field2.etc }}

In other words, the query result is stored in the data variable, which is accessible in your template. Because we selected get_results MySQL Result Type above, we expect data to hold an array, so we can loop over it with using Twig’s for statement:

{% for stat in data %} ... {% endfor %}

Everything inside this loop can access the stat variable, which is a particular row from our MySQL query result.

Now, we can use simple Twig’s if/else statement and {{stat.post_status}} and {{stat.count}} to output data in a simple <li> HTML tag, which makes up a single item in a list.

Finally, we wrap everything in <ul>...</ul> , which makes up a complete non-enumerated list in HTML.

When done, click on the blue Publish button to make your template available and ready to use.

You can optionally preview how your template renders by clicking on the “View rendered template” button right under the title:

Step 4. Embed it anywhere in WordPress

To output your template in WordPress, just use the standard WordPress shortcodes mechanism:

[twig-anything slug="abc"]

Replace abc by the slug found under the template title:

Alternatively, copy a ready-to-use shortcode from the Shortcode panel found to the right of your template code:

Now you can output your template wherever you wish: in WordPress posts, pages, widgets, footers etc.

[meglio (Anton)]

Import data from Discourse to WordPress

Anything you can see on the Discourse website you can also get in JSON format – because Discourse is backed by a complete JSON api.

To retrieve information from Discourse and output it in WordPress, here are some steps you can follow. In our example, let us fetch the results of searching for the “image” keyword and display the list of topics in WordPress.

Step 1. Find data that you want to display in WordPress

First, open your forums home page. For demonstration purposes, we will use the Discourse Meta home page. Open Developer Tools: Ctrl+Shift+J (Window) or Command+Option+i (MacOS) or Menu -> More Tools -> Developer Tools.

Now switch to the Network tab to show HTTP requests, then click on the XHR tab. Now search for the “Image” keyword and watch for a new HTTP request appearing in the Developer Tools:

Now click on the HTTP request on the left and a lot of details will appear on the right. All you need is to copy the Request URL:

In our example, the request URL is:

https://meta.discourse.org/search/query?term=image&include_blurbs=true&_=1436341924674

Next, remove the &_=1436341924674 part from the URL and add .json right before the question mark – you will get the JSON endpoint URL that outputs the search results in the JSON format:

https://meta.discourse.org/search/query.json?term=image&include_blurbs=true

Now, open the URL in your browser and copy all the output – Right Click, then Select All, then Right Click, then Copy:

Open this tiny nline JSON editor and paste the JSON text into the left panel, so that you can see the data in a structured way on the right panel:

Finally, you will find all the data that you want to fetch and display on your website, so let us proceed to the following step.

Step 2. Create a new Twig Template

Click on the “Add New” menu under the “Twig Templates” menu and configure from where data should be retrieved:

The next step is to enter the template that will display your data the way you want it. The template is simply HTML with some Twig syntax in it.

To output any field from the data, use the {{data.field1.field2.etc}} syntax. data is a starting point keyword that is always passed to Twig and holds the whole JSON data.

In our example, we want to loop through the topics array. We should also check if there are any results available to output. So our first version of the template looks like this:

{% if data.topics is iterable %}
 
  {% for topic in data.topics %}
    - {{ topic.title }}<br/>
  {% endfor %}
 
{% else %}
  No results
{% endif %}

Here is how this template renders:

[twig-anything] shortcode error: Cannot json-decode the data retrieved. The json_decode() PHP function returned NULL.

Step 3. Make topic titles clickable

Let us make topic titles clickable. For this, we will need to use the <a> HTML tag and build a topic link with using topic id and topic slug.

{% if data.topics is iterable %}
 
  {% for topic in data.topics %}
    - <a target="_blank"
          href="https://meta.discourse.org/t/{{topic.slug}}/{{topic.id}}">
        {{ topic.title }}
      </a>
      <br/>
  {% endfor %}
 
{% else %}
  No results
{% endif %}

That easy! Now our topic titles are clickable, check it right here:

[twig-anything] shortcode error: Cannot json-decode the data retrieved. The json_decode() PHP function returned NULL.

Step 4. Embed the template anywhere in WordPress

To output your template in WordPress, just use the standard WordPress shortcodes mechanism:

[twig-anything slug="abc"]

Replace abc by the slug found under the template title:

Alternatively, copy a ready-to-use shortcode from the Shortcode panel found to the right of your template code:

Now you can output your template wherever you need in WordPress: in posts, pages, widgets, footers etc.

[meglio (Anton)]

A piece of feedback from the former support forum:

[allu (Allen)]

@meglio

it’s now FREE and OPEN SOURCE

I don’t see any way to download this plugin, only a button to buy it for USD 59. What is the status of this plugin now?

[meglio (Anton)]

There’s a link to a BitBucket repository above — please download it from there. Just got no time to update the website yet.

[Stephen]

Is it possible to manipulate data from the JSON?

For example, using the Events Plugin from @angus to add topic.event.start which is in UTC format:

2018-07-25T02:00:37+00:100:

Could we easily convert to a human-readable format in a set time zone?

[meglio (Anton)]

@Stephen

Is it possible to manipulate data from the JSON?

Data can be manipulated by using features of the Twig template language: Documentation - Twig - The flexible, fast, and secure PHP template engine

[Stephen]

Where’s the best place to file bugs with the plugin?

[meglio (Anton)]

As they are rare, I think it’s okay if you report them just here in this thread. Thanks.

[baileys2611 (Simon)]

Hi Anton, I’m trying to use Twig Anything (we purchased it today by the way) and am trying to use the wp_get_current_user_id() inside an SQL query - coming up with “Failed preparing MySQL query: The function “wp_get_current_user_id” does not exist. Did you mean “wp_get_current_user”, “get_current_blog_id”, “get_current_user_meta” in “template” at line 1”. In other words, it’s not finding the function I’m trying to use. Is that something that needs to be updated in the plugin?

[baileys2611 (Simon)]

Sorry - let me clarify the above (it’s long!) - getting the error: “The function “wp_get_current_user_id” does not exist.” when attempting to call {{ wp_get_current_user_id() }} through Twig Anything.

Thoughts?

[PatrickH (Patrick)]

is this plugin still supported ?
and is it possible to have this data also trough this plugin:

[meglio (Anton)]

I’m using the plugin on my own websites, and it is working okay, but I haven’t been developing new features for a while.

The plugin has been open sourced some time ago, so feel free to give it a try.

Contributions are welcome.

[eextra (Sylwia)]

Why doesn’t the demo(s) work?
Is the plugin okay?
Can the plugin be used with other csm’s but not only with wordpress?