Discourse plugin for static site generators like Jekyll or Octopress

(Sjors) #1

It would be nice to have a plugin for static site generators like Jekyll or Octopress, since the performance and security benefits of those platforms are significant if you would take the time to learn it.

What do you guys think?

Migrating from Disqus to Discourse for static blog comments?
Proposing Jekyll as community wiki platform
Will Discourse play nicely with CMS' like 107?
(Régis Hanol) #2

Not sure I follow. What’s your use case?

(Lee_Ars) #3

I’d like this, too. I’ve got an Octopress/Jekyll blog, and I use Disqus for comments on each blog entry. Would be awesome to instead use Discourse as the comment engine, rather like how your big partner sites are using it.

If nothing else, it would save me from throwing mixed-content warnings when someone accesses my blog via https, since all content would come from the same server and I wouldn’t have to reach out to disqus to pull in comments for each page.

(Sjors) #4

Well, not so long ago I came across these static site generators and since I’m obsessed with website performance I became interested in them. I do realize there’s a trade-off in functionality (compared to a CMS) but since I don’t need that much functionality I thought maybe it fits my purposes just fine.

However I do want to use Discourse together with blogging functionality, and I’ve noticed that Drupal and Wordpress both have a plugin for the discussion integration. But Jekyll/Octopress doesn’t have a plugin (or anything else which will automate the Discourse implementation) for it yet and I wanted to request it and see if there are more people interested in this.

(Ben T) #5

I ended up giving this a whirl and came up with something halfway decent. It’s missing some key features… but it’s made with love and updates a little quicker!

  • It’s written in javascript and jQuery, so you’ll have to set an Access-Control-Allow-Origin to use it.
  • There is no integration with Jekyll, so you have to create posts by hand. I’m still trying to learn how this whole system works, and the languages used.
  • I don’t yet know of an API way to pull the thread ID from the post’s slug url. So, it just uses the title of the page. This isn’t perfect.

Give it a try, and let me know if you have any suggestions!

(Lee_Ars) #6

Nice! Will poke a bit at it over the next week or two.

(Мария Сергеева) #7

Using Discourse with static sites instead of Disqus is a interesting idea, especially if Discourse instance is hosted by Discourse.org. This way blog post not only gets comments but starts discussion.

(Jeff Atwood) #8

This is a pattern we believe in, and are very interested in supporting.

(Lee_Ars) #9

So, @trident, this is an excellent start. I think the thing it really needs is integration with Octopress’s rake tasks; ideally, from a thread creation perspective, the rake generate or rake deploy tasks could check for the presence of a thread for each post (maybe unless a post is marked with a nodiscussion tag in the header or something). I believe this could leverage the Discourse API.

I wonder if this is something Brandon Mathis (Octopress dev) would be interested in looking at?

(Ben T) #10

I’m sure it’s a somewhat simple task to create a thread, but I don’t know enough ruby to get the task done yet. I do know the steps though, and hope to keep working on a more full implementation. If someone had the ruby side of this equation that would be super.

For now, what needs to happen is a POST request to the forum’s post method, with some simple arguments (starting at line 71 on the wp-discourse.php file.) The downside for me is that this should not be done in the javascript, as it would expose the API key. If someone were to get this done, all the script would need is the topic ID placed in the #comments tag. Then, it’s fairly simple from there (and saves one API call). I’ll try to work through some ruby code, but that’s no small task.

(Sam Saffron) #11

Yes, this opens a big can of worms that needs to be tackled slowly

  • We need identity some how, who is this person?
  • If we allow for the traditional, you can just enter an email, how do we register the account?
  • If the account is already registered how do we validate that message was really written by that person?
  • How do they sign in using twitter/facebook/gmail ?

And so on.

(Ben T) #12

I think the simple way to handle it is the way the discourse plugin for wordpress does it already; generate a link to the relevant post and allow the user to contribute there. It’s not so simple a script to do auth across domains!

The above idea is for an admin’d user, and it was my belief that they can use the API key generated to verify who they are, like in wp-discourse. This API key could be loaded from a yml file and would evaluate in the jekyll framework when creating a post, to send that output and catch the response (which is the JSON file of the created thread,) and then use that on the page. The client-side script is happy to load the comments from the thread using the established wordpress.json file.

I had my window scrolled; I meant line 271 and not line 71 in the file wp-discourse.php.

As for a way to in-line comment, I think I’d be in over my head implementing that in a client-server linked script. I’ve mostly just tooled around with loading APIs and displaying the results at the moment. An iframe containing a formatted comments section seems like a solution, but iframes are ugly.

Finally, it’d be really cool if JSONP could be implemented so calls to an un-authed API can be done without adding a header.

(Lee_Ars) #13

Yeah, from an interactivity standpoint, I think having this function exactly like the Wordpress plug-in would be preferable—the blog entry page contains a list of the top 5 or 10 comments with author + avatar, and a link to continue the discussion. The presentation doesn’t need to differ at all from how the WP plugin functions.

Wish I was more helpful with the dev side, but anything more than printf hello world is sadly beyond me :frowning:

(Jorge Castro) #14

Thanks for working on this. I tried it and my comments got stuck with “Loading …”. I noticed that the example site has this problem too, so I am assuming something changed and broke the plugin?

(Ben T) #15

Hey there! I recently lost some data off of my development box, so the example site is pulling incorrect data. I’ll fix that soon. This is now fixed on my end.

However, it’s more likely that you did not yet set the Access-Control-Allow-Origin if it gets stuck just loading.

On a related note, if you can not set this header I’ll be writing some instructions for using a PHP script to proxy requests sometime soon. The real development though (syncing posts) is currently pending this bug. There may be another solution out there, but I’m not finding it just yet.

However, if someone writes the post handler for the discourse_api gem that would work as well!

If you’re checking this thread out, it seems that in a recent commit the CSRF error has been fixed. API calls with posting now work, and development continues. I’ll hopefully have something together fairly soon.

However, the only way around the cross site scripting error is to either set the access header, implement JSONP for api calls, or use a serverside script. I’ve had success with the last option for a different project, so if you don’t mind a PHP script on your server there is a third install method.

(Ben T) #16

Alright, I was able to get it done with no real blocking issues. It could use improvement from someone better at ruby-fu than me, as it reads the configuration file for every post instead of looping just once. I’m writing up how to install it here, because I like this editor… but it’ll be mirrored at the:

github repo.

So you want some discourse powered comments on your static site?

Well, good! This plugin combines javascript loaded on the page with a Liquid tag, written for jekyll to generate posts from your jekyll site, and then load the “best” comments from your discourse install. This is based on wp-discourse by sam. The only thing you have to bring to the table is some fancy CSS, as every jekyll site is different. Comments are created with IDs of “post#” starting at 1. Comments need to live in a div with an ID of “comments.”

Great! Let's go!

Now hold on for just a second. One of the big limitations of javascript is that it can not just pull data from another domain/subdomain. To have comments that update after your site is generated, we need to have something load them. So, we have to use one of two work-arounds to get past this. Our first option is to simply add a header to every response your server sends, permitting javscript to simply load across this restriction. However, the downsides to this are that you have to change settings with nginx/apache.

Your other option is to collect, with server-side code the responses we need. The major downside to this is you will have to allow non-static code to run where your static jekyll site lives. Kind of defeats the point in some ways. This part will be documented some time soon and is a valid way if the headers are not working out. If you’ve configured PHP, this is a script that will accomplish what’s needed. You’ll need to convert just the javascript request to be URL encoded.

For the first approach, follow the steps below. The second approach will require an edit to the javascript, and those directions will be at the bottom.

Preparing Discourse

You’ll need your API key handy (found in the admin section), and may want to create a category where posts will live. We next need to add a line of configuration to your config for nginx. The install guide leads you to create it in conf.d/discourse.conf.

add_header "Access-Control-Allow-Origin" "*";

This allows all sites to load your content via javascript. If you would like, replace the * with the public location of your jekyll install, including http://. The security of this is debated all over, but I find it safe. For more security, add the header only to your json files using a location block!

Preparing Octopress

This plugin requires httparty, a cool gem that makes HTTP requests easy. It also requires json, which is likely installed. We’ll double check by adding it anyway. To install it, simply remove your Gemfile.lock, and to your Gemfile add the lines below, and run a bundle install. You can’t just use gem install to do it.

gem 'httparty'
gem 'json'

Preparing Jekyll

*If you didn't install httparty and json above, do it here with:*
gem install httparty json

You’ll need the plugin files. Using some cool git submodules, we can add them in where needed and you can update on demand. For octopress, these are the paths, assuming you are in your octopress folder.

git submodule add https://github.com/trident523/jekyll-DiscourseBestOf.git plugins/discourse
git submodule add https://github.com/trident523/js-discourseBestOf.git source/javascripts/js

For just jekyll:

git submodule add https://github.com/trident523/jekyll-DiscourseBestOf.git _plugins/discourse
git submodule add https://github.com/trident523/js-discourseBestOf.git javascripts/js

Now, if you want the latest version just run git submodule update. Easier than it used to be! Remember when updating that the javascript file will need the correct URL.

Next, we need to add the liquid tag to one of your layouts. Edit _layouts/post.html to include

<script type="text/javascript" src="javascripts/js/js-discourse.js"></script>
<div id="comments" {% discourse_comments %}></div>

When the pages are created, this will add an attribute to the div tag called “tid”, which holds the related topic ID. We’re also loading the script right before it. This should probably be moved to your header instead, but for now let’s keep moving.

Back out of your source directory, and create a directory called _discourse. Next, edit your _config.yml to include:

discourse_api_key: Your api key
discourse_api_username: Username to post as
discourse_api_category: Category to post to
discourse_api_url: The URL of your discourse install, include http:// and no trailing slash.

Remember that spacing in a yaml file is important. Each entry is a new line, and one space after the colon.

Next, edit js-discourse.js and view the configuration at the top. Remember that java-script is run client side, so setting the update time low will hammer your server from many locations.

To recap:

  1. Add Access-Control headers to your nginx/apache config.
  2. Install httparty and json. Edit your Gemfile if you use one, otherwise install directly.
  3. Add the liquid tag to whatever layout you want comments for.
  4. Add needed information to your configuration yaml file.
  5. Add needed information to js-discourse.js
  6. Generate your site!

Congrats! You made it. You should be able to generate your site and all of your posts will sync. It’s important to note that the folder you created stores a pstore file with the title of your post. If it can’t find these for some reason, when you generate you will create new posts with duplicated content. While it won’t delete the old posts, you’ll find that it will load comments from your newly duplicated post instead of the old one. Treat these files just like your database, carefully!

It looks something like this. I didn’t style the output yet, though.

Stuff I want to fix!

1. I take the posts full contents over to discourse with no attempts to check the limits. If you don't write a lot you might like this, otherwise maybe we will both find what happens when you use the API to make a really long post! You might have to edit them down by hand for now. 2. Formatting assumes markdown. 3. It actually runs twice per post, for reasons I can't figure out.

Extra Notes

I've detailed some of the implementation in the posts above, but if you use some other kind of site generator all you need to create is a div/span with an ID of comments, and an attritube called "tid." Then, you can just load the javascript and comments will render in that space. The topic ID is the number that follows after `/t//` and is not the last number.

And if you’re really looking to build it out in some other kind of framework I’d recommend looking at the original wp-discourse. Or, just load up /t/<topicID>/wordpress.json?best=number and parse it from there.

Not Reccomended: PHP proxy instead of CORS Header

I don’t reccomend this approach at first, because it can create some security issues if your script is not secured. This likely won’t happen… but you never really know. This is the php script.

Edit line 19 of js-discourse.js from:

BASE_URL + '/t/' + $('.comments').attr('tid') + '/wordpress.json?best=' + COMMENTS


'http://<your-blog-url>/ba-simple.proxy.php?url=http%3A%2F%2F' + BASE_URL + '%2Ft%2F' + $('.comments').attr('tid') +'%2Fwordpress.json%3Fbest%3D' + COMMENTS

Then, replace every instance of a.posts with a.contents.posts.

(Lee_Ars) #17

Excellent, @trident! I’ll give it a shot this weekend when I have some time! Looking forward to it :smiley:

(Lee_Ars) #18

Started on implementing, though I’ll have to break off and do stuff a bit later.

Began with the easy stuff—the Access-Control-Allow-Origin header. Found that in a setup like mine, with a single nginx instance and about a dozen vhost files, it made the most sense to add the header in the server block of my discourse vhost definition file. That way, I’m not appending an extraneous header to all of my other sites.

(In fact, add_header has a pretty flexible scope, and if you’re feeling ambitious, you could create a regex-based Access-Control-Allow-Origin statement to allow control from multiple sites without needing a * wildcard. Fortunately, my needs are far simpler, because me and regexes have this thing where I hate them and they hate me.)

(Lee_Ars) #19

@trident - Found an issue that should be pretty easy to work around. I’ve got a blog post with a slash in the title, which causes the generator to treat it as if it were a directory and throw an exception:

Liquid Exception: directory _discourse/Openfire and SSL does not exist in post

I can rename the blog entry, though, and substitute in a hyphen instead.

(Lee_Ars) #20

Looks like it also has a problem with this entry—“Node.js, Redis, and Etherpad Lite”. I suspect the dot.

Liquid Exception: A JSON text must at least contain two octets! in post

Edit - Nope, wasn’t the dot. Shouldn’t be the comma, either, as I’ve got another entry with commas that doesn’t cause an error at all.

Edit - Actually, it’s not that post at all—it’s the post before it, this one. Lemme take a peek and see if anything about it is weird.