Discourse plugin for static site generators like Jekyll or Octopress

(Sam Saffron) #11

Yes, this opens a big can of worms that needs to be tackled slowly

  • We need identity some how, who is this person?
  • If we allow for the traditional, you can just enter an email, how do we register the account?
  • If the account is already registered how do we validate that message was really written by that person?
  • How do they sign in using twitter/facebook/gmail ?

And so on.

(Ben T) #12

I think the simple way to handle it is the way the discourse plugin for wordpress does it already; generate a link to the relevant post and allow the user to contribute there. It’s not so simple a script to do auth across domains!

The above idea is for an admin’d user, and it was my belief that they can use the API key generated to verify who they are, like in wp-discourse. This API key could be loaded from a yml file and would evaluate in the jekyll framework when creating a post, to send that output and catch the response (which is the JSON file of the created thread,) and then use that on the page. The client-side script is happy to load the comments from the thread using the established wordpress.json file.

I had my window scrolled; I meant line 271 and not line 71 in the file wp-discourse.php.

As for a way to in-line comment, I think I’d be in over my head implementing that in a client-server linked script. I’ve mostly just tooled around with loading APIs and displaying the results at the moment. An iframe containing a formatted comments section seems like a solution, but iframes are ugly.

Finally, it’d be really cool if JSONP could be implemented so calls to an un-authed API can be done without adding a header.

(Lee_Ars) #13

Yeah, from an interactivity standpoint, I think having this function exactly like the Wordpress plug-in would be preferable—the blog entry page contains a list of the top 5 or 10 comments with author + avatar, and a link to continue the discussion. The presentation doesn’t need to differ at all from how the WP plugin functions.

Wish I was more helpful with the dev side, but anything more than printf hello world is sadly beyond me :frowning:

(Jorge Castro) #14

Thanks for working on this. I tried it and my comments got stuck with “Loading …”. I noticed that the example site has this problem too, so I am assuming something changed and broke the plugin?

(Ben T) #15

Hey there! I recently lost some data off of my development box, so the example site is pulling incorrect data. I’ll fix that soon. This is now fixed on my end.

However, it’s more likely that you did not yet set the Access-Control-Allow-Origin if it gets stuck just loading.

On a related note, if you can not set this header I’ll be writing some instructions for using a PHP script to proxy requests sometime soon. The real development though (syncing posts) is currently pending this bug. There may be another solution out there, but I’m not finding it just yet.

However, if someone writes the post handler for the discourse_api gem that would work as well!

If you’re checking this thread out, it seems that in a recent commit the CSRF error has been fixed. API calls with posting now work, and development continues. I’ll hopefully have something together fairly soon.

However, the only way around the cross site scripting error is to either set the access header, implement JSONP for api calls, or use a serverside script. I’ve had success with the last option for a different project, so if you don’t mind a PHP script on your server there is a third install method.

(Ben T) #16

Alright, I was able to get it done with no real blocking issues. It could use improvement from someone better at ruby-fu than me, as it reads the configuration file for every post instead of looping just once. I’m writing up how to install it here, because I like this editor… but it’ll be mirrored at the:

github repo.

So you want some discourse powered comments on your static site?

Well, good! This plugin combines javascript loaded on the page with a Liquid tag, written for jekyll to generate posts from your jekyll site, and then load the “best” comments from your discourse install. This is based on wp-discourse by sam. The only thing you have to bring to the table is some fancy CSS, as every jekyll site is different. Comments are created with IDs of “post#” starting at 1. Comments need to live in a div with an ID of “comments.”

Great! Let's go!

Now hold on for just a second. One of the big limitations of javascript is that it can not just pull data from another domain/subdomain. To have comments that update after your site is generated, we need to have something load them. So, we have to use one of two work-arounds to get past this. Our first option is to simply add a header to every response your server sends, permitting javscript to simply load across this restriction. However, the downsides to this are that you have to change settings with nginx/apache.

Your other option is to collect, with server-side code the responses we need. The major downside to this is you will have to allow non-static code to run where your static jekyll site lives. Kind of defeats the point in some ways. This part will be documented some time soon and is a valid way if the headers are not working out. If you’ve configured PHP, this is a script that will accomplish what’s needed. You’ll need to convert just the javascript request to be URL encoded.

For the first approach, follow the steps below. The second approach will require an edit to the javascript, and those directions will be at the bottom.

Preparing Discourse

You’ll need your API key handy (found in the admin section), and may want to create a category where posts will live. We next need to add a line of configuration to your config for nginx. The install guide leads you to create it in conf.d/discourse.conf.

add_header "Access-Control-Allow-Origin" "*";

This allows all sites to load your content via javascript. If you would like, replace the * with the public location of your jekyll install, including http://. The security of this is debated all over, but I find it safe. For more security, add the header only to your json files using a location block!

Preparing Octopress

This plugin requires httparty, a cool gem that makes HTTP requests easy. It also requires json, which is likely installed. We’ll double check by adding it anyway. To install it, simply remove your Gemfile.lock, and to your Gemfile add the lines below, and run a bundle install. You can’t just use gem install to do it.

gem 'httparty'
gem 'json'

Preparing Jekyll

*If you didn't install httparty and json above, do it here with:*
gem install httparty json

You’ll need the plugin files. Using some cool git submodules, we can add them in where needed and you can update on demand. For octopress, these are the paths, assuming you are in your octopress folder.

git submodule add https://github.com/trident523/jekyll-DiscourseBestOf.git plugins/discourse
git submodule add https://github.com/trident523/js-discourseBestOf.git source/javascripts/js

For just jekyll:

git submodule add https://github.com/trident523/jekyll-DiscourseBestOf.git _plugins/discourse
git submodule add https://github.com/trident523/js-discourseBestOf.git javascripts/js

Now, if you want the latest version just run git submodule update. Easier than it used to be! Remember when updating that the javascript file will need the correct URL.

Next, we need to add the liquid tag to one of your layouts. Edit _layouts/post.html to include

<script type="text/javascript" src="javascripts/js/js-discourse.js"></script>
<div id="comments" {% discourse_comments %}></div>

When the pages are created, this will add an attribute to the div tag called “tid”, which holds the related topic ID. We’re also loading the script right before it. This should probably be moved to your header instead, but for now let’s keep moving.

Back out of your source directory, and create a directory called _discourse. Next, edit your _config.yml to include:

discourse_api_key: Your api key
discourse_api_username: Username to post as
discourse_api_category: Category to post to
discourse_api_url: The URL of your discourse install, include http:// and no trailing slash.

Remember that spacing in a yaml file is important. Each entry is a new line, and one space after the colon.

Next, edit js-discourse.js and view the configuration at the top. Remember that java-script is run client side, so setting the update time low will hammer your server from many locations.

To recap:

  1. Add Access-Control headers to your nginx/apache config.
  2. Install httparty and json. Edit your Gemfile if you use one, otherwise install directly.
  3. Add the liquid tag to whatever layout you want comments for.
  4. Add needed information to your configuration yaml file.
  5. Add needed information to js-discourse.js
  6. Generate your site!

Congrats! You made it. You should be able to generate your site and all of your posts will sync. It’s important to note that the folder you created stores a pstore file with the title of your post. If it can’t find these for some reason, when you generate you will create new posts with duplicated content. While it won’t delete the old posts, you’ll find that it will load comments from your newly duplicated post instead of the old one. Treat these files just like your database, carefully!

It looks something like this. I didn’t style the output yet, though.

Stuff I want to fix!

1. I take the posts full contents over to discourse with no attempts to check the limits. If you don't write a lot you might like this, otherwise maybe we will both find what happens when you use the API to make a really long post! You might have to edit them down by hand for now. 2. Formatting assumes markdown. 3. It actually runs twice per post, for reasons I can't figure out.

Extra Notes

I've detailed some of the implementation in the posts above, but if you use some other kind of site generator all you need to create is a div/span with an ID of comments, and an attritube called "tid." Then, you can just load the javascript and comments will render in that space. The topic ID is the number that follows after `/t//` and is not the last number.

And if you’re really looking to build it out in some other kind of framework I’d recommend looking at the original wp-discourse. Or, just load up /t/<topicID>/wordpress.json?best=number and parse it from there.

Not Reccomended: PHP proxy instead of CORS Header

I don’t reccomend this approach at first, because it can create some security issues if your script is not secured. This likely won’t happen… but you never really know. This is the php script.

Edit line 19 of js-discourse.js from:

BASE_URL + '/t/' + $('.comments').attr('tid') + '/wordpress.json?best=' + COMMENTS


'http://<your-blog-url>/ba-simple.proxy.php?url=http%3A%2F%2F' + BASE_URL + '%2Ft%2F' + $('.comments').attr('tid') +'%2Fwordpress.json%3Fbest%3D' + COMMENTS

Then, replace every instance of a.posts with a.contents.posts.

(Lee_Ars) #17

Excellent, @trident! I’ll give it a shot this weekend when I have some time! Looking forward to it :smiley:

(Lee_Ars) #18

Started on implementing, though I’ll have to break off and do stuff a bit later.

Began with the easy stuff—the Access-Control-Allow-Origin header. Found that in a setup like mine, with a single nginx instance and about a dozen vhost files, it made the most sense to add the header in the server block of my discourse vhost definition file. That way, I’m not appending an extraneous header to all of my other sites.

(In fact, add_header has a pretty flexible scope, and if you’re feeling ambitious, you could create a regex-based Access-Control-Allow-Origin statement to allow control from multiple sites without needing a * wildcard. Fortunately, my needs are far simpler, because me and regexes have this thing where I hate them and they hate me.)

(Lee_Ars) #19

@trident - Found an issue that should be pretty easy to work around. I’ve got a blog post with a slash in the title, which causes the generator to treat it as if it were a directory and throw an exception:

Liquid Exception: directory _discourse/Openfire and SSL does not exist in post

I can rename the blog entry, though, and substitute in a hyphen instead.

(Lee_Ars) #20

Looks like it also has a problem with this entry—“Node.js, Redis, and Etherpad Lite”. I suspect the dot.

Liquid Exception: A JSON text must at least contain two octets! in post

Edit - Nope, wasn’t the dot. Shouldn’t be the comma, either, as I’ve got another entry with commas that doesn’t cause an error at all.

Edit - Actually, it’s not that post at all—it’s the post before it, this one. Lemme take a peek and see if anything about it is weird.

(Ben T) #21

I assumed issues like this would come up; as I really didn’t have a true blog’s worth of content to work with. This is easy to fix programmitically, and it should be done as of this post

And for the next post, I’m looking at it right now. It has to do with a dot in the title, of course. I’m working on this now.

(Lee_Ars) #22

Does it? I’m wondering if it’s the next post, because every time I run the rake task to generate, it gives the “We’re making a new post for title: Platforms and Value Judgments,” like it’s failing there. Full output:

## Generating Site with Jekyll
unchanged sass/screen.scss
Configuration from /home/lee/octopress/_config.yml
/home/lee/octopress/plugins/gen.rb:42: warning: else without rescue is useless
Building site: source -> public
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
We're making a new post for title:Platforms and Value Judgments
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Configuration from /home/lee/octopress/_config.yml
Liquid Exception: A JSON text must at least contain two octets! in post
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/json-1.8.0/lib/json/common.rb:155:in `initialize'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/json-1.8.0/lib/json/common.rb:155:in `new'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/json-1.8.0/lib/json/common.rb:155:in `parse'
/home/lee/octopress/plugins/gen.rb:27:in `render'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/liquid-2.3.0/lib/liquid/block.rb:94:in `block in render_all'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/liquid-2.3.0/lib/liquid/block.rb:92:in `collect'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/liquid-2.3.0/lib/liquid/block.rb:92:in `render_all'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/liquid-2.3.0/lib/liquid/block.rb:82:in `render'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/liquid-2.3.0/lib/liquid/template.rb:124:in `render'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/liquid-2.3.0/lib/liquid/template.rb:132:in `render!'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/jekyll-0.12.1/lib/jekyll/convertible.rb:101:in `do_layout'
/home/lee/octopress/plugins/post_filters.rb:167:in `do_layout'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/jekyll-0.12.1/lib/jekyll/post.rb:195:in `render'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/jekyll-0.12.1/lib/jekyll/site.rb:200:in `block in render'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/jekyll-0.12.1/lib/jekyll/site.rb:199:in `each'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/jekyll-0.12.1/lib/jekyll/site.rb:199:in `render'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/jekyll-0.12.1/lib/jekyll/site.rb:41:in `process'
/home/lee/.rbenv/versions/1.9.3-p194/lib/ruby/gems/1.9.1/gems/jekyll-0.12.1/bin/jekyll:264:in `<top (required)>'
/home/lee/.rbenv/versions/1.9.3-p194/bin/jekyll:23:in `load'
/home/lee/.rbenv/versions/1.9.3-p194/bin/jekyll:23:in `<main>'
Build Failed

edited to add—if you’d like to see the raw markdown source file of any of these entries, I’m happy to make them available.

(Lee_Ars) #23

Another thing: In spite of the success message, it’s not generating posts on the forum. Or, rather, it’s only generated two—the very first post, and the SSL/TLS post. Both posts were the first posts generated in that particular rake generate run.

I’m wondering if the create topic API is rate limited, and this initial run trips the limit.

(Ben T) #24

The issue here is it’s getting back an empty response from the server. If any post fails, it does not write anything to the file (because the post does not exist.)

I’ve been able to generate multiple posts at once with simple terms, so it’s not rate limited. In my enviroment I’m reading the production log and I’m a little perplexed on my end so I’m continuing to look into it. Sometimes it refuses to believe my API request. And sometimes in programming is never good.

You should be able to see the failed request in your production.log file, for me I’m looking at:

ActionController::ParameterMissing (key not found: raw):

even when I provide one. Can you take a look for me?

(Lee_Ars) #25

@trident Woah—whatever change you made to the last commits from ~15 mins ago appears to have fixed the octet issue—I just had a successful generate, all the way through.

(Ben T) #26

This is my first ruby app after all, and the first chance I’ve had to test it against real content. :smiley:

Sanitizing input (i.e, converting a period to the word dot, or a / to slash) is something important to look into for long term use. For example, the slash in the title would trickle down to the request for content, but completely change the URL. /posts/andSSL/TLS/ is completely different! I’ll keep testing this and updating as time goes on.

(Lee_Ars) #27

It’s pretty messy in there with the post contents, but I do see a whole bunch of WARNING: Can't verify CSRF token authenticity errors; not sure if they’re advisory or actually causing problems. (edited to add: I’m current on my Discourse code—did a git pull this afternoon, and iirc @sam has already taken a stab at addressing the CSRF issues).

I see this several times immediately after the CSRF error, but not always:

NoMethodError (undefined method `empty?' for nil:NilClass):
  lib/pretty_text.rb:234:in `block in extract_links'
  lib/pretty_text.rb:234:in `extract_links'
  app/models/topic_link.rb:95:in `block in extract_from'
  app/models/topic_link.rb:89:in `extract_from'
  lib/post_creator.rb:263:in `extract_links'
  lib/post_creator.rb:64:in `block in create'
  lib/post_creator.rb:59:in `create'
  app/controllers/posts_controller.rb:29:in `create'
  config/initializers/quiet_logger.rb:10:in `call_with_quiet_assets'
  config/initializers/silence_logger.rb:19:in `call'
  config/initializers/99-rack-cache.rb:20:in `call'

Grepping the log for ‘ParameterMissing’ returns nothing.

Lemme run generate again while tailing the log file and see what shows up. Can I delete the topics from the category, delete the contents of the _discourse directory, and force it to re-post all the topics?

(Ben T) #28

Yes, deleting the _discourse folder will cause posts to re-create. In production this is bad, because you’ll then have to merge the posts.

This also skips the validation step with un-sanitized text which could be creating these errors. However, random errors are random at best. I’d love to track them down.

Finally, the CSRF error is something on discourse’s end. We’re not using it to create posts so it probably shouldn’t print.

(Lee_Ars) #29

Deleted the .pstore files and re-generated; got three topics created out of 30. The log appears to look identical on successful ones vs. unsuccessful ones.

A post with a successfully generated topic:

Started POST "/posts" for at 2013-07-20 21:52:07 -0500
Processing by PostsController#create as HTML
  Parameters: {"api_key"=>"xxx", "api_username"=>"BigDinoBlog", "title"=>"Setting up Discourse with Passenger and Nginx", "raw"=>(post body)
WARNING: Can't verify CSRF token authenticity
Completed 200 OK in 509ms (Views: 0.3ms | ActiveRecord: 318.0ms)

A post without a topic:

Started POST "/posts" for at 2013-07-20 21:52:06 -0500
Processing by PostsController#create as HTML
  Parameters: {"api_key"=>"xxx", "api_username"=>"BigDinoBlog", "title"=>"Node.js, Redis, and Etherpad Lite", "raw"=>(post body)
WARNING: Can't verify CSRF token authenticity
Completed 429  in 55ms (Views: 0.2ms | ActiveRecord: 33.5ms)

Lemme parse through these for a second and check each of them. I’ll also take a peek at the web server log and see if anything’s actually happening there.

edit - Nope, no other errors in production.log, no errors in the web server error log.

Can I simply manually create matching categories for now? If so, no big deal.

And can I make a feature request? It’d be great if instead of actually using the blog post’s content as the message body (which results in unprocessed liquid tags and all kinds of messiness) that the post body instead simply consists of the name of the post, hyperlinked back to the post.

Alternately, there’s a description: field in the JSON segment at the beginning of each post’s markdown file, up with the title and date and stuff. It’d be even more awesome if the DIscourse post body contained the contents of that field, hyperlinked back to the post.


(Ben T) #30

Huh, that is unusual. In my enviroment which is based around the default install, I have no rate limit in place against posts. So, they can all be created at once with no worries.

You get the HTTP code 429 after three posts, which is an attempt at rate limiting requests. I’m not sure where this gets added in along the way. If the limit can be found, then we can slow each request to meet the requirement.