Discourse Sitemap Plugin


(Richard - DiscourseHosting.com) #1

Ok, our Sitemap plugin is finished.

There has been some discussion over the need of a sitemap, and I think Discourse doesn’t really need one, although I do believe that a forum might be indexed a bit faster with one.

Anyway, we created this plugin because of a customer that needed a sitemap for Google News, and this trick didn’t work because they were using Amazon S3 to store their uploads (which broke the same domain requirement)

Your can find our plugin at GitHub - discoursehosting/discourse-sitemap

Just enable it using the site setting in the Plugins menu, and you will get

  • /sitemap.xml
  • /sitemap/news.xml
  • A Sitemap line in robots.txt

Ideas for improvements:

Let me know if you have any issues or ideas!


How to Create a Sitemap in Discourse?
SEO - Google optimisation through webmaster
Massive traffic drop from Google searches after migrating from myBB
Sitemap.xml for Google Webmaster
1 million Topics - Takes millions of days to get indexed without Sitemap in Robots
Big drop in Google traffic after migration
Help using Link Centaur API?
Sitemap plugin issue 502 Bad Gateway - in /sitemap.xml page
Sitemap.xml for Google Webmaster
Massive traffic drop from Google searches after migrating from myBB
Upload sitemap.xml
How to Noindex thin & duplicate pages in Discourse?
#2

Didn’t work for me. I’m using a digital ocean solution comprising of docker based discourse installation. Plugin was already enabled but when I went to the /sitemap.xml link nothing was there - just an apologetic redirect page.


(Richard - DiscourseHosting.com) #3

I’m sorry, but that’s not something I can work with. Please verify that you did install the plugin successfully and specify more details, like relevant excerpts of your logging.


(Freso) #4

I guess we’re seeing the same at https://community.metabrainz.org/ - going to /sitemap.xml generates this error log: gist:e44a009fe79cd0572664 · GitHub

“External Sitemap” shows up as enabled on /admin/plugins.

Edit: @RGJ, updated log link.


(Richard - DiscourseHosting.com) #5

Those logs are only visible for the forum admin, can you please copy/paste the relevant content?


(Richard - DiscourseHosting.com) #6

It apparently is not since the error is generated because it thinks it’s turned off.

Can you please toggle the Sitemap Enabled setting (turn if off and then back on) ?


#7

Take a look at http://www.aibuapp.com/ sitemap.xml for an example of the message displayed on the website.

Here’s the error log:

ActionController::RoutingError (Not Found) /var/www/discourse/plugins/discourse-sitemap/plugin.rb:50:in `generatesitemap’

/var/www/discourse/plugins/discourse-sitemap/plugin.rb:50:in `generatesitemap'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:198:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rendering.rb:10:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:117:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:505:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/callbacks.rb:19:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rescue.rb:29:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `block in instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/notifications.rb:164:in `instrument'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/abstract_controller/base.rb:137:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionview-4.2.5.2/lib/action_view/rendering.rb:30:in `process'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.9.2/lib/mini_profiler/profiling_methods.rb:102:in `block in profile_method'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:196:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_controller/metal.rb:237:in `block in action'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:43:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:43:in `block in serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/journey/router.rb:30:in `serve'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/routing/route_set.rb:815:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:186:in `call!'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/strategy.rb:164:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/omniauth-1.3.1/lib/omniauth/builder.rb:63:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/conditionalget.rb:25:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'
/var/www/discourse/lib/middleware/anonymous_cache.rb:129:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/flash.rb:260:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:225:in `context'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/session/abstract/id.rb:220:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/cookies.rb:560:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/query_cache.rb:36:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activerecord-4.2.5.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/activesupport-4.2.5.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/logster-1.1.1/lib/logster/middleware/reporter.rb:31:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:38:in `call_app'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/rack/logger.rb:22:in `call'
/var/www/discourse/config/initializers/100-quiet_logger.rb:10:in `call_with_quiet_assets'
/var/www/discourse/config/initializers/100-silence_logger.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/actionpack-4.2.5.2/lib/action_dispatch/middleware/request_id.rb:21:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/runtime.rb:18:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/sendfile.rb:113:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-mini-profiler-0.9.9.2/lib/mini_profiler/profiler.rb:281:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/message_bus-2.0.0.beta.5/lib/message_bus/rack/middleware.rb:60:in `call'
/var/www/discourse/lib/middleware/request_tracker.rb:73:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/engine.rb:518:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/application.rb:165:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `public_send'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/railties-4.2.5.2/lib/rails/railtie.rb:194:in `method_missing'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:66:in `block in call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `each'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/urlmap.rb:50:in `call'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:562:in `process_client'
/var/www/discourse/lib/scheduler/defer.rb:85:in `process_client'
/var/www/discourse/lib/middleware/unicorn_oobgc.rb:95:in `process_client'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:658:in `worker_loop'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:508:in `spawn_missing_workers'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/lib/unicorn/http_server.rb:132:in `start'
/var/www/discourse/vendor/bundle/ruby/2.0.0/gems/unicorn-5.0.1/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:22:in `load'
/var/www/discourse/vendor/bundle/ruby/2.0.0/bin/unicorn:22:in `<main>'

(Richard - DiscourseHosting.com) #8

the error is generated because it thinks it’s turned off.

Can you please toggle the Sitemap Enabled setting (turn if off and then back on) ?


#9

Thats done the trick and working now!


(Zas) #10

I enabled the plugin through Settings > Plugins > Sitemap enabled and guess what … it works of course :wink:

The bug is the state report in Plugins > Plugins, where it is always reported as : “External Sitemap 0.1 Y” <— enabled, whatever you do in Settings.


(Mittineague) #11

I think that might be a simple fix.

Try adding
enabled_site_setting :sitemap_enabled
to the plugin.rb file

EDIT
my testing confirms


(Richard - DiscourseHosting.com) #12

Pushed the fix, thanks a million @Mittineague !


(Mittineague) #13

No problem, happy to be of some help.

Looks like the plugin does a very good job. :thumbsup:

I was wondering about why sitemap vs. newssitemap handled quotation marks differently/
One had " the other had the entity &quot;

Maybe putting the post text content inside CDATA would be a good idea?


(Richard - DiscourseHosting.com) #14

Are you sure? Sitemap has only a URL, newssitemap has URL and title… Do you have an example?

Yes, CDATA for the title is a good idea indeed, will implement that this week, thanks for the suggestion!


(Mittineague) #15

sitemap has only loc and lastmod, the first uses the slug-ified URL the latter is datetime.

So indeed it was the title as browser-rendered-view vs. view-source I was thinking of

<url>
  <loc>http://localhost:4000/t/decorator-badge/48/</loc>
  <news:news>
    <news:publication>
      <news:name>Discourse Test Site</news:name>
      <news:language>en</news:language>
    </news:publication>
    <news:publication_date>2016-03-12T07:56:37Z</news:publication_date>
    <news:title>"Decorator" Badge</news:title>
  </news:news>
</url>

.

<url>
    <loc>http://localhost:4000/t/decorator-badge/48/</loc>
     <news:news>
        <news:publication>
            <news:name>Discourse Test Site</news:name>
            <news:language>en</news:language>
        </news:publication>
        <news:publication_date>2016-03-12T07:56:37Z</news:publication_date>
        <news:title>&quot;Decorator&quot; Badge</news:title>
    </news:news>
</url>

Now that I think about it, what are the chances of topic titles having characters that might be interpreted as HTML and break the XML?

And if < is entitized to &lt; then CDATA wouldn’t be needed anyway.


(Freso) #16

Pretty big. ", ', & & all need to be escaped for the XML to be valid (in addition to < and > of course). Those are not exactly rare characters to use.

See also:


(Richard - DiscourseHosting.com) #17

Topic titles are already entitized when they’re written in the view. So there is no issue here.


(Daniel Marquard) #18

It doesn’t look like I’m allowed to propose a change on your repo, but the topic URLs in the sitemap don’t exactly match canonical URLs. Mine have a trailing slash after the topic ID, but this should ideally be omitted in the sitemap.

Thanks for the plugin! Works just as it should otherwise.


(Daniel Marquard) #19

I’m a bit late on this, but when I didn’t hear back, I forked this plugin to fix the trailing slash bug. I will be supporting this fork if anyone is interested in using it themselves.

EDIT: it’s been fixed in the plugin author’s repo.


(Richard - DiscourseHosting.com) #20

I totally missed your remark 6 months ago, I’m sorry.
Will merge your changes into my repo.

BTW I do appreciate your fixes but please leave any copyright notices intact when you fork something.
I mean, you removed a single character in the code and that warranted to change the copyright in the readme and the URL in the sitemap files?