We have Discourse set to send comments to WordPress via webhook, using the Sync Comment Data option in the WordPress Discourse plug-in. That has, for the most part, worked fine. Recently, however, we’ve seen some situations where we’ll post an article in WordPress and the topic will be created in Discourse, but WordPress won’t display a link to create the first post. I’ve seen both nothing show up in that area, and “Comments aren’t available for this post.” Other times we’ll get the proper link, which in our case is “Join the discussion in the TidBITS Discourse forum,” but posts made in that topic won’t migrate back to WordPress.
We are using CloudFlare, but I put it in dev mode so it wasn’t caching anything, changed some posts on Discourse, and saw no difference. So I turned CloudFlare back on.
Next I turned off the Sync Comment Data checkbox and waited 10 minutes. That fixed things to a large extent—the posts that didn’t have the initial link to Discourse got them, and one post that had only the starting link brought in the posts associated with it. There was another article that knew it had two comments in WordPress, even though there were four in Discourse, and that didn’t update. I’ve fiddled with that topic in Discourse to see if that causes it to refresh.
So it seems that the Sync Comment Data webhook just isn’t working reliably. However, when I go look at the webhook’s events page, everything claims to have posted successfully.
I’m not too perturbed about a several-minute delay in syncing of comments from Discourse back to WordPress if the Sync Comment Data webhook can’t be relied on. But are there any other downsides to not using it? There was a comment about more API calls, but I don’t know if that’s a real problem or not.
Or, is there some fix for the Sync Comment Data webhook that I should be looking into?
Unless I’m wrong, the way the plugin is working is to listen any post event, and when it happens, it marks the concerned Wordpress Post to pull the latest comments from Discourse the next time the Post content is loaded.
The immediate issue I could see is you might hit the rate limit for the current API user.
Is the API user an admin? (it has generally less restrictions)
It looks like you are using the normal rates limits.
The biggest issue with not using the webhook is if you are displaying comment numbers for posts on an index page. Loading an index page that displays comment numbers can result in a lot of API calls to Discourse. With the webhook, this problem is completely eliminated. If you are not displaying comment numbers on your site, not using the webhook won’t result in many extra API calls to Discourse.
I’m unsure about why the webhook isn’t working reliably for you. I’m tempted to think the problem is related to Cloudflare. Is there any pattern that you can see for when it works correctly and when it doesn’t?
Perhaps this is relevant, then. We are displaying comment numbers on articles and on the home page (so that might entail a LOT of API calls), and I’ve noticed in a few cases where the comment number is correct but fewer posts appear below the article. That’s still true here for instance, where the comment number is correctly reported as 4 but only two posts show:
Unfortunately, I haven’t been able to identify any pattern with when it has worked and when it hasn’t. We’ve been using CloudFlare the entire time, and most settings have been quite static during this time (we brought the site online in April 2018). I recently turned off CloudFlare’s Auto Minify JavaScript and Brotli compression to work around posts not refreshing quickly after editing, but I can’t see how that would be related. And I think the problem predates that change anyway.
The comment number may be different than the number of comments displayed on WordPress. Have a look at the plugin’s Commenting options tab to see if you have set any options that could prevent comments from being pulled to Discourse.
No, all my Commenting options are set to bring in everything—there are no exclusions by user level or the like. And most of the time, the comment number does match up.
After I turned off the Sync Comment Data webhook in the WordPress plugin, that webhook over on Discourse showed a bunch of 404 error codes instead of 200 successful codes. Not unreasonable, I thought, but after you mentioned the problem with the API calls, I turned Sync Comment Data back on and redelivered a post in the thread I mentioned earlier that was out of sync.
It succeeded, and now when you look at it, it says five comments and displays five comments. So the basic communication for the webhook is working, just not reliably, and I can’t figure out any way to get it to do so, even with a manual trigger (this sort of redelivering didn’t work before with an event that had a 200 code to start).
For our API key, it’s currently set up as All Users. Is that wrong?
I don’t quite understand what behavior would generate an API request. Would the comment number that @simon was talking about below be counted? Or an article load? Our site is moderately busy—about 175,000 page views per month—but our Discourse install is relatively lightly used, I think, with about 700 posts per month.
I meant the Publishing Username you have provided in the options, whether it’s an admin or not.
If a normal user and your forum busy (people comment a lot in a short time), then this could be the reason. I don’t know if it’s really that, but the way you’re saying it happens sometimes and it’s working after a few moment makes me think this is related.
@simon I think it would be a good idea to alert by mail one time if it happens that a request failed with a ‘too many requests’ response.This would make the admin aware that the limit needs to be increased and clear any wild guessing.
Ah! So yes, the Publishing Username is an admin user, but the posts in Discourse that are created from WordPress posts are linked to the author of the article, and those people aren’t admins. But we’re still talking about 1-3 WordPress posts per day, and probably no more than 5-10 Discourse comments on those posts, so I wouldn’t think it would run afoul of rate limits.
The webhook isn’t affected by rate limiting. I’d like to know why it is working intermittently though. If you find out any more about what’s going on, please let us know. I’ll keep my eye out for any similar reports.
I’m going to turn off Sync Comment Data again, at least temporarily, so this article can get linked to its comment thread in Discourse, which was created correctly.
Clicking Redeliver didn’t make any difference.
Oh, and for what it’s worth, I disabled CloudFlare for the talk.tidbits.com domain, so it should just be passing traffic through now.
Other ideas about how to troubleshoot this welcome… Thanks!
Just to keep this thread active, this problem continues. It comes and goes in severity, but a whole bunch of recent WordPress posts didn’t get any links to their Discourse topics over the last few days. I just get “Comments are not currently available for this post.” even though our design also shows the number of comments in the topic, and it’s non-zero.
This is one of the articles in question, although I’ve turned off the webhook checkbox so perhaps the API will have fixed it up by the time someone looks. It usually does.
Any new thinking about how to troubleshoot this, @simon?
Data point! Since there was a Redis-related bug with the WordPress plugin recently that was preventing topics from being created, I thought I’d flush the Redis cache and see what happened. When I did that (keep in mind that webhooks are still off), the comment links appeared instantly on all the relevant articles.
So I’d suggest that there’s something going wrong in the Discourse WordPress plugin related to caching here. Does that help, @simon?
I posted another article today that had “Comments are not available for this post” under it, instead of the link to the Discourse topic, so I turned the webhook setting back on and then flushed the Redis cache. No change at all. But when I turned the webhook setting back off, the link appeared immediately.
So it’s conceivable that there are two issues here:
Some sort of bug with caching that prevents API calls from being made.
Webhooks not working reliably, regardless of caching.
Thanks for looking into this! I’ll take a close look at it tomorrow. Are you using a plugin for Redis caching? If so, what is the name of the plugin?
There’s also an issue with the plugin not properly handling the case where webhook requests fail for any reason. If webhooks are failing to trigger updates to the comments, the plugin needs to fall back to updating comments based on the amount of time that has passed since they were last updated. There’s a small change that can be made to fix this issue.
I think this was the issue. Before comments are synced, a call is made to a MYSQL locking function. The results of that call were being cached with wp_cache_set and retrieved with wp_cache_get. The WordPress caching functions use the WordPress Object Cache. Caching plugins like Redis Cache override the WordPress Object Cache.
I’ve updated the plugin to remove the WordPress caching functions from the call to the MYSQL lock. Can you let me know if updating to version 1.8.9 of the plugin fixes the issue for you?
Thanks for letting us know. The problem that the old code was causing was fairly obvious once I realized how caching plugins override the WordPress Object Cache.
