Almost 3 weeks ago we migrated an old forum from myBB to Discourse, bringing all the content and creating proper 301 redirects for all old threads, categories and topics. Since the move, our incoming traffic from search engine has dropped over 20%, so far inexplicably.
Our SEO consultant has highlighted that serving different content to the crawler may be one of the reasons that Google treats the new forums so poorly compared to the past. The version seen by Google of a topic page for example contains less content than what a browser sees (no indication of reply, login, search or suggested topics, among other things.)
Has anybody noticed similar traffic drops after a migration from âold-schoolâ html forums? Any suggestion on how to fix things on our forums?
I was hoping youâd have good news did that mass migration from Ning include 301 redirects? This shouldnât happen, itâs not normal that traffic drops as dramatically when the content is migrated following SEO best practice. A drop in pagevies I expected (as there are much less pages in Discourse than mybb) but a drop in sessions and users is not normal. Somethingâs off. Any other thoughts of whatâs wrong?
Upon diving into this issue further, I found that what Discourse is doing is called cloaking and itâs considered a violation of Googleâs Webmaster Guidelines because it provides our users with different results than they expected. (from Cloaking - Search Console Help)
Iâm quite disappointed to find this out. Any way to simply disable this cloak?
What discourse does is a case of this paragraph from âHidden Text and Linksâ:
However, not all hidden text is considered deceptive. For example, if your site includes technologies that search engines have difficulty accessing, like JavaScript, images, or Flash files, using descriptive text for these items can improve the accessibility of your site. Remember that many human visitors using screen readers, mobile browsers, browsers without plug-ins, and slow connections will not be able to view that content either and will benefit from the descriptive text as well.
Note, we are not cloaking, if you disable javascript you will see the exact same text as Google sees. (Which is also the same text that is on the page)
What you could experiment with is the sitemap plugin, Discourse Sitemap Plugin, cause now that tons has moved Google may be playing catchup.
Catch up with what? The content didnât move, itâs all there where it was and itâs 301 redirected. I was not expecting traffic to drop. If nothing else, I was expecting it to stay stable until Google started realizing that the pages were responsive, usable on mobile, etc. and therefore see better search results. And the drop is not insignificant, weâre talking 20% less visits from search results week over week in the past 2 weeks.
I showed you the graph above, that indicated one year, if you are unwilling to wait that long perhaps just switch things back to the way they were and keep it that way if you prefer safety.
Plus crawling with JavaScript enabled will be way slower than disabled cause it costs Google resources to generate HTML for the pages.
The content did move, in fact, you just said it is all 301 redirected.
I think it does make sense to make it configurable what we consider a âcrawlerâ (especially if new crawler types pop up). If you want to be a guinea pig and disable crawler special behavior for Googlebot feel free to.
I agree with @codinghorror here though, if you are not willing to work with us, probably best for you to migrate back to your previous software.
Iâm more than willing to work with you folks, donât get me wrong: I picked Discourse after a long selection process. I simply have a problem and Iâm looking for solutions: traffic is dropping at an alarming rate and itâs not clear why. The main suggestions I got from our SEO consultants is that âsort-of-cloakingâ (for lack of better term) but Iâm hoping that having this conversation with you can lead to solving a problem.
I found that, thank you. I installed it on a clone site, produced the maps, fixed the URL to the correct domain and submitted to Google Webmaster. Iâll let you know when theyâre processed.
Iâd be interested to run your commit but Iâd have to do it on an experimental site, I canât run that in production⊠Iâll put together a dev environment tomorrow and run some tests.
Thatâs a very alarming chart, Iâm concerned you take it so lightly. âWait for a yearâ while a crucial KPI of my community gets down the drain is a disappointing option.
Going back to finding solutions to the problem, am I understanding it right that youâre saying that the drop in search traffic is simply to be expected? Migrating a community to Discourse will take ~1 year before traffic goes back to pre-migration levels? Even if one migrates keeping the same domain, 301 redirects and no other screw ups?
Not an expert on SEO here or anything, but are you sure that Google is treating your site as it was?
Even when youâre 301-ing everything, the pages come out different. It is obvious to Google that your content is no longer the same (even though the texts are the same). So it is not a simple 301. 301 means âmovedâ. This is âmovedâ and âchangedâ.
If I were Google, Iâd be very cautious about a site with high ranking suddenly having all its links point to some other pages which are not at all similar to the old pages. If I were Google, Iâd treat these as ânewâ pages and maybe start counting from zero or something.
I used for several years participated in the development of search engines. And a little bit familiar with search engines, structutral index, rank them. 3 months is really very little time. I want to say that the drop in traffic is a natural process. With the transition to the new script you completely change the site structure. There is a redirect or not, doesnât matter. Ljuboja search script will react to this. In the Discourse there are different places for improvement from the point of view of compliance with the requirements of search engines. I wrote about this previously, for example, on many pages there is no Title. It is not very good can affect the ranking of these pages. But itâs the little details, and it canât say so globally (down traffic). You need to wait at least 6 months to draw conclusions. Analyze the indexes of search engines, using the mechanisms provided, for example, in Google. Sorry for my English, but I think my idea is clear.
are clear to me: I knew that switching to Discourse would have an impact. But I thought it would be a positive one, or at best neutral.
The old forum were made of pages that looked old, not-mobile friendly, with bad URL. With the switch to Discourse Googlebot now should see beautiful responsive, mobile-first pages⊠So Google notices a switch to a better URL structure, same content, pretty presentation and penalizes the new pages?
OLD_URL has some accrued relevance score in Google. It has not changed in a long time, so Google is going to wait quite awhile before it crawls it again. NEW_URL, of course, has identical content to OLD_URL but no relevance score.
We both know that OLD_URL has been changed to a 301 redirect leading to NEW_URL, but Google wonât know that until it re-crawls it, which itâs not going to do very often. So instead of seeing a moved page, Google thinks thereâs a copied page. That wonât change until it re-crawls all of the old pages and sees the 301s.
Maybe you could make a sitemap with the old URLs and a recent updated date, so that Googlebot will recrawl all of the old URLs and learn about the 301 redirects?
Ranking in search engines depends on a number of parameters.
To the old version of the site Google has developed a level of trust. He determined the frequency of scans and calculated the weight of the pages, etc⊠And everything has changed. How to respond to this search engine? At least, mistrust. This reaction is common.
Redirect only showed that a new page has changed address. But to search engines these pages have changed! They do not become better or worse (depending on many factors), they just became different.
The weight of the pages (PR), everything has changedâŠ
Like we keep saying, you need to wait at least three months. At least. Maybe more. If you are looking for quick fixes, SWITCH BACK TO YOUR OLD SOFTWARE.
Another area Iâm investigating is the default nginx rate limits as well, as maybe Googlebot is hitting them? Thanks for the help everybody, got plenty of ideas already.
@sam I have applied your patch but to another forum that I am building out from scratch. Iâm not comfortable applying it in production though as at the moment I donât think thatâs the root cause of the dropping traffic.
OK. You go to a grocery store all the time. Ad old, mom-n-pop style, old-60âs store. Very reputable. You trust it. You tell your friends that their goods there have good quality and come from reputable sources.
Then they moved to a new, shiny location. With a brand-new store, and all new faces. On the door of the old store thatâs a notice: â301 - Weâve moved!â
Inside the brand-new store, they sell exactly the same stuff. But you donât recognize the store any more. Sure, itâs modern and great and everything, but do you TRUST it? Will you recommend it to all your friends?
Or will you observe for a while? Do you smell something fishy is going on? Are these the same people? Or have they sold out to a money-grabbing new MNC?
In other words, will the new storeâs Page Rank be the same as the old store?
How would you describe when many of the top performing pages have lost their top-10 status in the Google results page and are now on page 2?
Google is indexing the site at the rate of 8-10k pages a day, up from the 2k/day average before the migration to Discourse. Googlebot is getting pages a lot faster than before, from 800ms to 70ms with Discourse. Everything seems to be going fine except the massive traffic drop, which keeps on dropping.
Is it possible that the default rate-limiting protections are slowing down googlebot?
did that mass migration from Ning include 301 redirects? This shouldnât happen, itâs not normal that traffic drops as dramatically when the content is migrated following SEO best practice. A drop in pagevies I expected (as there are much less pages in Discourse than mybb) but a drop in sessions and users is not normal. Somethingâs off. Any other thoughts of whatâs wrong?