I showed you the graph above, that indicated one year, if you are unwilling to wait that long perhaps just switch things back to the way they were and keep it that way if you prefer safety.
Discourse has been around way before 2015 afaik.
The content did move, in fact, you just said it is all 301 redirected.
I think it does make sense to make it configurable what we consider a “crawler” (especially if new crawler types pop up). If you want to be a guinea pig and disable crawler special behavior for Googlebot feel free to.
I agree with @codinghorror here though, if you are not willing to work with us, probably best for you to migrate back to your previous software.
I’m more than willing to work with you folks, don’t get me wrong: I picked Discourse after a long selection process. I simply have a problem and I’m looking for solutions: traffic is dropping at an alarming rate and it’s not clear why. The main suggestions I got from our SEO consultants is that ‘sort-of-cloaking’ (for lack of better term) but I’m hoping that having this conversation with you can lead to solving a problem.
I found that, thank you. I installed it on a clone site, produced the maps, fixed the URL to the correct domain and submitted to Google Webmaster. I’ll let you know when they’re processed.
I’d be interested to run your commit but I’d have to do it on an experimental site, I can’t run that in production… I’ll put together a dev environment tomorrow and run some tests.
That’s a very alarming chart, I’m concerned you take it so lightly. “Wait for a year” while a crucial KPI of my community gets down the drain is a disappointing option.
Going back to finding solutions to the problem, am I understanding it right that you’re saying that the drop in search traffic is simply to be expected? Migrating a community to Discourse will take ~1 year before traffic goes back to pre-migration levels? Even if one migrates keeping the same domain, 301 redirects and no other screw ups?
Not an expert on SEO here or anything, but are you sure that Google is treating your site as it was?
Even when you’re 301-ing everything, the pages come out different. It is obvious to Google that your content is no longer the same (even though the texts are the same). So it is not a simple 301. 301 means “moved”. This is “moved” and “changed”.
If I were Google, I’d be very cautious about a site with high ranking suddenly having all its links point to some other pages which are not at all similar to the old pages. If I were Google, I’d treat these as “new” pages and maybe start counting from zero or something.
I used for several years participated in the development of search engines. And a little bit familiar with search engines, structutral index, rank them. 3 months is really very little time. I want to say that the drop in traffic is a natural process. With the transition to the new script you completely change the site structure. There is a redirect or not, doesn’t matter. Ljuboja search script will react to this. In the Discourse there are different places for improvement from the point of view of compliance with the requirements of search engines. I wrote about this previously, for example, on many pages there is no Title. It is not very good can affect the ranking of these pages. But it’s the little details, and it can’t say so globally (down traffic). You need to wait at least 6 months to draw conclusions. Analyze the indexes of search engines, using the mechanisms provided, for example, in Google. Sorry for my English, but I think my idea is clear.
are clear to me: I knew that switching to Discourse would have an impact. But I thought it would be a positive one, or at best neutral.
The old forum were made of pages that looked old, not-mobile friendly, with bad URL. With the switch to Discourse Googlebot now should see beautiful responsive, mobile-first pages… So Google notices a switch to a better URL structure, same content, pretty presentation and penalizes the new pages?
Since Google doesn’t strictly penalize 301 redirects I’m looking for other reasons that may affect SEO negatively.
OLD_URL has some accrued relevance score in Google. It has not changed in a long time, so Google is going to wait quite awhile before it crawls it again. NEW_URL, of course, has identical content to OLD_URL but no relevance score.
We both know that OLD_URL has been changed to a 301 redirect leading to NEW_URL, but Google won’t know that until it re-crawls it, which it’s not going to do very often. So instead of seeing a moved page, Google thinks there’s a copied page. That won’t change until it re-crawls all of the old pages and sees the 301s.
Maybe you could make a sitemap with the old URLs and a recent updated date, so that Googlebot will recrawl all of the old URLs and learn about the 301 redirects?
Go to any other script have the same effect.
Ranking in search engines depends on a number of parameters.
To the old version of the site Google has developed a level of trust. He determined the frequency of scans and calculated the weight of the pages, etc… And everything has changed. How to respond to this search engine? At least, mistrust. This reaction is common.
Redirect only showed that a new page has changed address. But to search engines these pages have changed! They do not become better or worse (depending on many factors), they just became different.
The weight of the pages (PR), everything has changed…
Like we keep saying, you need to wait at least three months. At least. Maybe more. If you are looking for quick fixes, SWITCH BACK TO YOUR OLD SOFTWARE.
I like this suggestion, I’ll do it.
Another area I’m investigating is the default nginx rate limits as well, as maybe Googlebot is hitting them? Thanks for the help everybody, got plenty of ideas already.
@sam I have applied your patch but to another forum that I am building out from scratch. I’m not comfortable applying it in production though as at the moment I don’t think that’s the root cause of the dropping traffic.
Google is not penalizing anything. It just needs to adjust. Adjusting takes time. Be patient
OK. You go to a grocery store all the time. Ad old, mom-n-pop style, old-60’s store. Very reputable. You trust it. You tell your friends that their goods there have good quality and come from reputable sources.
Then they moved to a new, shiny location. With a brand-new store, and all new faces. On the door of the old store that’s a notice: “301 - We’ve moved!”
Inside the brand-new store, they sell exactly the same stuff. But you don’t recognize the store any more. Sure, it’s modern and great and everything, but do you TRUST it? Will you recommend it to all your friends?
Or will you observe for a while? Do you smell something fishy is going on? Are these the same people? Or have they sold out to a money-grabbing new MNC?
In other words, will the new store’s Page Rank be the same as the old store?
How would you describe when many of the top performing pages have lost their top-10 status in the Google results page and are now on page 2?
Google is indexing the site at the rate of 8-10k pages a day, up from the 2k/day average before the migration to Discourse. Googlebot is getting pages a lot faster than before, from 800ms to 70ms with Discourse. Everything seems to be going fine except the massive traffic drop, which keeps on dropping.
Is it possible that the default rate-limiting protections are slowing down googlebot?
Seems like you do not have many active members engaging in the threads on your site. You have to give it some time and try to get some of your members to help promote your forum. Encourage your members to start threads. This is just an outside observation, based on the reply counts. Dont worry so much about google, keep your site moving with interesting and engaging content. Hope this helps. Good luck.
Yes, it’s running in production now.
I see. And even though the whole hundreds of thousands of topics are in the sitemap, Google’s only looking at 10K of those per day? (My understanding of how this stuff works is entirely theoretical.)
It would seem like there’d be a way to tell search engines that it’s the same data and that 301s would be that way, or that you could create a sitemap that passed all of the 301s en masse.
Here is a chart from google webmaster crawl stats:
My interpretation is that Google noticed that something was off on that domain, and a few days after the migration it crawled a massive amount of pages (160k, the peak on Sep 11 --migration was Sep 6). After that, it kept on crawling at higher rates than before.
There are no other obvious signs for Google to drop our ranking so dramatically.
Engagement on the forum hasn’t changed since the migration (actually, it has improved slightly since the migration). Of course we have margins to grow there but this discussion is specifically about SEO after a migration from an old platform.
Possibly something on this thread can help you? There are also some good links with even more info.
One of the important reasons is that the weight of each page has changed. It changes not only because the structure of the page has changed, but also for a number of reasons. For example:
1. Internal linking became different.
2. External sites link to other pages.
Against the background of other, general changes, this has led to your results.
If you still want (but I do not advise) to promote the site (at your own peril and risk), then it is necessary.
1. Determine which sites link to you, which pages, which anchor in the links.
2. Look at the addresses of links and anchors.
In no case should you change old links to new addresses and anchors. But you can try to add new links to the internal pages of the site.
Search engine algorithms are complex and interdependent items. We, for example, back in 2006 wrote more than 1,000 terms.
To guess, not knowing exact dependence, does not make sense. But the general recommendations written above can help. Just do not change the anchor and addresses of old links on other sites.
Continue adding weight not only to the central page but also to the internal pages of the site by linking to external sites, always taking into account the subject matter of the sites. And take care of the activity on the site.
The site has changed, if activity has also fallen, then this is another sign that the search can find that people did not like the site after the changes. This is not true. But the algorithm only looks at the data. He needs time.
Try to do, say, on one of your pages that is written above.
Wrote with Google Translator