Google crawling - should it be this slow?

I migrated to discourse and a new domain about a month ago. My sitemap has over 20k URLs in it but so far less than 900 have been indexed?

As someone with little SEO experience, this seems concerning. We have about 350k pageviews so far so we are receiving good traffic but little to none is being driven from search engines. Obviously I’m concerned that without new people finding the site, it could lead to a community that is only losing members.


Here are the sitemaps. I’ve read online that the “Couldn’t fetch” actually means that it’s ok and just waiting to be processed. You can see my attempts at reuploading the sitemaps a few days ago.

Anyway I’m just trying to get perspective on what is “normal”. Should I be concerned at this point?

2 Likes

How did you handle the transition between domains?

Is anything redirecting traffic from the old FQDN to the new?

2 Likes

If you don’t have proper 301 redirection to same content on new domain, and/or you did’n tell migration from old to new on google search console, that is quite normal. But if your redirections are right and migration is told that is not normal.

In most of cases the reason is the first one and there is just somekind generic redirection or no redirection at all. Then googlebot reacts to it as it would be a brand new site — and it is.

But googlebot doesn’t increase its activity just because of migration. That shoud remember too. It can get some extra boost with new and fresh content.

I’m just wondering if you have that much traffic why you get so low numbers.

1 Like

Thanks for the quick response @Stephen and @Jagster

The migrations was from proboards so there is no redirection unfortunately. It’s effectively a new site from google’s perspective, as mentioned. As for content we have about 160 new topics and 4k posts since migration - not sure if that is considered significant enough.

1 Like

Right, so Google is treating your site as if it’s totally new.

The crawler allocates resources based on the perceived search value of the site. Newer sites have a lower crawler budget and take much longer for those first few passes.

Once you have a bunch of stuff in the index and Google has seen users clicking through from search results that budget will increase.

Patience!

4 Likes

So it’s just a frustrating loop where we have to be clicked to be indexed but also we need to be indexed to be clicked :upside_down_face:

Thank you for the perspective. We appear to be trending the right way at least.

3 Likes

Consider that Google has finite crawler resources too - they’re going to focus on crawling high traffic sites frequently, medium sites regularly and small sites occasionally. There’s no point re checking a site for updates if it isn’t seeing regular change and nobody is visiting it.

1 Like