Sudden drop in traffic

[jcasman (Jesse Casman)]

I help run theta360.guide, a developers forum for a 360 degree camera with an open API. We’ve been running on Discourse for over 5 years. We use Google Analytics to track page views.

We saw a sudden drop in page views just in July. Basically from about 90K page views per month to 70K page views. Posts in the forum were at the same level or even slightly higher than previous month. Outgoing automatic emails same level. In other words, no indication that page views should drop, especially such a big, sudden drop.

Is anyone else seeing a drop in page views?

There is some inconclusive information on changes to Page Experience by Google. According to the following article, changes will be visible between June 15, 2021 and August 31, 2021.

Page Experience Update after One Month - What Happened? - SISTRIX

[simon]

@jcasman

We saw a sudden drop in page views just in July. Basically from about 90K page views per month to 70K page views. Posts in the forum were at the same level or even slightly higher than previous month.

Are you noticing a drop in the Consolidated Pageviews report that’s on your Discourse admin dashboard, or are you only seeing the drop on your Google Analytics page?

[jcasman (Jesse Casman)]

@simon

report that’s on your Discourse admin dashboard

There’s a drop in Discourse analytics but not nearly as large. Last month was 71K, this month 67K.

[osioke (Osioke Itseuwa)]

So this means you are seeing the drop in your Google Analytics page yes? If so, then this may be because of how Discourse and Google differ in counting Pageviews:

@sam

Google uses JavaScript ajax calls to count page views. We work hard only to issue them on first load or when you transition routes (like move from topics to latest list)

Our Page View counting is performed on the “raw requests” made (accounting for a header that informs us when to count)

@sam

Our tracking code only counts “1 API request” for a topic view weather you scroll through a 1 million post topic or not.

This could mean a suite of different things, but I do not think it means real traffic as in users using your site traffic, has dropped significantly, but I can’t say for sure and my analysis which is wholly based on assumptions can be wrong.

[jcasman (Jesse Casman)]

@osioke

So this means you are seeing the drop in your Google Analytics page yes?

Yes, correct, it’s a drop in Google Analytics. My guess is that real traffic has not dropped too much.

I’ve been aware of the difference between Google Analytics and Discourse, but only vaguely understand why. Thanks for the links. This is useful information. The posts are both quite old. Are they still accurate?

[codetricity (Craig Oda)]

@jcasman

My guess is that real traffic has not dropped too much.

I have a different opinion. I believe real traffic has dropped due to behavior changes in the audience that normally reads the forum content. I believe the behavior change is a result of the opening up of health restrictions in the US which may have led to less people searching for API documentation on solutions that help with virtual buying experiences and image documentation of the inside of buildings.

You can compare the Google Analytics with another Discourse install for oppkey.host and see if there is any correlation to the Google Experience changes potential impact on Google Analytics. As that is a smaller site, the best way to verify is to keep asking here.

I just upgraded community.theta360.guide to 2.8.0.beta4, the newest version. If there is a change to how Google Analytics records Discourse traffic, I would assume that other people would have seen the same problem.

Has anyone else seen a drop in traffic as measured with Google Analytics?

Is there a way to strip out the crawlers from the Discourse “Consolidated Pageviews” metrics and just show the “Logged in users” and “Anonymous users” for different timeframes?

Does this report from the admin panel only show the user views?

There is another report for consolidated pageviews that includes the crawlers.

[mstm]

@jcasman

Is anyone else seeing a drop in page views?

Mine is a much smaller site but from April to July I noticed a very noticeable decline.
Have you also checked the google search console?

[HAWK]

@jcasman

The posts are both quite old. Are they still accurate?

Yes, they are still accurate.

[jcasman (Jesse Casman)]

@mstm

google search console

I’ve never used the Google Search Console. I’ve been using Google Analytics. I’m going to look into this. Do you use both?

[mstm]

@jcasman

Do you use both?

Yes and it is very useful.

[codetricity (Craig Oda)]

@jcasman

I’ve never used the Google Search Console. I’ve been using Google Analytics. I’m going to look into this. Do you use both?

To use the Google Search Console, you need to verify ownership with the DNS. I just verified ownership on the site you are seeing the traffic drop on. I also added you as full access.

It is currently processing data.

Do you suspect that all Discourse sites have a drop in traffic from Google Search? It would be good to see if anyone else had a drop in traffic in the June timeframe. Keep asking.

[jcasman (Jesse Casman)]

@codetricity

It is currently processing data.

Thanks for setting this up. I just checked. It says “Processing data, please check again in a day or so.” Looking forward to seeing what we get. I’ll report back here.

[codetricity (Craig Oda)]

@jcasman

I’ll report back here.

If the report shows a drop in traffic, a really good piece of information is if other people running a Discourse site also saw a drop in traffic. Hopefully, the open discussion here with real-world data will help us to get other data points on other Discourse sites.

Again, I suspect that the real traffic actually dropped. However, it is possible that the way Google builds the search index contributed to the drop. IMO, the other factors are:

1. change in people’s online behavior as a result of changes to shelter-in-place mandates
2. change in content that people are consuming as the content on the site in question is used to build better virtual buying experiences and the workflow to take the pictures to support virtual buying
3. change in content on the site

This said, if the Google referral traffic dropped, I would like to know.

Isn’t there additional data on Google analytics for the views that come from Google search? Did those analytics show a humongous drop?

[ttytango (Tym)]

Appears to coincide with the new captcha appearing on Google Chrome every few pages of browsing to make sure you’re human.

You might have lost 20k of viewers competing in some kind of automated browsing and web scraping contest. Or just a change in the analytical algorithm to perhaps include a timeout, to take people who clicked on the page by accident and clicked back to return to where they wanted to be before your site had loaded for a certain amount of time for it to be considered true engagement time with your site.

Perhaps they’ve switched to a serverless function that increments their web site view counts, and they decided that they don’t want to wake a separate lambda function to increment a website view count if a website was never really viewed for more than a couple of seconds.

Coincidentally your experience has coincided with a decisive uptick in marketeers/bots filling in forms on one of my sites so that my inbox has gone from roughly one email a month on how I can get more site traffic, to about 10 a week since July. Not sure what that means for my hypotheses though…

[mstm]

The decline continues in my two discourse sites, all previously indexed pages are becoming “Crawled - currently not indexed” and disappearing from Google search.

I can understand on one site but is it possible that it happens with in two completely different forums?

[simon]

@mstm

all previously indexed pages are becoming “Crawled - currently not indexed”

If you have a chance, it would be great if you could test a few of the URLs that are being marked as “Crawled - currently not indexed.” This can be done with the Live URL test. Details about running the test are here: URL Inspection tool - Search Console Help.

[jcasman (Jesse Casman)]

I’m trying to improve Value to Excluded ratio. One possible improvement, I believe, is the sitemap. Can I edit the sitemap on Discourse? Where is it and how do I get access?

[cocococosti (Constanza Abarca)]

By default, Discourse doesn’t include a sitemap, but you can add one with this plugin: Discourse Sitemap

[abrambailey]

Here’s mine:

[codetricity (Craig Oda)]

@ttytango

Appears to coincide with the new captcha appearing on Google Chrome every few pages of browsing to make sure you’re human.

You might have lost 20k of viewers competing in some kind of automated browsing and web scraping contest. Or just a change in the analytical algorithm to perhaps include a timeout, to take people who clicked on the page by accident and clicked back to return to where they wanted to be before your site had loaded for a certain amount of time for it to be considered true engagement time with your site.

Perhaps they’ve switched to a serverless function that increments their web site view counts, and they decided that they don’t want to wake a separate lambda function to increment a website view count if a website was never really viewed for more than a couple of seconds.

Do you have any more information on a possible change to the timeout? I looked at the site content using Google Search Console and the top 15 responses are for terms that aren’t a good match for the content. It’s likely that the person clicking on the search term link on the Google page didn’t find what they wanted in the discussion on the Discourse forum we are analyzing.

It’s also not clear why Google would direct them to the article in the first place as the article doesn’t appear to contain some critical keywords. For example, although it does “camera” for a 360 consumer camera in the article, the keywords used to find the article contain CCTV, which has nothing to do with a handheld consumer 360 camera.

Update: August 22, 2021

I don’t know how to interpret or use Google Search Console properly. Is there any idea what the 679 failing means?

The report above is from this area: