I want Google to index the content in the link format domain/u/username, what should I do?

support
1

I want Google to index the content in the link format domain/u/username, what should I do?

I have removed the Disallow: /u part to let the link format domain.com/u be indexed by Google, but what should I do if I want the slug format domain/u/username to be indexed by Google? Is there any other configuration?

I have not logged in and cannot access the slug format: domain/u/username, so I see that it is a problem for Google to index this slug format (attached image).

image
image3039Ă—1421 93 KB

Thank you for supporting me, I am new to source discourse so there are still many things I do not understand. Thanks!

2

Hello :wave:

Is this site setting enabled on your site?
hide user profiles from public

If yes, than disable it and you’ll be able to access user profiles without login too.

2 Likes
3

Are you sure your users are equally happy that you would like to tell whole wide world about their membership and some other stuff?

6 Likes
4

Thanks a lot!

1 Like
5

but I still can’t submit the link in the form of domain/u/username to index with Google, what else do I need to open?

(I have attached the image for you to see the status)

image
image3348Ă—1399 139 KB

6

If I remember right that limit comes a bit deeper than robots.txt. Sure, it is quite fast test allowing /u in robots.txt. But if it comes deeper from the core you may need a plugin.

Dig here. That was a thing perhaps a year ago, came along with SEO-stuff, and there was strong reason why it is not allowed.

1 Like
7

I put it in robots.txt (attached image) but it didn’t work, so I came here to ask you.

image
image1491Ă—1827 105 KB

8

Does “View live test” give you any more information?

1 Like
9

It’s probably because user profiles send this HTTP header:

  • x-robots-tag: noindex

You can see it if you run this command in a terminal, replacing <url> with the URL of a user profile:

curl -sSL -D - -o /dev/null <url>

If you remove the robots.txt rule, Google will see that header and deindex the URLs.

Google’s documentation:

1 Like
10

I know the url is Noindex but I don’t know where to edit it in the source discourse. In the robots.txt file, the Disallow: /u part has been removed. I have attached a photo.

image
image2707Ă—1465 121 KB