We just launched a new discourse install and everything is working fairly smoothly. One issue I’ve noticed is that Google is crawling links with query params (ex.
answers/c/architecture?no_subcategories=false&page=1). This leads to issues with the permalink system since the permalink lookup is for the exact url match. In the above case, the problem arose because a category was renamed, which created a permalink but the permalink can’t handle the query param.
- is there a way to no-index certain links? For example, user profile pages and links with params. Despite this (Excluding user profiles in robots.txt (or allow edit of file)), user profiles are still being indexed.
- what is the best practice for modifying a Discourse robots.txt?
- is there any support for updating PermalinkConstraint.rb (discourse/permalink_constraint.rb at master · discourse/discourse · GitHub) to check