Watching and tracking implementation for tags and categories

I just revised the implementation of “tracking” and “watching” of tags and categories:

Watching categories and tags has always been a rather limited feature. When you started watching a category you would only watch topics in that category from that time on. (with an edge case that allowed you to also watch recategorized old topics)

This limitation was … hmm … limiting. Start watching #bug and you would not be notified on a large amount of current #bug discussion on historic topics.

The new implementation now attacks the history problem in a rather clean way.

Topic tracking state

For every topic you visit Discourse will store a record with user information about said topic.

  • When did you first visit it?
  • What post are you up to?
  • Are you tracking/watching/muting?
  • and so on

When an account is first created there are no records in this table, it is created, on demand as you visit topics.

This means that on an old established forum with 200k topics we don’t need to create 200k rows per user with tracking state for every new account.

This is the original reason, watching, used to be a “going forward from now”. I did not want to have to backfill 100s of thousands of rows when you watch a category. It would be a performance nightmare.

The new implementation attacks this issue by defining a concept on “implicitly watching”. When a new post is created we also check category and tag prefs (in addition to topic level prefs) to decide who to notify. This allows us to cleanly notify on old topics without having to carry record in topic tracking state (until the last moment).

When a new topic is entered and we create the tracking state row we also check on a user’s tag and category prefs to see if they should be watching in the initialized object.

New and tracking and watching implementation

The new watching implementation works like this

When a user watches a category/tag:

  • Any old topics tracking state records that are in “tracking” or “regular” in that category become “watched” (with “you are watching this category” reason)
  • We store a record to denote user is watching category
  • Topics with missing state are watched implicitly, a tracking record is created when we notify users.

When a user stops watching a category/tag:

  • Any watched topics in that category that were watched due to a category watch will become tracked
  • We remove the record denoting user is watching category

Note: I removed the "decision making* about what to do with history from the user. Since is is fairly straightforward to “stop tracking” if needed by hitting “Dismiss”. Vast majority of the time the decision to stop watching is correct and there is not reason to involve the user here.

Tracking implementation is simpler:

  • Any old topics tracking state that are in “regular” become “tracked”.
  • Rest of stuff is tracked implicitly without record

We do not “stop tracking” stuff for you if you stop tracking a category, the expectation is that you will not just be fiddling with settings there and reverting is not trivial. What if you read the topic or posted in the topic or manually tracked it or met the read time? Simpler implementation is just to leave it up to users to hit the “dismiss” button if they are stuck with topics they don’t want to track.

Categories as "mailing lists"

This implementation unlocks the ability to use “category” or “tag” watches as mailing list replacements. This feature has been frequently asked for.

Transition period

There is no “migration” from old system to new system. If you have a lot of “tracking state” for old topics and a “category watch” you should

  • Stop watching category
  • Start watching category

This will fix up all the tracking state. I will consider adding a migration for this perhaps in a week or so when the system is fully stabilized.

I hope you prefer this simplified implementation, feedback? Questions?

21 Likes

This is awesome! :100:

I’m sure some edge cases will pop up, but this is way better than the old system.

How does muting a category work now?

2 Likes

Zero changes were made to muting. Works exactly as it used to work, hides from front page.

5 Likes

Thank you @sam this seems to be exactly what I had expected all along.

I’m not entirely sure about existing Watchings… is the

Just for performance, removing many many rows of watching states? Or is it required to get the “hey I’m watching a category” state?

If the migration you mentioned will set the category watching state once the dust settles, that’ll be awesome.

Again, thank you for this such quick, decisive action on the matter of Watching Categories.

I feel like I’m missing something. Why put this state on a record associated with the topic at all now? Why not just always check the preference?

Because you could be not watching a category but see a topic of interest and want to watch that topic specifically (which wouldn’t be covered under preference).

1 Like

Yeah, of course. I’m not suggesting eliminating topic-based tracking states all together. Only in the cases of tracking categories or tags.

Because a user can override a specific topic

Watch #bug but mute a particular bug

7 Likes

I see. I guess there’s have to be a nil or default topic tracking state to know when to fall back on the category or tag preference.

It’s a bit hard to tell without the migration you mention, but so far this seems to be just right.

I look forward to a migration that will set the new style tracking state, so I can finish testing this out.

2 Likes

This sounds terrific, @sam! It sounds like this change is what the mailing list crowd was asking for.

What Discourse version is this change on?

I’m not sure how to transition, though. Would we need a script that removes and re-adds all users’ watched categories?

Edit: do these changes impact the watching/tracking states of Group PMs?

A migration is quite risky here, I am worried about adding it and its just for an edge case.

The edge case we are not handling now is:

  • User visits topic X in category Y
  • User watches category Y
  • Topic X is not watched
  • Admin upgrades Discourse
  • topic X is still not watched

However this works just fine

  • User never visited topic X in category Y
  • User watches category Y
  • Admin upgrades Discourse
  • topic X is watched

@alehandrof this is in latest beta, I would not worry about transition edge cases you can deal with them on a case by case basis imo. This has no impact on group PMs

1 Like

What if:

  • User watches category Y (because I have that as the site default for users)
  • User visits topic X in category Y
  • Topic X is not watched (in other words, they took no action while viewing it)
  • Admin upgrades Discourse

Would topic X still be watched [by way of the category watch]?

Unfortunately the edge case you mention is exactly my situation.

I have specific groups watching specific categories (automated via a plugin). I need to know that everyone in that group is receiving notifications from every topic in that category.

2 Likes

@sam In my primary forum, about 1/2 the users are in the edge case you describe, the other are in the one I describe.

Like Alex, the goal is to know that users are Watching specific categories. Perhaps removing them & reading them to a Group could reset the Watching state, via this Group-watch feature request?

2 Likes

encountered this bug recently:

  1. Start by clicking the Category dropdown menu, and choose any category.
  2. While in the selected category space, click on the watch/track button next to + New Topic
  3. Choose to watch the category
  4. Refresh page, then try to unwatch the category by setting it to other option (like Normal or Watching First Post)
  5. Refresh page, the category is still being watched (Watching is still selected).
  6. Only way to unwatch the category is through the account preference page.

Shouldn’t user be able to unwatch any category the same way they use to watch one?

1 Like

@sam Has this actually been implemented yet? If so, could you reference the commit?

Not following, this has been implemented for ages.

2 Likes

@sam That’s what I thought, which is why I’m a bit confused.
The way I understand it, if I am watching a category, a new topic in this category would result in a record being created for this topic where the initial state would be watching.

Is that correct?
Because the way I am experiencing it now is that you will for example get a notification about the new topic, but if you vist /latest?state=watching, the new topic will not show up. It is not until you go visit the topic that the watching state will be set.

@sam did you receive my question?