Disappearing bookmarks and expected behavior of /bookmarks

This, along with bookmarking chat messages (which only works when the chat plugin is enabled) has been around for some time now, at least since FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) · discourse/discourse@fcc2e7e · GitHub .

This is the crux of the issue really. We still have this old /bookmarks route, which is somewhat useful because of the unread/activity sorting but is missing key things from the new route, and also likely that the query it is using has some issues with post/topic bookmarks and doesn’t show chat bookmarks at all.

This makes sense technically since it’s just a topic list. not a post list, it’s one of the main differences between the old route and the new.

I am not sure what we need to do here, I will look into it more and discuss internally. It is not ideal to have both routes, it would be nicer if the new one could do the missing feature from the old one.

1 Like

I realize now that perhaps I should have split my a topic in two: one for ux and another in bug as my main issue (disappearing bookmarks) is getting shadowed by the usability one.

I have in the meantime been poking at the bookmarks feature and found another bug, but I’m still lost at why bookmarks are randomly disappearing from /bookmarks. It seems to be more common with some topics than others for some reason, but I haven’t found a common pattern. To me what’s really strange is that after bookmarking something, it seems to display fine under /bookmarks for some arbitrary number of hours/days, until it disappears from there and is only “fixed” by removing the bookmark and bookmarking it again.

To be clear, this is not an issue with setting a bookmark reminder to delete the bookmark: no reminders involved, and the bookmarks are not getting deleted (they still show under /my/activity/bookmarks) — they are only missing from /bookmarks after a while.

2 Likes

Moved this into bug, since support was neither of these two, in hope it gets some more eyeballs in here.

Apologies for the indirect bump, but we have quite a bunch of frustrated users, very unhappy with this. Perhaps the /bookmarks not being enabled by default (as in, visible in the UI) somewhat reduces the exposure to this bug, but I can’t imagine this being specific to our forum.

1 Like

Just to say we’re not ignoring you. :slight_smile: We’re just looking at how best to handle it. Hopefully we’ll have more news for you soon. :crossed_fingers:

1 Like

This issue sounds frustrating but I’ve yet to be able to reproduce it and I’m not coming up with any good guesses yet about what could cause it.

I’m going to ignore the UX issue for now and focus on the bug.

I’m going to start by restating the problem:

Theoretical steps to reproduce, but not with 100% certainty:

  1. Start with no bookmarks
    • Go to /bookmarks and verify that there are no topics in the list
    • Go to /my/activity/bookmarks and verify there are no bookmarks in the list
  2. Add bookmarks to some number of posts and topics. (For example, 15 bookmarks across 10 unique topics)
    • We don’t think reminders are in play here, but maybe they are?
    • We don’t know yet whether it matters if you bookmark a post or a topic
  3. Make a mental note of how many unique topics have been bookmarked
  4. Go to /bookmarks and verify that there are 10 topics in the list
  5. Go to /my/activity/bookmarks and verify there are 15 items in the list from 10 topics
  6. Wait
  7. Go back to /my/activity/bookmarks and verify there are still 15 items in the list from 10 topics
  8. Go to /bookmarks and see that there are less than 10 topics in the list

If anyone could start with a fresh site and reproduce this with greater certainty, that’d very much help unblock identifying the issue here.

In lieu of that, if anyone sees the issue, just with these steps, perhaps we can advise on a bit of debugging steps to help identify the issue:

  1. Go to /my/activity/bookmarks and scroll to the bottom.
  2. Count the number of unique topics (N)
  3. Go to /bookmarks and scroll to the bottom.
  4. See that there are less than N topics in the list
1 Like

Thank you for giving a go at this — I know this doesn’t rank high on the exciting-bugs-to-nail list. :sweat_smile:

Yes, those are pretty much the right steps to get there, with minor notes I’ll mention later below.

Having said that, because of the random nature of the problem (happens to a random topic at a time, disappearing after an arbitrary amount of time) it might be hard to reproduce it by just setting the stage with those steps since regular use seems to be required and it’s not clear if or what is the trigger (which could be time): I have not been able to reproduce the bug myself on demand and the users reporting this are all much more active than I am in using /bookmarks.

However, I do have reports of several different users being affected by it which I have verified through account impersonation. I have asked some to not re-bookmark at least some of those topics that disappeared from /bookmarks so, while I can’t reproduce on demand, I still have access to accounts where that is happening and comparing /bookmarks and /my/activity/bookmarks shows the differences.

While I can’t provide admin access to our forum, I’m happy to run any sql queries, or requests to the Discourse API of our instance (even as the users experiencing this) if that helps. Please PM me for our forum details if someone would like to take a look.

I had a brief play with /bookmarks.json and /my/activity/bookmarks.json endpoints but didn’t get far: it seems that the page /bookmarks already loads with the first page of results, and only hits the endpoint for more pages, unlikely /my/activity/bookmarks which seems to fetch all bookmarks data from the endpoint. Hence, couldn’t quite compare the API responses.

I’m not a Ruby developer, but tried to figure out where in the code those 2 endpoints land to try to understand the differences, but as I’m not familiar with the tech stack I got lost in the controller and only found UsersController#bookmarks.

Now for some comments on the steps, possibly not very relevant, just for the sake of being explicit.

That’s probably correct, but I can’t confirm this as all users being affected on our forum are long time users with loads of existing bookmarks and it was upgrading to 3.0 that brought this. But, as the issue repeats when removing a bookmark and adding it again, I assume if starting with a blank slate it would happen as well. One single reporter said it was more common just after the upgrade and it’s been less frequent… but it might be they are just loosing bookmarks without noticing. :sweat_smile:

I have checked with a couple of users who have been patient enough to help with understand this and was told:

  • they both always bookmark topics, not posts
  • they never use the reminders on bookmarks.

I suppose this doesn’t mean it wouldn’t happen with posts and reminders, just that it doesn’t seem to be dependent on that.

Small remark just to mention that /my/activity/bookmarks can also show bookmarks from private messages, so a different N can be from that too (something that got me lost at some point).


Again, happy to run sql queries, API requests or provide more info if it’s of any help. I’m fully aware that it’s tricky to debug this due to its random behavior, but that’s also why it is frustrating users.

1 Like

One theory is that perhaps topic_users.bookmarked is getting out of sync somehow. If you have a problematic topic identifed, can you try to query the topic_users table for that user/topic combo you find, and see if bookmarked is true or false?

Also, I’m curious whether the new experimental /filter feature shows the “missing” topics with in:bookmarked, e.g. https://meta.discourse.org/filter?q=in%3Abookmarked. Can you try that too and let us know?

1 Like

Yep, that seems to be the case.

I looked into one of the “missing” bookmarks, and topic_users.bookmarked was set to false for it. This topic does not show under /bookmarks, but it is under /my/activity/bookmarks.

And I’m not sure if it’s relevant, but the topic_users.last_posted_at for that record has a date that is almost a month old (around when it disappeared), while the topic in question has posts nearly daily since then.

Anyhow, not sure what touches that bookmarked flag, but I guess it is a suspect.

Anything else I can check?

Would happily try, but we are on stable (3.0.3), so don’t have that yet.

2 Likes

This is very useful. Thank you. If you have an idea of what version it was before updating to 3.0.3, that would help us isolate when the bug was introduced too.

I can’t guarantee when we can prioritise this, but I think we have something to go with when we do start.

2 Likes

We have always been on the stable branch, so we likely were on 2.8.14 before. We updated on March 11th, likely to 3.0.1 and the first complaint came up on March 18th, so I suppose in theory it could also be until 3.0.2. Hence, quite a lot of 2.9 beta releases in between I’m afraid.

If there’s a log somewhere on the filesystem to infer previous versions, I can double check this, but 2.8.14...3.0.2 should be right.


Not sure if this is of any help as I’m not familiar with the schema, but I ran the query below:

SELECT
  topic_users.user_id,
  topic_users.topic_id
FROM
  bookmarks,
  topic_users
WHERE
  bookmarks.user_id = topic_users.user_id
  AND bookmarkable_type = 'Topic'
  AND bookmarks.bookmarkable_id = topic_users.topic_id
  AND topic_users.bookmarked = false;

and got 3000+ matches.

I guess something equivalent can be run on any production system to find some “missing” bookmarks which hopefully brings it one step closer to reproducing the problem. Sorry if I’m stating the obvious here, just trying to be helpful. :sweat_smile:

2 Likes

FWIW, we are now on 3.1.2 and the issue still remains.

Users report that some bookmarks “disappear” as often as 5 times day (after re-bookmarking it each time). As is, the functionality is just too broken to be used, and there’s no good alternative providing the same level of functionality.

Now that there’s a way to reproduce it (by sql query), and a range of versions for when the bug was introduced, may I have some hope that this will be addressed at some point?

Oh, now that you’re on 3.1.2, next time you see this, can you check whether those topics are also missing from /filter or whether they show up there?

I think knowing the answer to that question will help us decide what we do here.

3 Likes

I have tested with the /filter feature and it seems the problem manifests itself also with a in:bookmarked search in it.

Due to the random nature nature of when they disappear, I tested this by first finding an “offending” bookmark by using the SQL query I posted earlier and then impersonating that user, and successfully found a bookmark that,

  1. the bookmark is present at /my/activity/bookmarks (as to be expected),
  2. but, the bookmark is not listed at /bookmarks at all (but should)
  3. and, doesn’t show at /filter?q=in%3Abookmarked

Querying the bookmarks db table it is, of course, there. Querying the topic_users table however, the bookmarked column is set to false, which I’m guessing may be related with the issue?

To be clear, this account also had bookmarks in which the bookmark was being displayed on all the 3 places above (as they should) — but this is a temporary state and they eventually “disappear”. The SQL query seems to return only the problematic ones. Hence, running it on a busy production system and impersonating the respective user should allow to reproduce the problem.

1 Like

Thanks @mentalstring. Let me take this back to the team again with this new information and see what we come up with next.

2 Likes

@mentalstring thanks for bearing with us over all this time, sorry it’s taken so long to get to the root of this. I found the cause and a way to reproduce it today based on the data you have given and a report in Data Explorer on Meta. To reproduce:

  • Bookmark a topic and don’t bookmark any posts within
  • Delete or recover any post in the topic

It is caused by this job (added by me years ago) which doesn’t take into account Topic-level bookmarks:

So, a fix is on the way along with a data migration to correct existing records. I will post here again when I have it merged.

3 Likes

Great news!

This can explain the seemingly random nature of it.

I think this roughly matches our timeline of when we started seeing it which was when we updated in March this year to 3.0, which had been released in January. So, it reached us then, even though the commit was done in ~May 2022.

I’m aware that this is not the policy in place… but, given how broken the feature is at the moment, any chance this may be backported to stable too? Otherwise we’ll be months away from getting the fix. We’ll keep our fingers, toes (and eyes) crossed that this is possible.

Thank you for looking into this!

1 Like

If you’re tracking stable then I think you’ll have to wait. Making exceptions is a can of worms, unfortunately.

On the plus side, we plan to continue making stable releases regularly on a 6 month cadence, so you should expect it to be available at the end of January / early February.

Fix has been merged now:

4 Likes

Thank you for the fix for this. :+1: This will have a big impact in our community which relies heavily on /bookmarks to keep track of their favourite topics and has been going through quite a bit of frustration.

I understand the reasons this won’t be backported into stable… so we’ll only really test this early next year then.

Have to admit it’s been a little puzzling to me how it’s been broken for this long and nobody else seem to have spotted it (or at least reported it). My best guess is that /bookmarks not being a default entry on the top menu (these days?) makes it less used than it could which is a shame because for some scenarios, it’s a super useful browsing pattern to have available. But, given that this also affects the upcoming /filter feature, I hope this also helped other people too.

Thanks again!

4 Likes

This topic was automatically closed after 3 days. New replies are no longer allowed.