Disappearing bookmarks and expected behavior of /bookmarks

bug
1

Our community has been reporting that, after we updated to Discourse 3.0 about ~2 months ago, their bookmarks sometimes mysteriously disappear. I’ve narrowed down the complaints to users primarily using /bookmarks (and not /my/activity/bookmarks) — there’s a related usability issue there between the two, but I’ll come later to that.

Right now I’m mostly confused about what is the expected behavior of /bookmarks as they don’t have all the users bookmark (as listed in /my/activity/bookmarks). Why so? For my own account, the difference is almost 1/3 bookmarks missing: shouldn’t those two sections essentially be listing the exact same amount of bookmarks/topics?

I thought perhaps /bookmarks only supported bookmarked posts (and not topics), but that doesn’t seem to be the case: our users consistently bookmark a post/topic, see it fine showing under /bookmarks for a few hours or days, and then it just… disappears? :confused: If they delete the bookmark, bookmark it again, it appears for a while, and later disappears yet again. To be clear, the bookmarks always show fine under /my/activity/bookmarks, so the bookmark itself exists — but for some reason doesn’t show under /bookmarks after a while. There must be a difference between the contents of the two sections but I don’t see it — what am I missing? :thinking:

Now, the reason this is particularly annoying for our users is that they find the /my/activity/bookmarks unusable for their main use case which is to find which of their bookmarked topics have updates (within a particular category, or just on the whole forum).

I can see a lot of nice functionality on the interface /my/activity/bookmarks but it still doesn’t do their primary use: being able to sort by last activity while at the same time seeing which topics have updates (number of unread). Would be great if sorting would work there too and if the number of unread posts in each topic would show, but I’m not sure how functionality to filter the bookmarks by category would work under /my/activity/bookmarks, so perhaps it’s justifiable that there’s /bookmarks too.

Anyhow, perhaps these are two separate issues here (maybe a bug (?) and a UX issue), but they are related: I have been pushing the community to avoid the “disappearing bookmarks” by using the /my/activity/bookmarks instead, but for what they are using it for, it really doesn’t work. Perhaps if we understand the “disappearing” bookmarks we can find a workaround — so: why don’t all bookmarks show under /bookmarks?

5 Likes
4

I’ve reduced some of this difference down to the fact that /bookmarks doesn’t include bookmarked private messages and if a topic has multiple bookmarked posts, it shows the topic only once under /bookmarks.

However, bookmarks are still missing. We have several members that have topics showing under their /my/activity/bookmarks that don’t always show under /bookmarks. Emphasis on the always because initially they always do, until they eventually disappear some hours/days later. :anguished:

Again, this only started happening when we updated to Discourse 3.0, so I suspect a bug.

5

We can’t bookmark posts, only topics :thinking:

6

My understanding is that you can bookmark specific posts; even multiple posts on the same topic:

Screenshot 2023-05-16 at 19.36.42
Screenshot 2023-05-16 at 19.36.422228×840 113 KB

4 Likes
7

Your understanding is really valid and solid. Mine is not.

When the heck did we got that?

8

This, along with bookmarking chat messages (which only works when the chat plugin is enabled) has been around for some time now, at least since FEATURE: Promote polymorphic bookmarks to default and migrate (#16729) · discourse/discourse@fcc2e7e · GitHub .

This is the crux of the issue really. We still have this old /bookmarks route, which is somewhat useful because of the unread/activity sorting but is missing key things from the new route, and also likely that the query it is using has some issues with post/topic bookmarks and doesn’t show chat bookmarks at all.

This makes sense technically since it’s just a topic list. not a post list, it’s one of the main differences between the old route and the new.

I am not sure what we need to do here, I will look into it more and discuss internally. It is not ideal to have both routes, it would be nicer if the new one could do the missing feature from the old one.

1 Like
11

I realize now that perhaps I should have split my a topic in two: one for ux and another in bug as my main issue (disappearing bookmarks) is getting shadowed by the usability one.

I have in the meantime been poking at the bookmarks feature and found another bug, but I’m still lost at why bookmarks are randomly disappearing from /bookmarks. It seems to be more common with some topics than others for some reason, but I haven’t found a common pattern. To me what’s really strange is that after bookmarking something, it seems to display fine under /bookmarks for some arbitrary number of hours/days, until it disappears from there and is only “fixed” by removing the bookmark and bookmarking it again.

To be clear, this is not an issue with setting a bookmark reminder to delete the bookmark: no reminders involved, and the bookmarks are not getting deleted (they still show under /my/activity/bookmarks) — they are only missing from /bookmarks after a while.

2 Likes
12

Moved this into bug, since support was neither of these two, in hope it gets some more eyeballs in here.

Apologies for the indirect bump, but we have quite a bunch of frustrated users, very unhappy with this. Perhaps the /bookmarks not being enabled by default (as in, visible in the UI) somewhat reduces the exposure to this bug, but I can’t imagine this being specific to our forum.

1 Like
14

Just to say we’re not ignoring you. :slight_smile: We’re just looking at how best to handle it. Hopefully we’ll have more news for you soon. :crossed_fingers:

1 Like
16

This issue sounds frustrating but I’ve yet to be able to reproduce it and I’m not coming up with any good guesses yet about what could cause it.

I’m going to ignore the UX issue for now and focus on the bug.

I’m going to start by restating the problem:

Theoretical steps to reproduce, but not with 100% certainty:

  1. Start with no bookmarks
    • Go to /bookmarks and verify that there are no topics in the list
    • Go to /my/activity/bookmarks and verify there are no bookmarks in the list
  2. Add bookmarks to some number of posts and topics. (For example, 15 bookmarks across 10 unique topics)
    • We don’t think reminders are in play here, but maybe they are?
    • We don’t know yet whether it matters if you bookmark a post or a topic
  3. Make a mental note of how many unique topics have been bookmarked
  4. Go to /bookmarks and verify that there are 10 topics in the list
  5. Go to /my/activity/bookmarks and verify there are 15 items in the list from 10 topics
  6. Wait
  7. Go back to /my/activity/bookmarks and verify there are still 15 items in the list from 10 topics
  8. Go to /bookmarks and see that there are less than 10 topics in the list

If anyone could start with a fresh site and reproduce this with greater certainty, that’d very much help unblock identifying the issue here.

In lieu of that, if anyone sees the issue, just with these steps, perhaps we can advise on a bit of debugging steps to help identify the issue:

  1. Go to /my/activity/bookmarks and scroll to the bottom.
  2. Count the number of unique topics (N)
  3. Go to /bookmarks and scroll to the bottom.
  4. See that there are less than N topics in the list
1 Like
18

Thank you for giving a go at this — I know this doesn’t rank high on the exciting-bugs-to-nail list. :sweat_smile:

Yes, those are pretty much the right steps to get there, with minor notes I’ll mention later below.

Having said that, because of the random nature of the problem (happens to a random topic at a time, disappearing after an arbitrary amount of time) it might be hard to reproduce it by just setting the stage with those steps since regular use seems to be required and it’s not clear if or what is the trigger (which could be time): I have not been able to reproduce the bug myself on demand and the users reporting this are all much more active than I am in using /bookmarks.

However, I do have reports of several different users being affected by it which I have verified through account impersonation. I have asked some to not re-bookmark at least some of those topics that disappeared from /bookmarks so, while I can’t reproduce on demand, I still have access to accounts where that is happening and comparing /bookmarks and /my/activity/bookmarks shows the differences.

While I can’t provide admin access to our forum, I’m happy to run any sql queries, or requests to the Discourse API of our instance (even as the users experiencing this) if that helps. Please PM me for our forum details if someone would like to take a look.

I had a brief play with /bookmarks.json and /my/activity/bookmarks.json endpoints but didn’t get far: it seems that the page /bookmarks already loads with the first page of results, and only hits the endpoint for more pages, unlikely /my/activity/bookmarks which seems to fetch all bookmarks data from the endpoint. Hence, couldn’t quite compare the API responses.

I’m not a Ruby developer, but tried to figure out where in the code those 2 endpoints land to try to understand the differences, but as I’m not familiar with the tech stack I got lost in the controller and only found UsersController#bookmarks.

Now for some comments on the steps, possibly not very relevant, just for the sake of being explicit.

That’s probably correct, but I can’t confirm this as all users being affected on our forum are long time users with loads of existing bookmarks and it was upgrading to 3.0 that brought this. But, as the issue repeats when removing a bookmark and adding it again, I assume if starting with a blank slate it would happen as well. One single reporter said it was more common just after the upgrade and it’s been less frequent… but it might be they are just loosing bookmarks without noticing. :sweat_smile:

I have checked with a couple of users who have been patient enough to help with understand this and was told:

  • they both always bookmark topics, not posts
  • they never use the reminders on bookmarks.

I suppose this doesn’t mean it wouldn’t happen with posts and reminders, just that it doesn’t seem to be dependent on that.

Small remark just to mention that /my/activity/bookmarks can also show bookmarks from private messages, so a different N can be from that too (something that got me lost at some point).

Again, happy to run sql queries, API requests or provide more info if it’s of any help. I’m fully aware that it’s tricky to debug this due to its random behavior, but that’s also why it is frustrating users.

1 Like
23

One theory is that perhaps topic_users.bookmarked is getting out of sync somehow. If you have a problematic topic identifed, can you try to query the topic_users table for that user/topic combo you find, and see if bookmarked is true or false?

Also, I’m curious whether the new experimental /filter feature shows the “missing” topics with in:bookmarked, e.g. https://meta.discourse.org/filter?q=in%3Abookmarked. Can you try that too and let us know?

1 Like
24

Yep, that seems to be the case.

I looked into one of the “missing” bookmarks, and topic_users.bookmarked was set to false for it. This topic does not show under /bookmarks, but it is under /my/activity/bookmarks.

And I’m not sure if it’s relevant, but the topic_users.last_posted_at for that record has a date that is almost a month old (around when it disappeared), while the topic in question has posts nearly daily since then.

Anyhow, not sure what touches that bookmarked flag, but I guess it is a suspect.

Anything else I can check?

Would happily try, but we are on stable (3.0.3), so don’t have that yet.

2 Likes
25

This is very useful. Thank you. If you have an idea of what version it was before updating to 3.0.3, that would help us isolate when the bug was introduced too.

I can’t guarantee when we can prioritise this, but I think we have something to go with when we do start.

2 Likes
26

We have always been on the stable branch, so we likely were on 2.8.14 before. We updated on March 11th, likely to 3.0.1 and the first complaint came up on March 18th, so I suppose in theory it could also be until 3.0.2. Hence, quite a lot of 2.9 beta releases in between I’m afraid.

If there’s a log somewhere on the filesystem to infer previous versions, I can double check this, but 2.8.14...3.0.2 should be right.

Not sure if this is of any help as I’m not familiar with the schema, but I ran the query below:

SELECT
  topic_users.user_id,
  topic_users.topic_id
FROM
  bookmarks,
  topic_users
WHERE
  bookmarks.user_id = topic_users.user_id
  AND bookmarkable_type = 'Topic'
  AND bookmarks.bookmarkable_id = topic_users.topic_id
  AND topic_users.bookmarked = false;

and got 3000+ matches.

I guess something equivalent can be run on any production system to find some “missing” bookmarks which hopefully brings it one step closer to reproducing the problem. Sorry if I’m stating the obvious here, just trying to be helpful. :sweat_smile:

2 Likes
28

FWIW, we are now on 3.1.2 and the issue still remains.

Users report that some bookmarks “disappear” as often as 5 times day (after re-bookmarking it each time). As is, the functionality is just too broken to be used, and there’s no good alternative providing the same level of functionality.

Now that there’s a way to reproduce it (by sql query), and a range of versions for when the bug was introduced, may I have some hope that this will be addressed at some point?

29

Oh, now that you’re on 3.1.2, next time you see this, can you check whether those topics are also missing from /filter or whether they show up there?

I think knowing the answer to that question will help us decide what we do here.

3 Likes
31

I have tested with the /filter feature and it seems the problem manifests itself also with a in:bookmarked search in it.

Due to the random nature nature of when they disappear, I tested this by first finding an “offending” bookmark by using the SQL query I posted earlier and then impersonating that user, and successfully found a bookmark that,

  1. the bookmark is present at /my/activity/bookmarks (as to be expected),
  2. but, the bookmark is not listed at /bookmarks at all (but should)
  3. and, doesn’t show at /filter?q=in%3Abookmarked

Querying the bookmarks db table it is, of course, there. Querying the topic_users table however, the bookmarked column is set to false, which I’m guessing may be related with the issue?

To be clear, this account also had bookmarks in which the bookmark was being displayed on all the 3 places above (as they should) — but this is a temporary state and they eventually “disappear”. The SQL query seems to return only the problematic ones. Hence, running it on a busy production system and impersonating the respective user should allow to reproduce the problem.

1 Like
32

Thanks @mentalstring. Let me take this back to the team again with this new information and see what we come up with next.

2 Likes
36

@mentalstring thanks for bearing with us over all this time, sorry it’s taken so long to get to the root of this. I found the cause and a way to reproduce it today based on the data you have given and a report in Data Explorer on Meta. To reproduce:

  • Bookmark a topic and don’t bookmark any posts within
  • Delete or recover any post in the topic

It is caused by this job (added by me years ago) which doesn’t take into account Topic-level bookmarks:

So, a fix is on the way along with a data migration to correct existing records. I will post here again when I have it merged.

3 Likes