This is the crux of the issue really. We still have this old /bookmarks route, which is somewhat useful because of the unread/activity sorting but is missing key things from the new route, and also likely that the query it is using has some issues with post/topic bookmarks and doesn’t show chat bookmarks at all.
This makes sense technically since it’s just a topic list. not a post list, it’s one of the main differences between the old route and the new.
I am not sure what we need to do here, I will look into it more and discuss internally. It is not ideal to have both routes, it would be nicer if the new one could do the missing feature from the old one.
I realize now that perhaps I should have split my a topic in two: one for ux and another in bug as my main issue (disappearing bookmarks) is getting shadowed by the usability one.
I have in the meantime been poking at the bookmarks feature and found another bug, but I’m still lost at why bookmarks are randomly disappearing from /bookmarks. It seems to be more common with some topics than others for some reason, but I haven’t found a common pattern. To me what’s really strange is that after bookmarking something, it seems to display fine under /bookmarks for some arbitrary number of hours/days, until it disappears from there and is only “fixed” by removing the bookmark and bookmarking it again.
To be clear, this is not an issue with setting a bookmark reminder to delete the bookmark: no reminders involved, and the bookmarks are not getting deleted (they still show under /my/activity/bookmarks) — they are only missing from /bookmarks after a while.
Moved this into bug, since support was neither of these two, in hope it gets some more eyeballs in here.
Apologies for the indirect bump, but we have quite a bunch of frustrated users, very unhappy with this. Perhaps the /bookmarks not being enabled by default (as in, visible in the UI) somewhat reduces the exposure to this bug, but I can’t imagine this being specific to our forum.
Thank you for giving a go at this — I know this doesn’t rank high on the exciting-bugs-to-nail list.
Yes, those are pretty much the right steps to get there, with minor notes I’ll mention later below.
Having said that, because of the random nature of the problem (happens to a random topic at a time, disappearing after an arbitrary amount of time) it might be hard to reproduce it by just setting the stage with those steps since regular use seems to be required and it’s not clear if or what is the trigger (which could be time): I have not been able to reproduce the bug myself on demand and the users reporting this are all much more active than I am in using /bookmarks.
However, I do have reports of several different users being affected by it which I have verified through account impersonation. I have asked some to not re-bookmark at least some of those topics that disappeared from /bookmarks so, while I can’t reproduce on demand, I still have access to accounts where that is happening and comparing /bookmarks and /my/activity/bookmarks shows the differences.
While I can’t provide admin access to our forum, I’m happy to run any sql queries, or requests to the Discourse API of our instance (even as the users experiencing this) if that helps. Please PM me for our forum details if someone would like to take a look.
I had a brief play with /bookmarks.json and /my/activity/bookmarks.json endpoints but didn’t get far: it seems that the page /bookmarks already loads with the first page of results, and only hits the endpoint for more pages, unlikely /my/activity/bookmarks which seems to fetch all bookmarks data from the endpoint. Hence, couldn’t quite compare the API responses.
I’m not a Ruby developer, but tried to figure out where in the code those 2 endpoints land to try to understand the differences, but as I’m not familiar with the tech stack I got lost in the controller and only found UsersController#bookmarks.
Now for some comments on the steps, possibly not very relevant, just for the sake of being explicit.
That’s probably correct, but I can’t confirm this as all users being affected on our forum are long time users with loads of existing bookmarks and it was upgrading to 3.0 that brought this. But, as the issue repeats when removing a bookmark and adding it again, I assume if starting with a blank slate it would happen as well. One single reporter said it was more common just after the upgrade and it’s been less frequent… but it might be they are just loosing bookmarks without noticing.
I have checked with a couple of users who have been patient enough to help with understand this and was told:
they both always bookmark topics, not posts
they never use the reminders on bookmarks.
I suppose this doesn’t mean it wouldn’t happen with posts and reminders, just that it doesn’t seem to be dependent on that.
Small remark just to mention that /my/activity/bookmarks can also show bookmarks from private messages, so a different N can be from that too (something that got me lost at some point).
Again, happy to run sql queries, API requests or provide more info if it’s of any help. I’m fully aware that it’s tricky to debug this due to its random behavior, but that’s also why it is frustrating users.
One theory is that perhaps topic_users.bookmarked is getting out of sync somehow. If you have a problematic topic identifed, can you try to query the topic_users table for that user/topic combo you find, and see if bookmarked is true or false?
I looked into one of the “missing” bookmarks, and topic_users.bookmarked was set to false for it. This topic does not show under /bookmarks, but it is under /my/activity/bookmarks.
And I’m not sure if it’s relevant, but the topic_users.last_posted_at for that record has a date that is almost a month old (around when it disappeared), while the topic in question has posts nearly daily since then.
Anyhow, not sure what touches that bookmarked flag, but I guess it is a suspect.
Anything else I can check?
Would happily try, but we are on stable (3.0.3), so don’t have that yet.
We have always been on the stable branch, so we likely were on 2.8.14 before. We updated on March 11th, likely to 3.0.1 and the first complaint came up on March 18th, so I suppose in theory it could also be until 3.0.2. Hence, quite a lot of 2.9 beta releases in between I’m afraid.
If there’s a log somewhere on the filesystem to infer previous versions, I can double check this, but 2.8.14...3.0.2 should be right.
Not sure if this is of any help as I’m not familiar with the schema, but I ran the query below:
bookmarks.user_id = topic_users.user_id
AND bookmarkable_type = 'Topic'
AND bookmarks.bookmarkable_id = topic_users.topic_id
AND topic_users.bookmarked = false;
and got 3000+ matches.
I guess something equivalent can be run on any production system to find some “missing” bookmarks which hopefully brings it one step closer to reproducing the problem. Sorry if I’m stating the obvious here, just trying to be helpful.
FWIW, we are now on 3.1.2 and the issue still remains.
Users report that some bookmarks “disappear” as often as 5 times day (after re-bookmarking it each time). As is, the functionality is just too broken to be used, and there’s no good alternative providing the same level of functionality.
Now that there’s a way to reproduce it (by sql query), and a range of versions for when the bug was introduced, may I have some hope that this will be addressed at some point?
I have tested with the /filter feature and it seems the problem manifests itself also with a in:bookmarked search in it.
Due to the random nature nature of when they disappear, I tested this by first finding an “offending” bookmark by using the SQL query I posted earlier and then impersonating that user, and successfully found a bookmark that,
the bookmark is present at /my/activity/bookmarks (as to be expected),
but, the bookmark is not listed at /bookmarks at all (but should)
and, doesn’t show at /filter?q=in%3Abookmarked
Querying the bookmarks db table it is, of course, there. Querying the topic_users table however, the bookmarked column is set to false, which I’m guessing may be related with the issue?
To be clear, this account also had bookmarks in which the bookmark was being displayed on all the 3 places above (as they should) — but this is a temporary state and they eventually “disappear”. The SQL query seems to return only the problematic ones. Hence, running it on a busy production system and impersonating the respective user should allow to reproduce the problem.
@mentalstring thanks for bearing with us over all this time, sorry it’s taken so long to get to the root of this. I found the cause and a way to reproduce it today based on the data you have given and a report in Data Explorer on Meta. To reproduce:
Bookmark a topic and don’t bookmark any posts within
Delete or recover any post in the topic
It is caused by this job (added by me years ago) which doesn’t take into account Topic-level bookmarks:
So, a fix is on the way along with a data migration to correct existing records. I will post here again when I have it merged.
This can explain the seemingly random nature of it.
I think this roughly matches our timeline of when we started seeing it which was when we updated in March this year to 3.0, which had been released in January. So, it reached us then, even though the commit was done in ~May 2022.
I’m aware that this is not the policy in place… but, given how broken the feature is at the moment, any chance this may be backported to stable too? Otherwise we’ll be months away from getting the fix. We’ll keep our fingers, toes (and eyes) crossed that this is possible.
Thank you for the fix for this. This will have a big impact in our community which relies heavily on /bookmarks to keep track of their favourite topics and has been going through quite a bit of frustration.
I understand the reasons this won’t be backported into stable… so we’ll only really test this early next year then.
Have to admit it’s been a little puzzling to me how it’s been broken for this long and nobody else seem to have spotted it (or at least reported it). My best guess is that /bookmarks not being a default entry on the top menu (these days?) makes it less used than it could which is a shame because for some scenarios, it’s a super useful browsing pattern to have available. But, given that this also affects the upcoming /filter feature, I hope this also helped other people too.