There have been a couple of reports of this:
I think it may be to do with having the Pavilion Question/Answer plugin installed previously?
I’ll see if I can get someone to take a deeper look.
There have been a couple of reports of this:
I think it may be to do with having the Pavilion Question/Answer plugin installed previously?
I’ll see if I can get someone to take a deeper look.
We are using this plugin: https://github.com/angusmcleod/discourse-question-answer
.
Just to check, which plugins do you have in your app.yml
?
The site that I saw had a bunch of those Post Custom Fields that had a bunch of strings that had been encoded a bunch of times so that they were unusable.
Some solutions, in order of complexity
The site that I saw had bad data that was years old. This appears to be a bug that existed years ago that is only now being discovered.
It might be possible to write code to fix the broken json strings, but I wasn’t able to see how to do that in 10 minutes.
See Question Answer Plugin - #301 by pfaffman for examples
I had thought of this too, but a quick check with Angus’ fork, it doesn’t seem to have any data migrations done. https://github.com/angusmcleod/discourse-question-answer/compare/main...discourse:discourse-upvotes:main
This feels like data contamination from old Q&A posts as you said. I also find it funny that this is popping up now given that that migration is from Nov 2021 – I suppose this is because JayJay is only switching to the new plugin now? In any case we’ll have a look at that migration again.
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/unfoldingWord-dev/discourse-mermaid.git
- git clone https://github.com/angusmcleod/discourse-question-answer.git
- git clone https://github.com/discourse/discourse-checklist.git
- git clone https://github.com/discourse/discourse-cakeday.git
- git clone https://github.com/discourse/discourse-canned-replies.git
- git clone https://github.com/discourse/discourse-footnote.git
- git clone https://github.com/discourse/discourse-staff-notes.git
- git clone https://github.com/discourse/discourse-graphviz.git
- git clone https://github.com/discourse/discourse-assign.git
- git clone https://github.com/discourse/discourse-voting.git
- git clone https://github.com/discourse/discourse-yearly-review.git
- git clone https://github.com/discourse/discourse-saved-searches.git
@nat I’m not switching to the new plugin at all. I’m still running with the set of plugins that we have had for quite some time.
At least every month, we run our updates, so it’s not like there is a huge gap between versions.
Just as a small ‘FYI’ aside from the question-answer issue, there’s also a replacement to canned-replies now:
And I believe there’s an #official Mermaid theme component now as well:
Thank you. Will look into that after my original problem is fixed
If I would remove the plugin, restoring the backup would still cause the offending query to run.
So, I would need to remove both the plugin and any table + query that is used by this plugin.
How would I know the involved tables?
Have you commented out the line for the plugin and attempted a rebuild to confirm that?
I’m going to try that now. I’ve looked through the dump.sql file, that is part of the backup, and there is no mention of any question_answer_* table at all. So this gives me hope…
I’ve had success on my scratch system!
sudo ./launcher rebuild app
Going to work on my live system. I’ll keep you in the loop.
The corrupt data are in the PostCustmField table. But if you don’t have the plugin, it won’t try to migrate those data into the new table.
The issue isn’t the migration itself, it’s that at some point in the past the data got corrupted. If been broken, but it broke in a way that went unnoticed for years. I think that perhaps each new upvote corrupted it further. A reasonable fix might be to ignore the corrupt data, perhaps marking it as corrupt (perhaps renaming the custom field) so that future migrations could ignore it and someone could go and fix them up by hand if they so desired.
Success on live as well. Disabled the plugin, and forth we went.
Still wondering why I did not encounter this issue earlier. As I said, every month as a routine we are updating all our systems, so I should have seen this issue earlier.
How would I be able to connect to the Discourse DB myself, to verify what’s in the post_custom_fields table?
@Jaap-Jan_Swijnenburg I’m sorry you’ve experienced an issue.
I’ll respectfully disagree with you there
The old QnA plugin used standard Discourse custom field type casting, which is meant to handle casting that field to JSON. But that just doesn’t work in some cases (like this one), which is why you (ideally) need to build in format checks when you’re working with Discourse custom fields (particularly if you’re working with quite old code and data like this). I would suggest that that is what the Upvotes Plugin needs to do here, which indeed is where the immediate error is coming from.
You can use the data explorer plugin to check what you’ve got there.
Well, I mostly defer to you on this, and I think we really agree. I think it’s true that the migration works if the data aren’t broken. We agree that if the data are broken it should fail more gracefully that it does now.
Yes, but the data likely were corrupted years ago (that’s the case on the site that I am familiar with), but you didn’t notice because it didn’t fail catastrophically. I’m pretty sure that it wasn’t managing the upvotes as expected, but no one noticed.
I’d do it from rails, something like this:
./launcher enter app
rails c
Then stuff like this:
all_votes=PostCustomField.where(name: "vote_history")
likely_broken_votes=PostCustomField.where(name: "vote_history").where("value like '\"%'")
Look at just the id and data:
all_votes.pluck(:id,:value)
Get just one pcf:
pcf=PostCustomField.find(1234)
Fix it
pcf.value='the stuff you really want in it'
pcf.save
What is happening here is this
Some people have a very old version of the question answer plugin with my personal github username in the repository URL in their app.yml file.
I transferred the QnA plugin to paviliondev
years ago. Github redirects repository URLs when they’re transferred, so the old urls with my personal username in it kept working.
More years later Pavilion transferred the Question Answer Plugin to Discourse. Discourse initially kept the name discourse-question-answer
.
A few months ago I created my own fork of the plugin hosted in discourse
, while it was still called discourse-question-answer
.
People with very old links to the QnA plugin with my personal github account in them were now cloning my new fork of the significantly updated discourse-question-answer
hosted in discourse
. In other words a very old link was now pointing to a fork of a new plugin. Despite the intervening years, I should have forseen that, so apologies for that. I’ve removed that fork.
Discourse changed the name of discourse-question-answer
to discourse-upvotes
. This change of name has not had a material impact on your case @Jaap-Jan_Swijnenburg, but it’s why you’re now (unexpectedly) cloning a fork of that repo.
A migration in discourse/discourse-upvotes
(ex discourse-question-answer
) assumes the value
column in post_custom_fields
is type-safe as JSON.
Old data created by the plugin when it was angusmcleod/discourse-question-answer
(years ago) was not saved as valid JSON by the HasCustomFields
concern in discourse/discourse
. I’m guessing, but this data was probably added before JSON type-checking was added 4 years ago (It’s still possible to end up with invalid JSON in custom fields registered as JSON in edge cases).
Therefore
angusmcleod/discourse-question-answer
url in their app.yml update their Discourse the migration in the new version of the plugin is cloned, the migration runs and potentially creates this error.There are a few solutions to this:
@Jaap-Jan_Swijnenburg in your case you just need to remove the reference to my old QnA plugin and you’ll be able to rebuild your site. That’s it; nothing more. Looks like that’s what you’ve done
The discourse/discourse-upvotes
plugin migration could be updated to handle non-JSON values in the value
column in post_custom_fields
.
I’d note that 2
would also handle the additional case of people who do indeed want to switch from the an old version of the QnA plugin to discourse-upvotes
. In that case the migration will run and it will fail if any entries in the value
column of post_custom_fields
are not valid JSON.
We’ve made a decision internally that we’ll drop support for migrating the data from the older implementation. We don’t want to prevent folks from migrating to the new one, but also cannot handle all the cases of contamination from the custom field.
Mentioned in the PR, we welcome PRs to add a rake task which can safely migrate them, similar to what we do for discourse-templates.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.