Plugin causing errors during rebuild

[JammyDodger]

There have been a couple of reports of this:

Bootstrap failed with exit code 1

I am trying to rebuild one (forum.techandtiny.com) of three Discourse instances that I run on my server. The crazy part is the plugins used are identical to the two other instances (forum.tabletpc.review and forum.farfarawaynews.com) that I run on my server. However, in updating all three instances today, only this one fails to spin up after rebuilding. Attached is the terminal output.

Unable to install Discourse Question Answer: ON CONFLICT DO NOTHING

Unable to install Discourse Question Answer == 20211020062413 CreateQuestionAnswerVotes: migrating ======================== -- create_table(:question_answer_votes, {}) -> 0.0478s -- add_index(:question_answer_votes, [:post_id, :user_id], {:unique=>true}) -> 0.0024s -- execute("INSERT INTO question_answer_votes (post_id, user_id, created_at)\nSELECT\n X.post_id AS post_id,\n (X.value->>'user_id')::int AS user_id,\n (X.value->>'created_at')::timestamp AS created_at\nFROM (\n SELECT\n …

I think it may be to do with having the Pavilion Question/Answer plugin installed previously?

I’ll see if I can get someone to take a deeper look.

[Jaap-Jan_Swijnenburg (JayJayS)]

We are using this plugin: https://github.com/angusmcleod/discourse-question-answer.

• It seems to be related to the Pavilion Question and Answer plugin.
• It is a fork from the discourse/discourse-upvotes plugin.

[JammyDodger]

Just to check, which plugins do you have in your app.yml?

[pfaffman (Jay Pfaffman)]

The site that I saw had a bunch of those Post Custom Fields that had a bunch of strings that had been encoded a bunch of times so that they were unusable.

Some solutions, in order of complexity

• stop using the plugin (and perhaps switch to the new up votes)
• delete all of them
• delete the bad ones
• edit those fields to make the data back into valid json strings.

The site that I saw had bad data that was years old. This appears to be a bug that existed years ago that is only now being discovered.

It might be possible to write code to fix the broken json strings, but I wasn’t able to see how to do that in 10 minutes.

See Question Answer Plugin - #301 by pfaffman for examples

[nat (Natalie T)]

@pfaffman

stop using the plugin (and perhaps switch to the new up votes)

I had thought of this too, but a quick check with Angus’ fork, it doesn’t seem to have any data migrations done. https://github.com/angusmcleod/discourse-question-answer/compare/main...discourse:discourse-upvotes:main

This feels like data contamination from old Q&A posts as you said. I also find it funny that this is popping up now given that that migration is from Nov 2021 – I suppose this is because JayJay is only switching to the new plugin now? In any case we’ll have a look at that migration again.

[Jaap-Jan_Swijnenburg (JayJayS)]

          - git clone https://github.com/discourse/docker_manager.git
          - git clone https://github.com/discourse/discourse-solved.git
          - git clone https://github.com/unfoldingWord-dev/discourse-mermaid.git
          - git clone https://github.com/angusmcleod/discourse-question-answer.git
          - git clone https://github.com/discourse/discourse-checklist.git
          - git clone https://github.com/discourse/discourse-cakeday.git
          - git clone https://github.com/discourse/discourse-canned-replies.git
          - git clone https://github.com/discourse/discourse-footnote.git
          - git clone https://github.com/discourse/discourse-staff-notes.git
          - git clone https://github.com/discourse/discourse-graphviz.git
          - git clone https://github.com/discourse/discourse-assign.git
          - git clone https://github.com/discourse/discourse-voting.git
          - git clone https://github.com/discourse/discourse-yearly-review.git
          - git clone https://github.com/discourse/discourse-saved-searches.git

[Jaap-Jan_Swijnenburg (JayJayS)]

@nat I’m not switching to the new plugin at all. I’m still running with the set of plugins that we have had for quite some time.

At least every month, we run our updates, so it’s not like there is a huge gap between versions.

[JammyDodger]

Just as a small ‘FYI’ aside from the question-answer issue, there’s also a replacement to canned-replies now:

Discourse Templates

Summary Discourse Templates allows the creation of content templates for repeated use. The templates are stored as topics in a special category, where they can be edited and discussed as needed. Repository Link https://github.com/discourse/discourse-templates Install Guide How to install plugins in Discourse This plugin is pretty much a new version of Discourse Canned Replies but it is more flexible and stores…

And I believe there’s an #official Mermaid theme component now as well:

Discourse Mermaid

Summary Discourse Mermaid lets you create very powerful graphs using the Mermaid syntax. Preview Preview on theme-creator.discourse.org Repository Link https://github.com/discourse/discourse-mermaid-theme-component New to Discourse Themes? Beginner’s guide to using Discourse Themes Install this theme component [Screenshot 2022-02-15 at 15.12.14] Usage: ```mermaid height=200 graph TD; A --> B; ``` This compone…

[Jaap-Jan_Swijnenburg (JayJayS)]

Thank you. Will look into that after my original problem is fixed

[Jaap-Jan_Swijnenburg (JayJayS)]

If I would remove the plugin, restoring the backup would still cause the offending query to run.
So, I would need to remove both the plugin and any table + query that is used by this plugin.
How would I know the involved tables?

[Stephen]

Have you commented out the line for the plugin and attempted a rebuild to confirm that?

[Jaap-Jan_Swijnenburg (JayJayS)]

I’m going to try that now. I’ve looked through the dump.sql file, that is part of the backup, and there is no mention of any question_answer_* table at all. So this gives me hope…

[Jaap-Jan_Swijnenburg (JayJayS)]

I’ve had success on my scratch system!

• Disabled the plugin
• sudo ./launcher rebuild app
• Restore the backup

Going to work on my live system. I’ll keep you in the loop.

[pfaffman (Jay Pfaffman)]

The corrupt data are in the PostCustmField table. But if you don’t have the plugin, it won’t try to migrate those data into the new table.

@nat

In any case we’ll have a look at that migration again.

The issue isn’t the migration itself, it’s that at some point in the past the data got corrupted. If been broken, but it broke in a way that went unnoticed for years. I think that perhaps each new upvote corrupted it further. A reasonable fix might be to ignore the corrupt data, perhaps marking it as corrupt (perhaps renaming the custom field) so that future migrations could ignore it and someone could go and fix them up by hand if they so desired.

[Jaap-Jan_Swijnenburg (JayJayS)]

Success on live as well. Disabled the plugin, and forth we went.
Still wondering why I did not encounter this issue earlier. As I said, every month as a routine we are updating all our systems, so I should have seen this issue earlier.

How would I be able to connect to the Discourse DB myself, to verify what’s in the post_custom_fields table?

[angus (Angus McLeod)]

@Jaap-Jan_Swijnenburg I’m sorry you’ve experienced an issue.

@pfaffman

The issue isn’t the migration itself

I’ll respectfully disagree with you there

The old QnA plugin used standard Discourse custom field type casting, which is meant to handle casting that field to JSON. But that just doesn’t work in some cases (like this one), which is why you (ideally) need to build in format checks when you’re working with Discourse custom fields (particularly if you’re working with quite old code and data like this). I would suggest that that is what the Upvotes Plugin needs to do here, which indeed is where the immediate error is coming from.

@Jaap-Jan_Swijnenburg

How would I be able to connect to the Discourse DB myself, to verify what’s in the post_custom_fields table?

You can use the data explorer plugin to check what you’ve got there.

Discourse Data Explorer

Summary Summary Discourse Data Explorer allows you to make SQL queries against your live database, allowing for up-to-the-minute stats reporting. Repository Link https://github.com/discourse/discourse-data-explorer Install Guide How to install plugins in Discourse Features After installing the plugin, head to /admin/plugins . [de1] Click on the “Settings” button, and turn on data explorer enabled, then return and reload the pa…

[pfaffman (Jay Pfaffman)]

@angus

I’ll respectfully disagree with you there

Well, I mostly defer to you on this, and I think we really agree. I think it’s true that the migration works if the data aren’t broken. We agree that if the data are broken it should fail more gracefully that it does now.

@Jaap-Jan_Swijnenburg

At least every month, we run our updates, so it’s not like there is a huge gap between versions.

Yes, but the data likely were corrupted years ago (that’s the case on the site that I am familiar with), but you didn’t notice because it didn’t fail catastrophically. I’m pretty sure that it wasn’t managing the upvotes as expected, but no one noticed.

@Jaap-Jan_Swijnenburg

How would I be able to connect to the Discourse DB myself, to verify what’s in the post_custom_fields table?

I’d do it from rails, something like this:

./launcher enter app
rails c

Then stuff like this:

all_votes=PostCustomField.where(name: "vote_history")
likely_broken_votes=PostCustomField.where(name: "vote_history").where("value like '\"%'")

Look at just the id and data:

all_votes.pluck(:id,:value)

Get just one pcf:

pcf=PostCustomField.find(1234)

Fix it

pcf.value='the stuff you really want in it'
pcf.save

[angus (Angus McLeod)]

What is happening here is this

1. Some people have a very old version of the question answer plugin with my personal github username in the repository URL in their app.yml file.

2. I transferred the QnA plugin to paviliondev years ago. Github redirects repository URLs when they’re transferred, so the old urls with my personal username in it kept working.

3. More years later Pavilion transferred the Question Answer Plugin to Discourse. Discourse initially kept the name discourse-question-answer.

4. A few months ago I created my own fork of the plugin hosted in discourse, while it was still called discourse-question-answer.

5. People with very old links to the QnA plugin with my personal github account in them were now cloning my new fork of the significantly updated discourse-question-answer hosted in discourse. In other words a very old link was now pointing to a fork of a new plugin. Despite the intervening years, I should have forseen that, so apologies for that. I’ve removed that fork.

6. Discourse changed the name of discourse-question-answer to discourse-upvotes. This change of name has not had a material impact on your case @Jaap-Jan_Swijnenburg, but it’s why you’re now (unexpectedly) cloning a fork of that repo.

7. A migration in discourse/discourse-upvotes (ex discourse-question-answer) assumes the value column in post_custom_fields is type-safe as JSON.

8. Old data created by the plugin when it was angusmcleod/discourse-question-answer (years ago) was not saved as valid JSON by the HasCustomFields concern in discourse/discourse. I’m guessing, but this data was probably added before JSON type-checking was added 4 years ago (It’s still possible to end up with invalid JSON in custom fields registered as JSON in edge cases).

Therefore

9. When people with the (years-out-of-date) angusmcleod/discourse-question-answer url in their app.yml update their Discourse the migration in the new version of the plugin is cloned, the migration runs and potentially creates this error.

There are a few solutions to this:

1. @Jaap-Jan_Swijnenburg in your case you just need to remove the reference to my old QnA plugin and you’ll be able to rebuild your site. That’s it; nothing more. Looks like that’s what you’ve done

2. The discourse/discourse-upvotes plugin migration could be updated to handle non-JSON values in the value column in post_custom_fields.

I’d note that 2 would also handle the additional case of people who do indeed want to switch from the an old version of the QnA plugin to discourse-upvotes. In that case the migration will run and it will fail if any entries in the value column of post_custom_fields are not valid JSON.

[nat (Natalie T)]

We’ve made a decision internally that we’ll drop support for migrating the data from the older implementation. We don’t want to prevent folks from migrating to the new one, but also cannot handle all the cases of contamination from the custom field.

Mentioned in the PR, we welcome PRs to add a rake task which can safely migrate them, similar to what we do for discourse-templates.