# Plugin causing errors during rebuild
**URL:** https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110
**Category:** Self-hosting
**Created:** [October 6, 2022, 9:35am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110 "2022-10-06T09:35:29Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 9:35am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/1 "2022-10-06T09:35:29Z")
Today I tried to upgrade my Discourse installation, from 2.9.0.beta9 to 2.9.0.beta10. But this went horribly wrong.
1. My first attempt was on the console
```plaintext
cd /var/discourse
sudo git pull
sudo ./launcher rebuild app
```
This finally broke off, with the following message somewhere in the middle of all the logs
```plaintext
PG::InvalidParameterValue: ERROR: cannot extract elements from a scalar
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/rack-mini-profiler-3.0.0/lib/patches/db/pg.rb:110:in `exec'
```
I was able to simply restore the forum by starting the old image (which was still hanging around).
1. My second attempt was through the web console (/admin/upgrade). But this finally choked on the same kind of error: `ERROR: cannot extract elements from a scalar`. When going back to the upgrade panel, it told me all components were updated fine. But after a restart of the container, the server now threw a HTTP 500 error ☹
2. I did a clean install on a separate machine, starting from scratch. I was able to install the beta10 version, but trying to restore from a backup caused exactly the same error!
Anybody out there who could be of any help?
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 10:17am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/2 "2022-10-06T10:17:33Z")
I found the query that it is choking on
```plaintext
INSERT INTO question_answer_votes (post_id, user_id, created_at)
SELECT
X.post_id AS post_id,
(X.value->>'user_id')::int AS user_id,
(X.value->>'created_at')::timestamp AS created_at
FROM (
SELECT
post_id,
jsonb_array_elements(value::jsonb) AS value
FROM post_custom_fields WHERE name = 'vote_history'
) AS X
WHERE (X.value->>'action') != 'destroy'
ORDER BY (X.value->>'created_at')::timestamp DESC
ON CONFLICT DO NOTHING
```
If I understand correctly, this happens because the psql function `jsonb_array_elements` expects an array, but receives a `NULL` value.
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [October 6, 2022, 10:20am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/3 "2022-10-06T10:20:27Z")
There have been a couple of reports of this:
> [@Bootstrap failed with exit code 1](https://meta.discourse.org/t/bootstrap-failed-with-exit-code-1/237270):
>
> I am trying to rebuild one ([forum.techandtiny.com](http://forum.techandtiny.com)) of three Discourse instances that I run on my server. The crazy part is the plugins used are identical to the two other instances (forum.tabletpc.review and [forum.farfarawaynews.com](http://forum.farfarawaynews.com)) that I run on my server. However, in updating all three instances today, only this one fails to spin up after rebuilding. Attached is the terminal output.
> [@Unable to install Discourse Question Answer: ON CONFLICT DO NOTHING](https://meta.discourse.org/t/unable-to-install-discourse-question-answer-on-conflict-do-nothing/238128):
>
> Unable to install Discourse Question Answer == 20211020062413 CreateQuestionAnswerVotes: migrating ======================== -- create\_table(:question\_answer\_votes, {}) -\> 0.0478s -- add\_index(:question\_answer\_votes, [:post\_id, :user\_id], {:unique=\>true}) -\> 0.0024s -- execute("INSERT INTO question\_answer\_votes (post\_id, user\_id, created\_at)\nSELECT\n X.post\_id AS post\_id,\n (X.value-\>\>'user\_id')::int AS user\_id,\n (X.value-\>\>'created\_at')::timestamp AS created\_at\nFROM (\n SELECT\n …
I think it may be to do with having the Pavilion Question/Answer plugin installed previously?
I’ll see if I can get someone to take a deeper look. 👍
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 10:28am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/5 "2022-10-06T10:28:53Z")
We are using this plugin: `https://github.com/angusmcleod/discourse-question-answer`.
- It seems to be related to the Pavilion Question and Answer plugin.
- It is a fork from the discourse/discourse-upvotes plugin.
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [October 6, 2022, 10:42am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/8 "2022-10-06T10:42:03Z")
Just to check, which plugins do you have in your `app.yml`?
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 6, 2022, 10:44am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/9 "2022-10-06T10:44:40Z")
The site that I saw had a bunch of those Post Custom Fields that had a bunch of strings that had been encoded a bunch of times so that they were unusable.
Some solutions, in order of complexity
- stop using the plugin (and perhaps switch to the new up votes)
- delete all of them
- delete the bad ones
- edit those fields to make the data back into valid json strings.
The site that I saw had bad data that was years old. This appears to be a bug that existed years ago that is only now being discovered.
It might be possible to write code to fix the broken json strings, but I wasn’t able to see how to do that in 10 minutes.
See [Question Answer Plugin - #301 by pfaffman](https://meta.discourse.org/t/question-answer-plugin/56032/301) for examples
---
### Author:  [@nat](https://meta.discourse.org/u/nat)
#### Post date: [October 6, 2022, 10:57am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/10 "2022-10-06T10:57:11Z")
> [@pfaffman](#):
>
> stop using the plugin (and perhaps switch to the new up votes)
I had thought of this too, but a quick check with Angus’ fork, it doesn’t seem to have any data migrations done. [https://github.com/angusmcleod/discourse-question-answer/compare/main...discourse:discourse-upvotes:main](https://github.com/angusmcleod/discourse-question-answer/compare/main...discourse:discourse-upvotes:main)
This feels like data contamination from old Q&A posts as you said. I also find it funny that this is popping up now given that that migration is from Nov 2021 – I suppose this is because JayJay is only switching to the new plugin now? In any case we’ll have a look at that migration again.
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 11:06am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/11 "2022-10-06T11:06:30Z")
```plaintext
- git clone https://github.com/discourse/docker_manager.git
- git clone https://github.com/discourse/discourse-solved.git
- git clone https://github.com/unfoldingWord-dev/discourse-mermaid.git
- git clone https://github.com/angusmcleod/discourse-question-answer.git
- git clone https://github.com/discourse/discourse-checklist.git
- git clone https://github.com/discourse/discourse-cakeday.git
- git clone https://github.com/discourse/discourse-canned-replies.git
- git clone https://github.com/discourse/discourse-footnote.git
- git clone https://github.com/discourse/discourse-staff-notes.git
- git clone https://github.com/discourse/discourse-graphviz.git
- git clone https://github.com/discourse/discourse-assign.git
- git clone https://github.com/discourse/discourse-voting.git
- git clone https://github.com/discourse/discourse-yearly-review.git
- git clone https://github.com/discourse/discourse-saved-searches.git
```
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 11:19am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/12 "2022-10-06T11:19:47Z")
@nat I’m not switching to the new plugin at all. I’m still running with the set of plugins that we have had for quite some time.
At least every month, we run our updates, so it’s not like there is a huge gap between versions.
---
### Author:  [@JammyDodger](https://meta.discourse.org/u/JammyDodger)
#### Post date: [October 6, 2022, 11:21am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/13 "2022-10-06T11:21:53Z")
Just as a small ‘FYI’ aside from the question-answer issue, there’s also a replacement to canned-replies now:
> [@Discourse Templates](https://meta.discourse.org/t/discourse-templates/229250):
>
> discourse2Summary Discourse Templates allows the creation of content templates for repeated use. The templates are stored as topics in a special category, where they can be edited and discussed as needed.open_bookInstall Guide This plugin is bundled with Discourse core. There is no need to install the plugin separately.information_source This plugin is pretty much a new version of [Discourse Canned Replies](https://meta.discourse.org/t/discourse-canned-replies/48296) but it is more flexible and stores the templates in a differen…
And I believe there’s an #official Mermaid theme component now as well:
> [@Discourse Mermaid](https://meta.discourse.org/t/discourse-mermaid/218242):
>
> discourse2Summary Discourse Mermaid lets you create very powerful graphs using the Mermaid syntax.eyeglassesPreview [Preview on Discourse Theme Creator](https://discourse.theme-creator.io/theme/Discourse/discourse-mermaid)hammer_and_wrenchRepository Link [https://github.com/discourse/discourse-mermaid-theme-component](https://github.com/discourse/discourse-mermaid-theme-component)open_bookNew to Discourse Themes? [Beginner’s guide to using Discourse Themes](https://meta.discourse.org/t/beginners-guide-to-using-discourse-themes/91966) Install this theme component Usage ```mermaid height=200 graph TD; A --\> B; ``` Learn more about the Merma…
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 11:30am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/15 "2022-10-06T11:30:07Z")
Thank you. Will look into that after my original problem is fixed 😄
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 11:34am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/16 "2022-10-06T11:34:07Z")
If I would remove the plugin, restoring the backup would still cause the offending query to run.
So, I would need to remove both the plugin and any table + query that is used by this plugin.
How would I know the involved tables?
---
### Author:  [@Stephen](https://meta.discourse.org/u/Stephen)
#### Post date: [October 6, 2022, 11:49am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/18 "2022-10-06T11:49:00Z")
Have you commented out the line for the plugin and attempted a rebuild to confirm that?
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 11:50am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/19 "2022-10-06T11:50:24Z")
I’m going to try that now. I’ve looked through the dump.sql file, that is part of the backup, and there is no mention of any question\_answer\_\* table at all. So this gives me hope…
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 12:10pm UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/20 "2022-10-06T12:10:46Z")
I’ve had success on my scratch system!
- Disabled the plugin
- `sudo ./launcher rebuild app`
- Restore the backup
Going to work on my live system. I’ll keep you in the loop.
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 6, 2022, 12:13pm UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/21 "2022-10-06T12:13:42Z")
The corrupt data are in the PostCustmField table. But if you don’t have the plugin, it won’t try to migrate those data into the new table.
> [@nat](#):
>
> In any case we’ll have a look at that migration again.
The issue isn’t the migration itself, it’s that at some point in the past the data got corrupted. If been broken, but it broke in a way that went unnoticed for years. I think that perhaps each new upvote corrupted it further. A reasonable fix might be to ignore the corrupt data, perhaps marking it as corrupt (perhaps renaming the custom field) so that future migrations could ignore it and someone could go and fix them up by hand if they so desired.
---
### Author:  [@Jaap-Jan\_Swijnenburg](https://meta.discourse.org/u/Jaap-Jan_Swijnenburg)
#### Post date: [October 6, 2022, 12:33pm UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/22 "2022-10-06T12:33:37Z")
Success on live as well. Disabled the plugin, and forth we went.
Still wondering why I did not encounter this issue earlier. As I said, every month as a routine we are updating all our systems, so I should have seen this issue earlier.
How would I be able to connect to the Discourse DB myself, to verify what’s in the post\_custom\_fields table?
---
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [October 6, 2022, 1:14pm UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/23 "2022-10-06T13:14:39Z")
@Jaap-Jan_Swijnenburg I’m sorry you’ve experienced an issue.
> [@pfaffman](#):
>
> The issue isn’t the migration itself
I’ll respectfully disagree with you there 🙂
The old QnA plugin used [standard Discourse custom field type casting](https://github.com/paviliondev/discourse-question-answer/blob/master/plugin.rb#L85), which is meant to handle casting that field to JSON. But that just doesn’t work in some cases (like this one), which is why you (ideally) need to build in format checks when you’re working with Discourse custom fields (particularly if you’re working with quite old code and data like this). I would suggest that that is what the Upvotes Plugin needs to do here, which indeed is where the immediate error is coming from.
> [@Jaap-Jan\_Swijnenburg](#):
>
> How would I be able to connect to the Discourse DB myself, to verify what’s in the post\_custom\_fields table?
You can use the [data explorer](https://meta.discourse.org/t/32566?silent=true) plugin to check what you’ve got there.
> [@Discourse Data Explorer](https://meta.discourse.org/t/discourse-data-explorer/32566):
>
> discourse2Summary Discourse [Data Explorer](https://meta.discourse.org/t/32566?silent=true) allows you to make SQL queries against your live database, allowing for up-to-the-minute stats reporting.open_bookInstall Guide This plugin is bundled with Discourse core. There is no need to install the plugin separately.information_source If you’re looking for examples or support for any custom queries, you can find lots of topics in our #Data & reporting category under the #sql-query tag. If there’s not one to suit your pa…
---
### Author:  [@pfaffman](https://meta.discourse.org/u/pfaffman)
#### Post date: [October 6, 2022, 8:35pm UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/24 "2022-10-06T20:35:47Z")
> [@angus](#):
>
> I’ll respectfully disagree with you there
Well, I mostly defer to you on this, and I think we really agree. I think it’s true that the migration works if the data aren’t broken. We agree that if the data are broken it should fail more gracefully that it does now.
> [@Jaap-Jan\_Swijnenburg](#):
>
> At least every month, we run our updates, so it’s not like there is a huge gap between versions.
Yes, but the data likely were corrupted years ago (that’s the case on the site that I am familiar with), but you didn’t notice because it didn’t fail catastrophically. I’m pretty sure that it wasn’t managing the upvotes as expected, but no one noticed.
> [@Jaap-Jan\_Swijnenburg](#):
>
> How would I be able to connect to the Discourse DB myself, to verify what’s in the post\_custom\_fields table?
I’d do it from rails, something like this:
```plaintext
./launcher enter app
rails c
```
Then stuff like this:
```plaintext
all_votes=PostCustomField.where(name: "vote_history")
likely_broken_votes=PostCustomField.where(name: "vote_history").where("value like '\"%'")
```
Look at just the id and data:
```plaintext
all_votes.pluck(:id,:value)
```
Get just one pcf:
```plaintext
pcf=PostCustomField.find(1234)
```
Fix it
```plaintext
pcf.value='the stuff you really want in it'
pcf.save
```
---
### Author:  [@angus](https://meta.discourse.org/u/angus)
#### Post date: [October 7, 2022, 9:38am UTC](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110/28 "2022-10-07T09:38:36Z")
What is happening here is this
1. Some people have a very old version of the question answer plugin with my personal github username in the repository URL in their app.yml file.
2. I transferred the QnA plugin to `paviliondev` years ago. Github redirects repository URLs when they’re transferred, so the old urls with my personal username in it kept working.
3. More years later Pavilion transferred the Question Answer Plugin to Discourse. Discourse initially kept the name `discourse-question-answer`.
4. A few months ago I created my own fork of the plugin hosted in `discourse`, while it was still called `discourse-question-answer`.
5. People with very old links to the QnA plugin with my personal github account in them were now cloning my new fork of the significantly updated `discourse-question-answer` hosted in `discourse`. In other words a very old link was now pointing to a fork of a new plugin. Despite the intervening years, I should have forseen that, so apologies for that. I’ve removed that fork.
6. Discourse changed the name of `discourse-question-answer` to `discourse-upvotes`. This change of name has not had a material impact on your case @Jaap-Jan_Swijnenburg, but it’s why you’re now (unexpectedly) cloning a fork of that repo.
7. A migration in `discourse/discourse-upvotes` (ex `discourse-question-answer`) assumes the `value` column in `post_custom_fields` is type-safe as JSON.
8. Old data created by the plugin when it was `angusmcleod/discourse-question-answer` (years ago) was not saved as valid JSON by the `HasCustomFields` concern in `discourse/discourse`. I’m guessing, but this data was probably added [before JSON type-checking was added 4 years ago](https://github.com/discourse/discourse/commit/e364547ff709ad15d888bd4e87c121d4ac5e8505) (It’s still possible to end up with invalid JSON in custom fields registered as JSON in edge cases).
Therefore
1. When people with the (years-out-of-date) `angusmcleod/discourse-question-answer` url in their app.yml update their Discourse the migration in the new version of the plugin is cloned, the migration runs and potentially creates this error.
There are a few solutions to this:
1. @Jaap-Jan_Swijnenburg in your case you just need to remove the reference to my old QnA plugin and you’ll be able to rebuild your site. That’s it; nothing more. Looks like that’s what you’ve done 👍
2. The `discourse/discourse-upvotes` plugin migration could be updated to handle non-JSON values in the `value` column in `post_custom_fields`.
I’d note that `2` would also handle the additional case of people who do indeed want to switch from the an old version of the QnA plugin to `discourse-upvotes`. In that case the migration will run and it will fail if any entries in the `value` column of `post_custom_fields` are not valid JSON.
[Next page](https://meta.discourse.org/t/plugin-causing-errors-during-rebuild/241110.md?page=2)