Problem for merge two forum

Hello,

I am merging two forums together via the tutorial present here.

I’m getting stuck on the command gunzip < /path/to/data/other-2018-08-02-134227-v2018xxx.tar.gz | psql -d copyme
It returns me plenty of lines with import errors in unintelligible characters.

Here are the steps performed:

  • Discourse installation via official method on VPS.
  • Import via GUI of the backup to merge (forumother)
  • Backup of the latter (to have a backup with the latest version of discourse)
  • Import of the backup of the destination forum (forumdest)
  • Import of the backup of forumother via SFTP in the folder /var/discourse/shared/standalone/backups/default.

Then I go to the above folder and run the command tar xvzf forumother.tar.gz (with the real name of the backup).

Direction cd /var/discourse then ./launcher enter app

sudo -u postgres psql and CREATE DATABASE "copyme" ENCODING = 'utf8'; and I quit with ``q

It’s now that I’m stuck.
The command gunzip < public/backups/default/forumother.tar.gz | psql -d copyme does not work as it is.
The error is psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: role "root" does not exist

I can see that this is a connection problem, but I don’t see how to solve it.

I tried logging in as the postgres user su - postgres and then running the command gunzip ... but that’s when the command output is completely inconsistent.

postgres@testforum-app:~$ gunzip < /var/www/discourse/public/backups/default/forum-francophon
e-home-assistant-2022-04-24-120337-v20220407195246.tar.gz | psql -d copyme3                  
invalid command \{wB                                                                         
invalid command \֬IeY┴WPC<P%£6؜◆BP␌U@Q█5                                                  
␋┼┴▒┌␋␍ ␌└└▒┼␍ \                                                                            
␋┼┴▒┌␋␍ ␌└└▒┼␍ \];*░Һ                                                                       
␋┼┴▒┌␋␍ ␌└└▒┼␍ \FMݤ!≥"≠±↓QʏCMHȽ◆)&E„/M6!◆┐                                           
␋┼┴▒┌␋␍ ␌└└▒┼␍ \↑                                                                          
ERROR:  ␋┼┴▒┌␋␍ ␉≤├␊ ␊─┤␊┼␌␊ ° ␊┼␌␍␋┼± "UTF8": █│␊° █│37 █│49                            
ERROR:  ␋┼┴▒┌␋␍ ␉≤├␊ ␊─┤␊┼␌␊ ° ␊┼␌␍␋┼± "UTF8": █│␉␉                                      
␋┼┴▒┌␋␍ ␌└└▒┼␍ \лT4٩3┼←                                                                    
␋┼┴▒┌␋␍ ␌└└▒┼␍ \^EC◆≥]B2Ɣ?8[N→^G9←@±4?WY°O͖↓HY┬4                                 
ERROR:  ␋┼┴▒┌␋␍ ␉≤├␊ ␊─┤␊┼␌␊ ° ␊┼␌␍␋┼± "UTF8": █│␉2                                      
...                                                           
invalid command \UWXJ&qkW*SQy2漆&%>N≤OI2·^;├                                               
␋┼┴▒┌␋␍ ␌└└▒┼␍ \2d-ɸjrR*֬&^_'Ƶ褚N2HhIu                                               
                                              S3                                             
invalid command \VI                                                                          
invalid command \}Jݐ                                                                         
invalid command \BQNr|΅tU~#_k#D,WHfqtC                                                       
invalid command \syJx%                                                                    
invalid command \Eq                                                                         
invalid command \-KmJ                                                                        
invalid command \                                                                            
invalid command \/ly=}E~ԚYj>nO                                                               
invalid command \SM!{Co~UH^x)*O}9vUǙk}LQ                                                   
invalid command \eH<=gTRPR1}{~/t!wiwhӒCґJmCĥ=;O^p*-                                   
invalid command \gq{cr}N;.                                                                  
invalid command \2p3a-.C:m_Y@c<4/mvΩߵj]SޙU"O}$[                                         
invalid command \)                                                                           
invalid command \                                                                            
invalid command \VOtDIM2+͆Yq4ŠR                                                           
invalid command \WHElnHBDm%?PBFrOڙ}d"                                                        
invalid command \PjsחPmqJ[B@$Wt*!&|Zz))Q6AJY%!/6=JRJBPþA)YTZ┘=!M^␉CO)→└␋9┤E▒5█E≤Pі
T│ȐOȾB)W┌Ad:}1,kNDoO')G^W5:B␊I│7!N;─À@#┘                                           
┼┴▒┌␋␍ ␌└└▒┼␍ \6◆3                                                                        
␋┼┴▒┌␋␍ ␌└└▒┼␍ \2�Ψ)VD4π]                                                                   
ERROR:  ␋┼┴▒┌␋␍ ␉≤├␊ ␊─┤␊┼␌␊ ° ␊┼␌␍␋┼± "UTF8": █│89                                      
├±␊@├␊├°┤└↑▒:·$                                                                    

I tried to search on the forum, but I didn’t find anything on this.

So I count on your help and thank you in advance.

PS: My English is bad, so it’s possible that I’m not using the right words for a good search on the forum or I’m misinterpreting the tutorial quoted at the beginning of this post.

Probably, psql is trying to use your current system user (root) which doesn’t exist in Postgres. You may need to specify the user when running psql: psql -U postgres

And then check this step:

5 Likes

Thanks for the answer, I’ll try tonight and get back to you to tell you.

1 Like

Hello again,

Let me start by thanking you for your help. My answer is a bit long, but it details the steps taken.

Ben after several tries since the last message, I still have the impossibility to execute the command gunzip < blabla with always an authentication error.

discourse@testforum-app:/var/www/discourse$ gunzip < public/backups/default/forum-francophone-home-assistant-2022-04-24-120337-v20220407195246.tar.gz | psql -U postgres -d copyme
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: Peer authentication failed for user "postgres"

I give you the steps performed :

  • I started from a fresh official installation.

  • My two backups are made with the latest version of Discourse.

  • I import and restore my “forumDest” backup.

  • I upload the “ForumOther” backup to /var/discourse/shared/standalone/backups/default.

  • I connect with SSH

  • I go to `cd /var/discourse

  • I enter the container with the command ./launcher enter app.
    I unzip the archive “ForumOther.tar.gz” located in /var/discourse/shared/standalone/backups/default * I reset the password

root@testforum-app:/var/www/discourse/public/backups/default# ls
dump.sql.gz
forum-francophone-home-assistant-2022-04-24-120337-v20220407195246.tar.gz
home-assistant-communaute-francophone-2022-04-24-111032-v20220304162250.tar.gz
uploads

Here is the result of the ls command forum-francophone-home-assistant-2022-04-24-120337-v20220407195246.tar.gz being ForumOther and home-assistant-communaute-francophone-2022-04-24-111032-v20220304162250.tar.gz being ForumDest

I reset the postgres password with the following commands:

sudo -u postgres psql
\password postgres
(enter the new password)
\q

This gives :

root@testforum-app:/var/www/discourse/public/backups/default# sudo -u postgres psql
psql (13.6 (Debian 13.6-1.pgdg110+1))
Type "help" for help.

postgres=# \password postgres
Enter new password for user "postgres": 
Enter it again: 
postgres=# \q
  • Creation of the “copyme” database
root@testforum-app:/var/www/discourse/public/backups/default# su postgres
postgres@testforum-app:/var/www/discourse/public/backups/default$ psql
psql (13.6 (Debian 13.6-1.pgdg110+1))
Type "help" for help.
postgres=# CREATE DATABASE "copyme2" ENCODING = 'utf8';
CREATE DATABASE
postgres=# \q
postgres@testforum-app:/var/www/discourse/public/backups/default$
  • I run the command postgres@testforum-app:/var/www/discourse/public/backups/default$ gunzip < forum-francophone-home-assistant-2022-04-24-120337-v20220407195246.tar.gz | psql -d copyme
    or `gunzip < forum-francophone-home-assistant-2022-04-24-120337-v20220407195246.tar.gz | psql -U postgres -d copyme
    as mentioned.

I can’t see the beginning of the process, but at the end it always puts me

invalid command \P�jvsחP�m���q��J[B���@$��W�t*����!&|Zz)��)��Q�6�AJ�Y��%!/6=�JR��J��BPþ�A�)�YTZ�j��=! M��^bCO)+mi9���u�p�E��a��50��E�y��P���і��TxȐ��OȾ����B�)Wl��Ad�������:���}1���,kNDo�O�'�)G��^W����5��
                                                                                                                                                                                                        :��B�eIx7!N��;q�À@�#j���*
invalid command \�6���`�3�
invalid command \�2��󟯹�Ψ�)��V��D��4{]�
ERROR: invalid byte sequence for encoding "UTF8": 0x89
postgres@testforum-app:/var/www/discourse/public/backups/default$

So I understand that there is a problem with the encoding?

What I tried:
Disassociating the commands:

  • Unzipping the ForumOther backup gunzip < forum-francophone-home-assistant-2022-04-24-120337-v20220407195246.tar.gz which seems to extract something in incomprehensible language
  • Enter on the DB psql -d copyme
postgres@testforum-app:/var/www/discourse/public/backups/default$ psql -d copyme
psql (13.6 (Debian 13.6-1.pgdg110+1))
Type "help" for help.

copyme=# 

But once connected to the “copyme” base I don’t see what to do.

I looked at the following for the IMPORT commands, but same thing, where should they be executed? How to do it? With which user?

Let’s try step by step.

If I succeed, I will put more precisely my procedure by detailing a little more than the original tutorial (which has the merit to explain the main principles and to exist).

Thanks again for your time and help.

Hello to all,

I have made good progress on my problem.

Now I’m stuck just before the import script is executed.

I run bundle install and then IMPORT=1 bundle install and at this point I get this error

An error occurred while installing tiny_tds(2.1.5, and Bundler cannot continue

I still run the full command IMPORT=1 DB_NAME=copyme DB_PASS=password SOURCE_BASE_URL= UPLOADS_PATH=/var/www/discourse/public/uploads/default bundle exec ruby script/bulk_import/discourse_merger.rb

This time, it’s this error

The git source https://github.com/nlalonde/ruby-bbcode-to-md is not yet checked out. 
Please run `bundle install` before trying to start your application

Do you have an idea?
Thanks

Hello,

Another step forward, in the goal of merging two forums.

I don’t have any dependency problem anymore, but I seem to have a right problem when launching the script.

I launch the script with a local base on an official installation of discourse.

The script command that I run and the return :

discourse@testforum-app:/var/www/discourse$ IMPORT=1 DB_NAME=copyme DB_PASS=hacf2022 SOURCE_B
exec ruby script/bulk_import/discourse_merger.rb                                             
Loading application...                                                                       
Traceback (most recent call last):                                                           
        3: from script/bulk_import/discourse_merger.rb:817:in `<main>'                       
        2: from script/bulk_import/discourse_merger.rb:817:in `new'                          
        1: from script/bulk_import/discourse_merger.rb:21:in `initialize'                    
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/activerecord-7.0.2.4/lib/active_record/dynam
e:Class (NoMethodError)
Did you mean?  connection_db_config                                                          
               connection_pool                                                               
               connection_class                                                              
               connection_class?                                                             
               connection_class=                                                             

Thanks

I also join you the launched commands (that could be useful for others).

Upload other.forum.backup in /var/discourse/shared/standalone/backups/default and extract tar xvzf other.forum.backup.tar.gz

cd `/var/discourse
./laucher enter app
su postgres
psql
CREATE DATABASE "copyme" ENCODING = 'utf8';
\password postgres
\q
exit
su discourse
gunzip < public/backups/default/dump.sql.gz | psql -d copyme
exit

Install dependances nano, sqlite3 et mysql2
With root user

apt update && apt upgrade
apt-get install -y nano sqlite3 libsqlite3-dev mariadb-server libmariadb-dev
gem install sqlite3 mysql2

ruby bbcode to md

git clone https://github.com/nlalonde/ruby-bbcode-to-md.git
cd ruby-bbcode-to-md
gem build ruby-bbcode-to-md.gemspec
gem install ruby-bbcode-to-md-*.gem

Install Tiny_tds
You need FreeTDS for install Tiny_tds.

wget http://www.freetds.org/files/stable/freetds-1.3.10.tar.gz
tar -xzf freetds-1.3.10.tar.gz
cd freetds-1.3.10
./configure --prefix=/usr/local --with-tdsver=7.3
make
make install

For tiny_tds

gem install tiny_tds
gem install ruby-bbcode-to-md

Run script

su discourse
bundle config unset deployment
IMPORT=1 bundle install
IMPORT=1 DB_NAME=copyme DB_PASS=passpostgres SOURCE_BASE_URL=http://other.forum.fr UPLOADS_PATH=/var/www/discourse/public/uploads/default bundle exec ruby script/bulk_import/discourse_merger.rb
1 Like

It looks like you got unlucky with the timing of when you are doing this. 11 days ago there is a commit upgrading Discourse to using Rails 7. The import scripts use the ActiveRecord::Base.connection_config method which was removed with Rails 7.0.0rc1.

@flink91 It looks like the import scripts might have been missed when doing the upgrade.

1 Like

Oh thank you for this answer,

It’s a great news, because I didn’t see where it could come from.

I will wait for the update, except if I can update the merge script myself?

Thanks

1 Like

Just opened a PR, it should be merged soon :slight_smile:

3 Likes

Hello,

Thank you very much for the follow-up and the reactivity.

As the migration of two forums is a long way full of pitfalls I am again confronted with a problem.
I have just made the modification to the discourse_merge.rb file and then re-run the command.

I now have a problem with an existing HASH.

I understand that it is a duplicate problem, but of what nature?

I already deleted a duplicate user, after trying to rename it for my trial, but now it’s this token problem (yet I think duplicates at the user level are handled?)

Thanks again for your help.

discourse@testforum-app:/var/www/discourse$ IMPORT=1 DB_NAME=copyme DB_PASS=password SOURCE_BASE_URL=http://other.forum.fr UPLOADS_PATH=/var/www/discourse/public/uploads/default bundle exec ruby script/bulk_import/discourse_merger.rb
Loading application...
Starting...
Preloading I18n...
Fixing highest post numbers...
Loading imported group ids...
Loading imported user ids...
Loading imported category ids...
Loading imported topic ids...
Loading imported post ids...
Loading groups indexes...
Loading users indexes...
Loading categories indexes...
Loading topics indexes...
Loading posts indexes...
Loading post actions indexes...

merging users...
copying email_tokens...
Traceback (most recent call last):
	7: from script/bulk_import/discourse_merger.rb:817:in `<main>'
	6: from script/bulk_import/discourse_merger.rb:71:in `start'
	5: from /var/www/discourse/script/bulk_import/base.rb:100:in `run'
	4: from script/bulk_import/discourse_merger.rb:82:in `execute'
	3: from script/bulk_import/discourse_merger.rb:145:in `copy_user_stuff'
	2: from script/bulk_import/discourse_merger.rb:445:in `copy_model'
	1: from /var/www/discourse/vendor/bundle/ruby/2.7.0/gems/pg-1.3.5/lib/pg/connection.rb:262:in `copy_data'
/var/www/discourse/vendor/bundle/ruby/2.7.0/gems/pg-1.3.5/lib/pg/connection.rb:262:in `get_last_result': ERROR:  duplicate key value violates unique constraint "index_email_tokens_on_token_hash" (PG::UniqueViolation)
DETAIL:  Key (token_hash)=(07165399523fcdce0679e45950f0260b61386a5c886e3e68fa6cf047d84d1caf) already exists.
CONTEXT:  COPY email_tokens, line 1

Hi.

Here I came to the end of this migration despite several problems.

The last problems came from users registered with different primary addresses on the two forums, but also from missing tables on one forum.

I will come back to put the complete procedure here as soon as I have some time.

Thanks again to @Constanza_Abarca for his help.

1 Like

Does this imply that the two instances were on different versions and/or had different plugins installed? I think my expectation if I were doing a merge like this myself, would be that before anything else I should:

  • Take backups from both
  • Ensure that both are running the same version of Discourse
  • Ensure that the target has all the plugins installed that the source does
  • (Optional?) Ensure the source has all the plugins installed that the target does
  • Take backups again
3 Likes

Hello,

The two forums are on the same version but maybe not the same plugins.

These are just small modifications that I made by hand.

For the real merge I will try with the installed plugins to see.