Posts failed to import from phpBB3


(Dan Maby) #1

Hi, we’ve completed an import of a phpBB3 forum using the import script provided, into a vanilla Discourse install. The import has completed but only imported the users. No posts or topics have been migrated in?

We’re also seeing 500 errors randomly, some pages receive a 500 error every time like /u/dan/preferences/account and some URLs are returning the “Ops The software powering this discussion forum encountered an unexpected problem”. I installed and activated the Data Explorer Plugin to see what’s going on in the database but each time I try to create a query it returns Internal Server Error modal.


(Gerhard Schlager) #2

Well, it sounds like the import failed completely. Did you get any errors during the import? Did the import script even try to import topics and posts? It should have printed a message like:

creating topics and posts
    89392 / 89392 (100%)   [983 items/min]

(Dan Maby) #3

That’s the strange thing @gerhard, it’s imported the thousands of users but nothing else. The import didn’t seem to end right either, this is what displayed at the end:

669056 / 670485 ( 99.8%)  [1768 items/min]  Error on record: {:email=>"matias394@pinamail.com", :username=>"ZUVPamel", :password=>nil, :name=>"Matias394", :created_at=>Sat, 04 Nov 2017 00:27:41 UTC +00:00, :last_seen_at=>Sat, 04 Nov 2017 00:27:41 UTC +00:00, :registration_ip_address=>nil, :active=>false, :trust_level=>0, :approved=>false, :approved_by_id=>nil, :approved_at=>nil, :moderator=>false, :admin=>false, :date_of_birth=>nil, :import_mode=>true, :last_emailed_at=>2018-06-10 15:54:40 +0000}
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-4.0.1/lib/redis/client.rb:119:in `call': ERR Error running script (call to f_b06356ba4628144e123b652c99605b873107c9be): @user_script:14: @user_script: 14: -MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.    (Redis::CommandError)
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-4.0.1/lib/redis.rb:2448:in `block in _eval'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-4.0.1/lib/redis.rb:45:in `block in synchronize'
from /usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-4.0.1/lib/redis.rb:45:in `synchronize'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-4.0.1/lib/redis.rb:2447:in `_eval'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-4.0.1/lib/redis.rb:2499:in `evalsha'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.5/lib/message_bus/backends/redis.rb:381:in `cached_eval'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.5/lib/message_bus/backends/redis.rb:141:in `publish'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.5/lib/message_bus.rb:253:in `publish'
from /var/www/discourse/lib/site_setting_extension.rb:258:in `notify_changed!'
from /var/www/discourse/lib/site_settings/db_provider.rb:7:in `block in initialize'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:413:in `instance_exec'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:413:in `block in make_lambda'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:270:in `block in simple'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `block in invoke_after'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `each'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:511:in `invoke_after'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:132:in `run_callbacks'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:827:in `_run_commit_callbacks'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/transactions.rb:343:in `committed!'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:86:in `commit_records'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:178:in `block in commit_transaction'
from /usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:168:in `commit_transaction'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:207:in `block in within_new_transaction'
from /usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/transactions.rb:210:in `transaction'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/transactions.rb:313:in `save!'
from /var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/suppressor.rb:46:in `save!'
from /var/www/discourse/lib/site_settings/db_provider.rb:40:in `save'
from /var/www/discourse/lib/site_setting_extension.rb:251:in `add_override!'
from /var/www/discourse/lib/site_setting_extension.rb:358:in `block in setup_methods'
from /var/www/discourse/lib/site_setting_extension.rb:285:in `set'
from /var/www/discourse/script/import_scripts/base.rb:103:in `block in reset_site_settings'
from /var/www/discourse/script/import_scripts/base.rb:101:in `each'
from /var/www/discourse/script/import_scripts/base.rb:101:in `reset_site_settings'
from /var/www/discourse/script/import_scripts/base.rb:66:in `ensure in perform'
from /var/www/discourse/script/import_scripts/base.rb:66:in `perform'
from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:20:in `perform'
from script/import_scripts/phpbb3.rb:31:in `<module:PhpBB3>'
from script/import_scripts/phpbb3.rb:12:in `<module:ImportScripts>'
from script/import_scripts/phpbb3.rb:11:in `<main>'

(Dan Maby) #4

Now if I try and rerun the import it won’t. This is what I see:

The phpBB3 import is starting...

Loading existing groups...

Loading existing users...

Loading existing categories...

Loading existing posts...

Loading existing topics...

/var/www/discourse/lib/site_settings/type_supervisor.rb:172:in `validate_value': undefined method `include?' for nil:NilClass (NoMethodError)

from /var/www/discourse/lib/site_settings/type_supervisor.rb:126:in `to_db_value'

from /var/www/discourse/lib/site_setting_extension.rb:250:in `add_override!'

from /var/www/discourse/lib/site_setting_extension.rb:358:in `block in setup_methods'

from /var/www/discourse/lib/site_setting_extension.rb:285:in `set'

from /var/www/discourse/script/import_scripts/base.rb:90:in `block in change_site_settings'

from /var/www/discourse/script/import_scripts/base.rb:88:in `each'

from /var/www/discourse/script/import_scripts/base.rb:88:in `change_site_settings'

from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:37:in `change_site_settings'

from /var/www/discourse/script/import_scripts/base.rb:45:in `perform'

from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:20:in `perform'

from script/import_scripts/phpbb3.rb:31:in `&lt;module:PhpBB3&gt;'

from script/import_scripts/phpbb3.rb:12:in `&lt;module:ImportScripts&gt;'

from script/import_scripts/phpbb3.rb:11:in `&lt;main&gt;'

(Gerhard Schlager) #5

The import didn’t finish. It didn’t even import all users. It aborted at 669056 / 670485 ( 99.8%).

Did you run out of disk space?


(Dan Maby) #6

The list of problems since running the import seems to be growing :frowning: There’s around 40GB disk space available. The site keeps falling over and requiring a rebuild, possibly because of the number of queued jobs

But now I’m getting an error when trying to run the rebuild. I’ve tried rebooting and powering down the droplet

FAILED

--------------------

Pups::ExecError: socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 &amp;&amp; echo postgres already running stop container ; exit 1 failed with return #&lt;Process::Status: pid 44 exit 1&gt;

Location of failure: /pups/lib/pups/exec_command.rb:108:in `spawn'

exec failed with the params "socat /dev/null UNIX-CONNECT:/shared/postgres_run/.s.PGSQL.5432 || exit 0 &amp;&amp; echo postgres already running stop container ; exit 1"

def4e57bc78bb999e0b92227ab486b92787d0e058aaa813c1fd855f97cb61699

** FAILED TO BOOTSTRAP ** please scroll up and look for earlier error messages, there may be more than one

(Gerhard Schlager) #7

You really shouldn’t worry about those errors when it’s clear that the import didn’t finish at all. I suggest you create a new droplet (preferably with more disk space) and try again. Or maybe even run the import on your local machine. Also, if you have a budget, you might consider posting in #marketplace to get some help with your import.


(Jay Pfaffman) #8

It could also be that the droplet for the import didn’t have enough RAM. I’d recommend at least 4GB for an import.


(Dan Maby) #9

@pfaffman yep it was the droplet maxing out it’s RAM. Upgraded to a 6GB, the import is now running and creating topics/posts. Thank you!