Hi, I’m getting the following Traceback when running the import script, seems one of the categories already exists (even though I’m importing into a fresh install). How can I tell which category is causing the problem?
creating categories
52 / 60 ( 86.7%) [500 items/min] Traceback (most recent call last):
24: from script/import_scripts/phpbb3.rb:11:in `<main>'
23: from script/import_scripts/phpbb3.rb:12:in `<module:ImportScripts>'
22: from script/import_scripts/phpbb3.rb:31:in `<module:PhpBB3>'
21: from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:20:in `perform'
20: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
19: from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:30:in `execute'
18: from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:101:in `import_categories'
17: from /var/www/discourse/script/import_scripts/base.rb:413:in `create_categories'
16: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.0/lib/patches/db/mysql2.rb:8:in `each'
15: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.0/lib/patches/db/mysql2.rb:8:in `each'
14: from /var/www/discourse/script/import_scripts/base.rb:432:in `block in create_categories'
13: from /var/www/discourse/script/import_scripts/base.rb:461:in `create_category'
12: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/suppressor.rb:48:in `save!'
11: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/transactions.rb:315:in `save!'
10: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/transactions.rb:383:in `with_transaction_returning_status'
9: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/transactions.rb:212:in `transaction'
8: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
7: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/transaction.rb:227:in `within_new_transaction'
6: from /usr/local/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
5: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/transaction.rb:230:in `block in within_new_transaction'
4: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
3: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/transactions.rb:386:in `block in with_transaction_returning_status'
2: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/transactions.rb:315:in `block in save!'
1: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/validations.rb:52:in `save!'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/activerecord-5.2.0/lib/active_record/validations.rb:80:in `raise_validation_error': Validation failed: Category Name has already been taken (ActiveRecord::RecordInvalid)
Look at the standard categories, staff, is a good guess, and see where the overlap is. I sometimes add code to add the category id to the category name if it exists already.
Or you could add a put statement in the category creator to print the category name before its created.
Edit: Just to update, I managed to deduce which forum/category names were causing the problem by inspecting a copy of the phpBB database in MySQL workbench and doing a COUNT from the phpbb_forums table where there was more than one occurrence in the forum_name column.
SELECT forum_name, COUNT(*) c FROM phpbb.phpbb_forums GROUP BY forum_name HAVING c > 1;
After reorganising/removing these duplicate forum names and taking another phpBB DB export, the importer script worked successfully.
I’ve followed the tutorial, and I get an error during the import script:
root@kxkm-dev-import:/var/www/discourse# import_phpbb3.sh
The phpBB3 import is starting...
Loading existing groups...
Loading existing users...
Loading existing categories...
Loading existing posts...
Loading existing topics...
importing from phpBB 3.1.10
creating users
Traceback (most recent call last):
18: from script/import_scripts/phpbb3.rb:11:in `<main>'
17: from script/import_scripts/phpbb3.rb:12:in `<module:ImportScripts>'
16: from script/import_scripts/phpbb3.rb:31:in `<module:PhpBB3>'
15: from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:20:in `perform'
14: from /var/www/discourse/script/import_scripts/base.rb:47:in `perform'
13: from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:28:in `execute'
12: from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:66:in `import_users'
11: from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:168:in `batches'
10: from /var/www/discourse/script/import_scripts/base.rb:877:in `batches'
9: from /var/www/discourse/script/import_scripts/base.rb:877:in `loop'
8: from /var/www/discourse/script/import_scripts/base.rb:878:in `block in batches'
7: from /var/www/discourse/script/import_scripts/phpbb3/importer.rb:67:in `block in import_users'
6: from /var/www/discourse/script/import_scripts/phpbb3/database/database_3_1.rb:7:in `fetch_users'
5: from /var/www/discourse/script/import_scripts/phpbb3/database/database_base.rb:16:in `query'
4: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rack-mini-profiler-1.0.0/lib/patches/db/mysql2.rb:22:in `query'
3: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb:130:in `query'
2: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb:130:in `handle_interrupt'
1: from /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb:131:in `block in query'
/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/mysql2-0.5.2/lib/mysql2/client.rb:131:in `_query': Unknown column 'f.pf_phpbb_website' in 'field list' (Mysql2::Error)
Is it a problem with my phpbb SQL dump, or with the Discourse database ?
Here is my import.yml
## this is the all-in-one, standalone Discourse Docker container template
##
## After making changes to this file, you MUST rebuild
## /var/discourse/launcher rebuild app
##
## BE *VERY* CAREFUL WHEN EDITING!
## YAML FILES ARE SUPER SUPER SENSITIVE TO MISTAKES IN WHITESPACE OR ALIGNMENT!
## visit http://www.yamllint.com/ to validate this file as needed
templates:
- "templates/postgres.template.yml"
- "templates/redis.template.yml"
- "templates/web.template.yml"
- "templates/web.ratelimited.template.yml"
- "templates/web.socketed.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
# #- "templates/web.ssl.template.yml"
# #- "templates/web.letsencrypt.ssl.template.yml"
- "templates/import/phpbb3.template.yml"
## which TCP/IP ports should this container expose?
## If you want Discourse to share a port with another webserver like Apache or nginx,
## see https://meta.discourse.org/t/17247 for details
#expose:
# - "80:80" # http
# - "443:443" # https
params:
db_default_text_search_config: "pg_catalog.french"
## Set db_shared_buffers to a max of 25% of the total memory.
## will be set automatically by bootstrap based on detected RAM, or you can override
db_shared_buffers: "3072MB"
## can improve sorting performance, but adds memory usage per-connection
#db_work_mem: "40MB"
## Which Git revision should this container use? (default: tests-passed)
#version: tests-passed
env:
LANG: fr_FR.UTF-8
# DISCOURSE_DEFAULT_LOCALE: en
## How many concurrent web requests are supported? Depends on memory and CPU cores.
## will be set automatically by bootstrap based on detected CPUs, or you can override
UNICORN_WORKERS: 4
## TODO: The domain name this Discourse instance will respond to
## Required. Discourse will not work with a bare IP number.
DISCOURSE_HOSTNAME: cds.kxkm.net
DISCOURSE_RELATIVE_URL_ROOT: /forum
## Uncomment if you want the container to be started with the same
## hostname (-h option) as specified above (default "$hostname-$config")
#DOCKER_USE_HOSTNAME: true
## TODO: List of comma delimited emails that will be made admin and developer
## on initial signup example 'user1@example.com,user2@example.com'
DISCOURSE_DEVELOPER_EMAILS: '*****@laposte.net'
## TODO: The SMTP mail server used to validate new accounts and send notifications
# SMTP ADDRESS, username, and password are required
# WARNING the char '#' in SMTP password can cause problems!
DISCOURSE_SMTP_ADDRESS: ******
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: ******
DISCOURSE_SMTP_PASSWORD: "*****"
#DISCOURSE_SMTP_ENABLE_START_TLS: true # (optional, default true)
## If you added the Lets Encrypt template, uncomment below to get a free SSL certificate
#LETSENCRYPT_ACCOUNT_EMAIL: me@example.com
## The CDN address for this Discourse instance (configured to pull)
## see https://meta.discourse.org/t/14857 for details
#DISCOURSE_CDN_URL: //discourse-cdn.example.com
## The Docker container is stateless; all data is stored in /shared
volumes:
- volume:
host: /var/discourse/shared/standalone
guest: /shared
- volume:
host: /var/discourse/shared/standalone/log/var-log
guest: /var/log
## Plugins go here
## see https://meta.discourse.org/t/19157 for details
hooks:
after_code:
- exec:
cd: $home/plugins
cmd:
- git clone https://github.com/discourse/docker_manager.git
## Any custom commands to run after building
run:
- exec: echo "Beginning of custom commands"
## If you want to set the 'From' email address for your first registration, uncomment and change:
## After getting the first signup email, re-comment the line. It only needs to run once.
#- exec: rails r "SiteSetting.notification_email='info@unconfigured.discourse.org'"
- exec:
cd: $home
cmd:
- mkdir -p public/forum
- cd public/forum && ln -s ../uploads && ln -s ../backups
- replace:
global: true
filename: /etc/nginx/conf.d/discourse.conf
from: proxy_pass http://discourse;
to: |
rewrite ^/(.*)$ /forum/$1 break;
proxy_pass http://discourse;
- replace:
filename: /etc/nginx/conf.d/discourse.conf
from: etag off;
to: |
etag off;
location /forum {
rewrite ^/forum/?(.*)$ /$1;
}
- replace:
filename: /etc/nginx/conf.d/discourse.conf
from: $proxy_add_x_forwarded_for
to: $http_your_original_ip_header
global: true
- exec: echo "End of custom commands"
and my settings.yml
# This is an example settings file for the phpBB3 importer.
database:
type: MySQL # currently only MySQL is supported
host: localhost
port: 3306
username: root
password:
schema: phpbb
table_prefix: phpbb_ # Change this, if your forum is using a different prefix. Usually all table names start with phpbb_
batch_size: 1000 # Don't change this unless you know what you're doing. The default (1000) should work just fine.
import:
# WARNING: Do not activate this option unless you know what you are doing.
# It will probably break the BBCode to Markdown conversion and slows down your import.
use_bbcode_to_md: false
# This is the path to the root directory of your current phpBB installation (or a copy of it).
# The importer expects to find the /files and /images directories within the base directory.
# You need to change this to something like /var/www/phpbb if you are not using the Docker based importer.
# This is only needed if you want to import avatars, attachments or custom smilies.
phpbb_base_dir: /shared/import/data
site_prefix:
# this is needed for rewriting internal links in posts
original: http://trucfou.pe.hu/forum # without http(s)://
new: https://cds.kxkm.net/forum # with http:// or https://
# Enable this, if you want to redirect old forum links to the the new locations.
permalinks:
categories: true # redirects /viewforum.php?f=1 to /c/category-name
topics: true # redirects /viewtopic.php?f=6&t=43 to /t/topic-name/81
posts: true # redirects /viewtopic.php?p=2455#p2455 to /t/topic-name/81/4
avatars:
uploaded: true # import uploaded avatars
gallery: true # import the predefined avatars phpBB offers
remote: false # WARNING: This can considerably slow down your import. It will try to download remote avatars.
# When true: Anonymous users are imported as suspended users. They can't login and have no email address.
# When false: The system user will be used for all anonymous users.
anonymous_users: false
# Enable this, if you want import password hashes in order to use the "migratepassword" plugin.
# This will allow users to login with their current password.
# The plugin is available at: https://github.com/communiteq/discourse-migratepassword
passwords: false
# By default all the following things get imported. You can disable them by setting them to false.
bookmarks: true
attachments: true
private_messages: true
polls: true
# When true: each imported user will have the original username from phpBB as its name
# When false: the name of each imported user will be blank unless the username was changed during import
username_as_name: false
# Map Emojis to smilies used in phpBB. Most of the default smilies already have a mapping, but you can override
# the mappings here, if you don't like some of them.
# The mapping syntax is: emoji_name: 'smiley_in_phpbb'
# Or map multiple smilies to one Emoji: emoji_name: ['smiley1', 'smiley2']
emojis:
# here are two example mappings...
smiley: [':D', ':-D', ':grin:']
heart: ':love:'
For some reason your phpbb_profile_fields_data table doesn’t contain the pf_phpbb_website column. Either add it to the database dump or edit /var/www/discourse/script/import_scripts/phpbb3/database/database_3_1.rb inside the Docker container and replace f.pf_phpbb_website with NULL
hi, is there any progress in updating script to import from phpbb 3.2?
or maybe anyone found other way to convert phpbb 3.2 to discourse, like, idk, convert to other forum and then to discourse?
I wrote a XML to Markdown converter for phpBB 3.2 and @pfaffman successfully tested it on a migration. So, I guess it’s already working quite well. It hasn’t been added to the repository yet, but that should happen this month, because I’m planning on migrating a forum with it.
It that converter publicly available? I would like to test a migration, specially my own forum in which I have custom BBCodes (from my own phpBB extensions).
I assume you will be able to figure this out on your own, since you want to add support for custom BBCodes. I won’t give any support until my work is finished and merged. Expect some heavy refactoring on my part which might break things for you.
thanks. After a few tries I got it working by copying the script files to the container then running the import script.
960,000 post forum and I paused it at 350,000 posts.
My tests so far have shown the import to be successful, everything looks good.
It does take a long time to import posts, I’m running at 600 posts per minute which isn’t ideal. But I shouldn’t complain!
One thing I encountered is that when you include “templates/import/phpbb3.template.yml” and rebuild, the discourse instance can’t be browsed to. I don’t know if this is by design? I’m still able to enter the container and run the import, then I simply remove the reference to the the template and rebuild.
