(No longer supported) Import posts from Facebook group into Discourse

[jesselperry (Jesse Perry)]

I finally got mine to work by using RAILS_ENV=production sudo -E -u discourse rake import:facebook_group with root user, instead of just the rake import:facebook_group per instructions. Not sure why that is?

Some tips for beginners…
Just did this successfully with a pretty small Facebook group. I’m new to Discourse and ruby - but I can Google So some tips below if any other newbies come along and need some help. I’m using the Docker for my Discourse install.

• The instructions refer to “config” and “lib” folders. Those are only accessible to you through ssh and doing cd to /var/discourse then ./launcher ssh app then the “config” and “lib” folders are in /var/www/discourse
• To get files (namely import_facebook.yml and import_facebook.rake) into those folders, you’ll have to do this by placing the files via SFTP into /var/discourse/shared/standalone. Then those files are available on the ./launcher ssh app side (for lack of a better term) in /var/www/discourse and you can use a command like mv to move them into the appropriate lib and config folders.
• Regarding installing the koala gem before you get started, that’s also by first accessing the ./launcher ssh app area and doing cd to /var/www/discourse. An easy way to add koala to the gemfile is to run gem install gemrat then gemrat koala. Then after that you can run bundle install but it will probably warn you about the Gemfile.lock. I just followed per the error instructions and ran bundle install --no-deployment with success.
• When it was all done, it crashed my Discourse I have 2GB of RAM and the import went great, but once it started processing the new users, I got the white screen on my site and it stopped working. So I waited a little bit, then rebuilt my Discourse using ./launcher - now it’s working great. That had the added benefit of cleaning out the koala gem and Gemfile settings (I was a bit wary of how I had done the bundle install as root).
• Be warned! Even with the fake email addresses set, it will still attempt a lot of emails! So you may want to change your SMTP settings (Mandrill for instance has a “test” API you can use that doesn’t actually email out), so you don’t get a lot of bounceback notifications.

That’s all. I know this probably isn’t the cleanest way to do it - but if there are any other beginners out there that need the gaps filled in, hope this helps.

[Sander78 (Sander Datema)]

Thanks! I’ll add your text to the readme, or you can submit a PR if you like.

I created this plugin way before Docker was cool, so that’s why the manual is a bit behind.

[jesselperry (Jesse Perry)]

Thank you (and others in here) for the rake code. It went great and very reliably.

Do you think the text is helpful as-is? I quickly wrote it late at night after long hours of researching and testing.

[Sander78 (Sander Datema)]

Do you think the text is helpful as-is?

This week I’m bit busy, but I’ll have a look at it after that. Your text is helpful in making the transition from pre-Docker to post-Docker.

The thing with this plugin is that it’s not very Docker friendly. It’s also not part of the ‘official’ importers, those work differently. But it’s the only Facebook importer, so we’ll have to work with it.

[thangngoc89 (Khoa Nguyen)]

I’m importing a large group with 10k users and 3 years activity. Got 500 erro from facebook because the importer send out so many requests. I think a simple cached for get user info would be nice.

P/s: I added a sleep on the loop as a temporary solution

[Sander78 (Sander Datema)]

Thanks! We needed a test like that.

[thangngoc89 (Khoa Nguyen)]

@Sander78 I managed to import it. But it fail on certain post. No ideas. Return Facebook API Error 2 (Http code 500). But as I said. The code need a simple cache

[Sander78 (Sander Datema)]

That simple cache is above my knowledge level I’m afraid… But I’m open to a PR of course.

[michaeld (Michael - Communiteq)]

However, since there Facebook username is in the database, they can simpy login using their Facebook login.

I’m having trouble getting this to work.

When trying to claim an account, a new account is created instead of claiming the existing account.

There is a mismatch between the user ID’s , I guess because of the “app scoping”. Does the token that is used making the import have to be connected in some way to the Facebook login configuration on Discourse in any way? If so, how?

I’m probably overlooking very simple, but what?

[riking (Kane York)]

Do the two emails there match?

[michaeld (Michael - Communiteq)]

No, the top one is empty. AFAIK that is by design:

Facebook won’t send the mail address when importing, so all users get a unique fake address. However, since there Facebook username is in the database, they can simpy login using their Facebook login.

[riking (Kane York)]

Well, the Facebook username isn’t what’s being used to match up the logins, so that’s a problem.

[michaeld (Michael - Communiteq)]

Of course it isn’t - that wouldn’t be very secure. I think the email address isn’t being used either ? I think the “link” is being compared, but the link is specific to the FB app, and the key used during import and the key/secret for FB auth aren’t seeing the same user.

[michaeld (Michael - Communiteq)]

I did some tests, and it’s the facebook_user_id that needs to match. But it’s different for each app (the so called app-scoped-user-id) since Graph API v2.0.

@Sander78 What’s the right way to solve this?

[jetatomic (Andrew)]

Being new to all things Rails/Docker myself, I’m following through @jesselperry very helpful tips.

I’ve dropped the files from github into /var/discourse/shared/standalone, but the files aren’t showing up (or I just don’t know where to look) in /var/www/discourse after ./launcher enter app.

How do I move them to the right place? Is there a step required to get them to show up, and/or which directory should I find the files in after entering the app?

[RESOLVED] found ./shared/config/import_facebook.yml by using find . -name *\facebook.yml\* from / after entering the app. I’d previously been using find from /var/www/discourse and getting no results.

[jesselperry (Jesse Perry)]

Good luck! I’m probably about the same experience level as you with Rails/Docker, so it’s kinda the blind leading the blind here

But my import went great and the Discourse installation is still running fine.

Hope it works out.

[jetatomic (Andrew)]

Error: LoadError: cannot load such file -- stringex
Resolved by: gemrat stringex then bundle install --no-deployment

Thought I was getting somewhere until…

Facebook token accepted
rake aborted!
NoMethodError: undefined method `[]' for nil:NilClass
/var/www/discourse/lib/tasks/import_facebook.rake:419:in `fb_extract_writer'
/var/www/discourse/lib/tasks/import_facebook.rake:146:in `block (2 levels) in fb_fetch_posts'
/var/www/discourse/lib/tasks/import_facebook.rake:145:in `each'
/var/www/discourse/lib/tasks/import_facebook.rake:145:in `block in fb_fetch_posts'
/var/www/discourse/lib/tasks/import_facebook.rake:141:in `each'
/var/www/discourse/lib/tasks/import_facebook.rake:141:in `fb_fetch_posts'
/var/www/discourse/lib/tasks/import_facebook.rake:86:in `block in <top (required)>'
Tasks: TOP => import:facebook_group

Any ideas as to whether this is script <—> facebook incompatibility, or some other config on my install?

[sam (Sam Saffron)]

that stringex dependency needs to be removed from our script, we not longer use it for anything.

[Sander78 (Sander Datema)]

Is that in my script or somewhere in Discourse?

[su_js1 (js1)]

Helping @jetatomic and running into a problem with RateLimiter:

Kind of wondering now if the import is dying on something else and the --trace just spits out the last call which just happens to be to RateLimiter::LimitExceeded.

[4%] Creating topic 'So what's the scoop on land without property taxes 2015-02-12 18:57:11 +0000
rake aborted!
RateLimiter::LimitExceeded: RateLimiter::LimitExceeded
/var/www/discourse/lib/rate_limiter.rb:56:in `performed!'
/var/www/discourse/lib/rate_limiter/on_create_record.rb:42:in `block in rate_limit'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:443:in `instance_exec'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:443:in `block in make_lambda'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:224:in `call'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:224:in `block in halting_and_conditional'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:503:in `call'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:503:in `block in call'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:503:in `each'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:503:in `call'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:86:in `run_callbacks'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/callbacks.rb:306:in `_create_record'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/timestamp.rb:57:in `_create_record'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/persistence.rb:484:in `create_or_update'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/callbacks.rb:302:in `block in create_or_update'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:113:in `call'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:113:in `call'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:552:in `block (2 levels) in compile'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:502:in `call'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:502:in `call'
/usr/local/lib/ruby/gems/2.0.0/gems/activesupport-4.1.10/lib/active_support/callbacks.rb:86:in `run_callbacks'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/callbacks.rb:302:in `create_or_update'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/persistence.rb:103:in `save'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/validations.rb:51:in `save'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/attribute_methods/dirty.rb:21:in `save'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/transactions.rb:268:in `block (2 levels) in save'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/transactions.rb:329:in `block in with_transaction_returning_status'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/connection_adapters/abstract/database_statements.rb:199:in `transaction'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/transactions.rb:208:in `transaction'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/transactions.rb:326:in `with_transaction_returning_status'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/transactions.rb:268:in `block in save'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/transactions.rb:283:in `rollback_active_record_state!'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/transactions.rb:267:in `save'
/var/www/discourse/lib/post_creator.rb:280:in `save_post'
/var/www/discourse/lib/post_creator.rb:115:in `block in create'
/var/www/discourse/lib/distributed_mutex.rb:21:in `synchronize'
/var/www/discourse/lib/distributed_mutex.rb:5:in `synchronize'
/var/www/discourse/lib/post_creator.rb:185:in `block in transaction'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `block in transaction'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in `within_new_transaction'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in `transaction'
/usr/local/lib/ruby/gems/2.0.0/gems/activerecord-4.1.10/lib/active_record/transactions.rb:208:in `transaction'
/var/www/discourse/lib/post_creator.rb:179:in `transaction'
/var/www/discourse/lib/post_creator.rb:113:in `create'
/var/www/discourse/lib/tasks/import_facebook.rake:223:in `block (2 levels) in fb_import_posts_into_dc'
/var/www/discourse/lib/tasks/import_facebook.rake:205:in `each'
/var/www/discourse/lib/tasks/import_facebook.rake:205:in `block in fb_import_posts_into_dc'
/var/www/discourse/lib/tasks/import_facebook.rake:158:in `each'
/var/www/discourse/lib/tasks/import_facebook.rake:158:in `fb_import_posts_into_dc'
/var/www/discourse/lib/tasks/import_facebook.rake:104:in `block in <top (required)>'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:240:in `call'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:240:in `block in execute'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:235:in `each'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:235:in `execute'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:172:in `invoke_with_call_chain'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/task.rb:165:in `invoke'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:150:in `invoke_task'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `each'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:106:in `block in top_level'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:115:in `run_with_threads'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:100:in `top_level'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:78:in `block in run'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:176:in `standard_exception_handling'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/lib/rake/application.rb:75:in `run'
/usr/local/lib/ruby/gems/2.0.0/gems/rake-10.4.2/bin/rake:33:in `<top (required)>'
/usr/local/bin/rake:23:in `load'
/usr/local/bin/rake:23:in `<main>'
Tasks: TOP => import:facebook_group