Db.rake fails for my email check plugin on rebuilding app

AI-generated summary

The discussion revolves around an issue with the db.rake command failing due to a plugin that checks for disposable emails. The error occurs because the plugin interferes with the seeding of data.

Terrapop explains that their plugin uses an external API to check for disposable emails and is causing the issue. They found a way to exclude their plugin from the seed data files but were unsure how to implement it.

blake provides guidance on how to filter out specific seed data files using the register_seedfu_filter method. However, he also warns that this approach should be used with caution as it could break other plugins.

Terrapop updates that they found the problematic seed file (001_discobot.rb) and fixed the issue by registering a seedfu filter. However, blake suggests a better approach would be to exclude system users and discobot from the email validation using the .human_users method.

Terrapop eventually implements this approach and adds additional conditions to the plugin to only validate emails on sign-up, avoiding the need for a seedfu filter.

I developed my first plugin that checks on signup for disposable emails via an external API.

Works as intended so far…

https://github.com/disclosetv/plugin-check-email

But now, it seem when rebuilding/upgrading the app db.rake has some kind of problem:

rails aborted!
ActiveRecord::RecordInvalid: Validation failed: Email is not allowed from that email provider. Please use another email address.
(eval):39:in `block (2 levels) in run_file'
/src/lib/tasks/db.rake:222:in `block in <main>'
bin/rails:17:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Probably because this line db.rake wants to seed some data but it can’t as my plugin checks for invalid emails and interferes:

SeedFu.seed(SeedHelper.paths, SeedHelper.filter)

But in the same file I found it is possible for an plugin to be excluded from seed data files being run:

  def self.filter
    # Allows a plugin to exclude any specified seed data files from running
    DiscoursePluginRegistry.seedfu_filter.any? ?
      /^(?!.*(#{DiscoursePluginRegistry.seedfu_filter.to_a.join("|")})).*$/ : nil
  end
end

But how? Are there any examples of plugins doing that? Or do I have to do something else to get past this point? Many thanks for any help in advance.

The error message you get doesn’t list the actual seed data file that is causing the issue?

Once you know which seed data file is causing the issue you can filter it out in your plugin with:

after_initialize do
  register_seedfu_filter "00X_seed_filename"
  ...
end

Awesome.

With regards to the error raised that’s all I have.

== 20200819021210 AddUserSelectableColumnToColorSchemes: migrating ============
-- add_column(:color_schemes, :user_selectable, :bool, {:null=>false, :default=>false})
   -> 0.0062s
== 20200819021210 AddUserSelectableColumnToColorSchemes: migrated (0.0100s) ===

== 20200819203846 AddColorSchemeIdToUserOptions: migrating ====================
-- add_column(:user_options, :color_scheme_id, :integer)
   -> 0.0035s
== 20200819203846 AddColorSchemeIdToUserOptions: migrated (0.0051s) ===========

== 20200820174703 AddPartialTargetIdIndexToReviewables: migrating =============
-- add_index(:reviewables, [:target_id], {:where=>"target_type = 'Post'", :algorithm=>:concurrently, :name=>"index_reviewables_on_target_id_where_post_type_eq_post"})
   -> 0.0079s
== 20200820174703 AddPartialTargetIdIndexToReviewables: migrated (0.0091s) ====

rails aborted!
ActiveRecord::RecordInvalid: Validation failed: Email is not allowed from that email provider. Please use another email address.
(eval):39:in `block (2 levels) in run_file'
/src/lib/tasks/db.rake:222:in `block in <main>'
bin/rails:17:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Any possibility to find out/debug which file is exactly causing the trouble for my little new plugin? I suspect anything setting up users with emails.

Here is more debug info:

Summary

terrapop@terrapop:/var/www/discourse$ d/rails --trace db:migrate RAILS_ENV=development
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke environment
** Invoke set_locale (first_time)
** Execute set_locale
** Execute db:migrate
** Invoke db:_dump (first_time)
** Execute db:_dump
** Invoke db:structure:dump (first_time)
** Invoke db:load_config
** Execute db:structure:dump
rails aborted!
ActiveRecord::RecordInvalid: Validation failed: Email is not allowed from that email provider. Please use another email address.
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/validations.rb:80:in raise_validation_error' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/validations.rb:53:in save!’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:318:in block in save!' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:212:in transaction’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:366:in with_transaction_returning_status' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:318:in save!’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/suppressor.rb:48:in save!' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/persistence.rb:635:in block in update!’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:375:in block in with_transaction_returning_status' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:278:in transaction’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:212:in transaction' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:366:in with_transaction_returning_status’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/persistence.rb:633:in update!' (eval):39:in block (2 levels) in run_file’
/usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:46:in eval' /usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:46:in block (2 levels) in run_file’
/usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:58:in block in open' /usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:57:in open’
/usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:57:in open' /usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:36:in block in run_file’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in block in transaction' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:280:in block in within_new_transaction’
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize' /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt’
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize' /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt’
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:278:in within_new_transaction’
/usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in transaction' /usr/local/lib/ruby/gems/2.6.0/gems/activerecord-6.0.3.2/lib/active_record/transactions.rb:212:in transaction’
/usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:35:in run_file' /usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:26:in block in run’
/usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:25:in each' /usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu/runner.rb:25:in run’
/usr/local/lib/ruby/gems/2.6.0/gems/seed-fu-2.3.9/lib/seed-fu.rb:29:in seed' /src/lib/tasks/db.rake:222:in block in ’
/usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in block in execute' /usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in each’
/usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:281:in execute' /usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:219:in block in invoke_with_call_chain’
/usr/local/lib/ruby/2.6.0/monitor.rb:235:in mon_synchronize' /usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:199:in invoke_with_call_chain’
/usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/task.rb:188:in invoke' /usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:160:in invoke_task’
/usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in block (2 levels) in top_level' /usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in each’
/usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:116:in block in top_level' /usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:125:in run_with_threads’
/usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:110:in top_level' /usr/local/lib/ruby/gems/2.6.0/gems/railties-6.0.3.2/lib/rails/commands/rake/rake_command.rb:23:in block in perform’
/usr/local/lib/ruby/gems/2.6.0/gems/rake-13.0.1/lib/rake/application.rb:186:in standard_exception_handling' /usr/local/lib/ruby/gems/2.6.0/gems/railties-6.0.3.2/lib/rails/commands/rake/rake_command.rb:20:in perform’
/usr/local/lib/ruby/gems/2.6.0/gems/railties-6.0.3.2/lib/rails/command.rb:48:in invoke' /usr/local/lib/ruby/gems/2.6.0/gems/railties-6.0.3.2/lib/rails/commands.rb:18:in ’
/usr/local/lib/ruby/gems/2.6.0/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in require' /usr/local/lib/ruby/gems/2.6.0/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in block in require_with_bootsnap_lfi’
/usr/local/lib/ruby/gems/2.6.0/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in register' /usr/local/lib/ruby/gems/2.6.0/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in require_with_bootsnap_lfi’
/usr/local/lib/ruby/gems/2.6.0/gems/bootsnap-1.4.8/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in require' /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in block in require’
/usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:291:in load_dependency' /usr/local/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.2/lib/active_support/dependencies.rb:324:in require’
bin/rails:17:in `’
Tasks: TOP => db:migrate

Update:

Found and fixed the problem:

def seed_primary_email
  UserEmail.seed do |ue|
    ue.id = DiscourseNarrativeBot::BOT_USER_ID
    ue.email = "discobot_email"
    ue.primary = true
    ue.user_id = DiscourseNarrativeBot::BOT_USER_ID
  end
end

in discourse/plugins/discourse-narrative-bot/db/fixtures/001_discobot.rb raised the problem.

Adding register_seedfu_filter "001_discobot" fixed the problem.

Thank you @blake for pointing me in the right direction. Works now.

Last question: For the future… how to register multiple filters? Like so?

register_seedfu_filter "001_discobot|002_xxxxxx|003_yyyy"

There might be a way to do it on one line, but to register multiple filters you can use multiple lines

register_seedfu_filter "001..."
register_seedfu_filter "002..."

This seedfu_filter api needs to be used with extreme caution though!

I really don’t think you should add it to your plugin because you could break discobot for other users of your plugin if they haven’t already ran that seed file.

Maybe what you could do is exclude users with id’s < 0 so that your plugin doesn’t check the emails for system and discobot. You can use the .human_users method to achieve this.

Interesting, will certainly look into this and change it as you suggested.

@blake

I added now the following lines:

return unless value.present?
return unless defined? record.id
return unless record.password_validation_required?

in additon to:

return unless record.should_validate_email_address?

As disco_bot is already created with an id on db.migrate this works now.

Furthermore, I had to add the password_validation_required validator, otherwise the plugin also fired when logged in and on changing the user avatar.

Now, it is really just bound to the sign_up form, and no seedfu filter required anymore.