Cannot execute the rake uploads:migrate_from_s3

I’m going to have to stop using S3 for file uploads, and today I tried the rake uploads:migrate_from_s3, but it failed:

$ rake uploads:migrate_from_s3
rake aborted!
NoMethodError: undefined method `Uploads' for #<Class:0x000056276a4f14f0>
Did you mean?  Upload
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.4/lib/active_record/dynamic_matchers.rb:22:in `method_missing'
/var/www/discourse/lib/tasks/uploads.rake:117:in `migrate_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:110:in `block in migrate_all_from_s3'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rails_multisite-1.1.2/lib/rails_multisite/connection_management.rb:132:in `block in each_connection'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rails_multisite-1.1.2/lib/rails_multisite/connection_management.rb:130:in `each'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/rails_multisite-1.1.2/lib/rails_multisite/connection_management.rb:130:in `each_connection'
/var/www/discourse/lib/tasks/uploads.rake:110:in `migrate_all_from_s3'
/var/www/discourse/lib/tasks/uploads.rake:89:in `block in <top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'
Tasks: TOP => uploads:migrate_from_s3
(See full trace by running task with --trace)

I previously entered the container with the ./launcher enter app command, and listed the available tasks, and the migrate_from_s3 is nowhere to be found:

Full output (long)
rake about                                                             # List versions of all Rails frameworks and the environment
rake add_topic_to_quotes                                               # Add the topic to quotes
rake admin:create                                                      # Creates a forum administrator
rake admin:invite[email]                                               # invite an admin to this discourse instance
rake api_key:get                                                       # generate api key if missing, return existing if already there
rake app:template                                                      # Applies the template supplied by LOCATION=(/path/to/template) or URL
rake app:update                                                        # Update configs and some other initially generated files (or use just update:configs or update:bin)
rake assets:clean[keep]                                                # Remove old compiled assets
rake assets:clobber                                                    # Remove compiled assets
rake assets:environment                                                # Load asset compile environment
rake assets:precompile                                                 # Compile all the assets named in config.assets.precompile
rake assets:prestage                                                   # pre-stage assets on cdn
rake autospec                                                          # Run all specs automatically as needed
rake avatars:clean                                                     # Clean up all avatar thumbnails (use this when the thumbnail algorithm changes)
rake avatars:refresh                                                   # Refresh all avatars (download missing gravatars, refresh system)
rake build:stamp                                                       # stamp the current build with the git hash placed in version.rb
rake build_test_topic                                                  # create pushstate/replacestate test topic
rake cache_digests:dependencies                                        # Lookup first-level dependencies for TEMPLATE (like messages/show or comments/_comment.html)
rake cache_digests:nested_dependencies                                 # Lookup nested dependencies for TEMPLATE (like messages/show or comments/_comment.html)
rake db:create[multisite]                                              # Creates the database from DATABASE_URL or config/database.yml for the current RAILS_ENV (use db:create:all to create all dat...
rake db:drop[multisite]                                                # Drops the database from DATABASE_URL or config/database.yml for the current RAILS_ENV (use db:drop:all to drop all databases...
rake db:environment:set[multisite]                                     # Set the environment value for the database
rake db:fixtures:load                                                  # Loads fixtures into the current environment's database
rake db:migrate[multisite]                                             # Migrate the database (options: VERSION=x, VERBOSE=false, SCOPE=blog)
rake db:migrate:status                                                 # Display status of migrations
rake db:rebuild_indexes                                                # Rebuild indexes
rake db:rollback                                                       # Rolls the schema back to the previous version (specify steps w/ STEP=n)
rake db:schema:cache:clear                                             # Clears a db/schema_cache.yml file
rake db:schema:cache:dump                                              # Creates a db/schema_cache.yml file
rake db:schema:dump                                                    # Creates a db/schema.rb file that is portable against any DB supported by Active Record
rake db:schema:load                                                    # Loads a schema.rb file into the database
rake db:seed                                                           # Loads the seed data from db/seeds.rb
rake db:seed_fu                                                        # Loads seed data for the current environment
rake db:setup                                                          # Creates the database, loads the schema, and initializes with the seed data (use db:reset to also drop the database first)
rake db:stats                                                          # Statistics about database
rake db:structure:load                                                 # Recreates the databases from the structure.sql file
rake db:version                                                        # Retrieves the current schema version number
rake dev:cache                                                         # Toggle development mode caching on/off
rake docker:test                                                       # Run all tests (JS and code in a standalone environment)
rake emails:import                                                     # use this task to import a mailbox into Disourse
rake emoji:test                                                        # test the emoji generation script
rake emoji:update                                                      # update emoji images
rake enqueue_digest_emails                                             # This task is called by the Heroku scheduler add-on
rake export:categories[category_ids]                                   # Export all the categories
rake export:category_structure[include_group_users,file_name]          # Export only the structure of all categories
rake highlightjs:update                                                # download latest version of highlight and prepare it
rake i18n:stats                                                        # show the current translation status
rake import:file[file_name]                                            # Import existing exported file
rake initializers                                                      # Print out all defined initializers in the order they are invoked by Rails
rake integration:create_fixtures                                       # Creates the integration fixtures
rake log:clear                                                         # Truncates all/specified *.log files in log/ to zero bytes (specify which logs with LOGS=test,development)
rake middleware                                                        # Prints out your Rack middleware stack
rake multisite:generate:config                                         # generate multisite config file (if missing)
rake multisite:migrate                                                 # migrate all sites in tier
rake multisite:rollback                                                # rollback migrations for all sites in tier
rake notes                                                             # Enumerate all annotations (use notes:optimize, :fixme, :todo for focus)
rake notes:custom                                                      # Enumerate a custom annotation, specify with ANNOTATION=CUSTOM
rake plugin:install[repo]                                              # install plugin
rake plugin:install_all_official                                       # install all official plugins (use GIT_WRITE=1 to pull with write access)
rake plugin:qunit[plugin,timeout]                                      # run plugin qunit tests
rake plugin:spec[plugin]                                               # run plugin specs
rake plugin:update[plugin]                                             # update a plugin
rake plugin:update_all                                                 # update all plugins
rake poll:migrate_old_polls                                            # Migrate old polls to new syntax
rake posts:defer_all_flags                                             # Defer all flags
rake posts:delete_all_likes                                            # Delete all likes
rake posts:delete_word[find,type]                                      # Delete occurrence of a word/string
rake posts:fix_letter_avatars                                          # Rebake all posts with a quote using a letter_avatar
rake posts:normalize_code                                              # normalize all markdown so <pre><code> is not used and instead backticks
rake posts:rebake                                                      # Update each post with latest markdown
rake posts:rebake_match[pattern,type,delay]                            # Rebake all posts matching string/regex and optionally delay the loop
rake posts:refresh_emails[topic_id]                                    # Refreshes each post that was received via email
rake posts:refresh_oneboxes                                            # Update each post with latest markdown and refresh oneboxes
rake posts:remap[find,replace,type]                                    # Remap all posts matching specific string
rake qunit:test[timeout,qunit_path]                                    # Runs the qunit test suite
rake release_note:generate[from,to]                                    # generate a release note from the important commits
rake restart                                                           # Restart app by touching tmp/restart.txt
rake routes                                                            # Print out all defined routes in match order, with names
rake scheduler:run_all                                                 # run every task the scheduler knows about in that order, use only for debugging
rake secret                                                            # Generate a cryptographically secure secret key (this is typically used to generate a secret for cookie sessions)
rake smoke:test                                                        # run chrome headless smoke tests on current build
rake stats                                                             # Report code statistics (KLOCs, etc) from the application or engine
rake test                                                              # Runs all tests in test folder except system ones
rake test:db                                                           # Run tests quickly, but also reset db
rake test:system                                                       # Run system tests only
rake time:zones[country_or_offset]                                     # List all time zones, list by two-letter country code (`rails time:zones[US]`), or list by UTC offset (`rails time:zones[-8]`)
rake tmp:clear                                                         # Clear cache and socket files from tmp/ (narrow w/ tmp:cache:clear, tmp:sockets:clear)
rake tmp:create                                                        # Creates tmp directories for cache, sockets, and pids
rake user_actions:rebuild                                              # rebuild the user_actions table
rake users:change_post_ownership[old_username,new_username,archetype]  # Change topic/post ownership of all the topics/posts by a specific user (without creating new revision)
rake yarn:install                                                      # Install all JavaScript dependencies as specified via Yarn

Am I doing something wrong or is it a bug?

2 Likes

Good catch, should be fixed per:

https://github.com/discourse/discourse/commit/83a346cd6879be75e6e9312e6b9d6edee1ae2c7b

4 Likes

Another bug related to this rake task.

When executing the task, even tough I have the s3_upload_bucket set, the task fails with the following error: The S3 upload bucket must be set before running that task.

https://github.com/discourse/discourse/blob/83a346cd6879be75e6e9312e6b9d6edee1ae2c7b/lib/tasks/uploads.rake#L121

I am not sure why this is required @zogstrip don’t we already store full urls in the upload table?

That rake task’s logic hasn’t been updated in 4 years…

@Apecengo, can you try again with this newer version?

https://github.com/discourse/discourse/commit/28d432263e58ae828470a87278f0c87a122be232

3 Likes

Doesn’t seem to be working as intended:

Migrating uploads from S3 to local storage for 'default'...
..........................................................Done!

Each . should be a #, if i have understood correctly the code:

https://github.com/discourse/discourse/blob/28d432263e58ae828470a87278f0c87a122be232/lib/tasks/uploads.rake#L150

My version: Discourse 2.0.0.beta1 - https://github.com/discourse/discourse version 782d75069ecc7d710df70bf0366e260ddfc59a72

1 Like

I’ve fixed the rake task to ensure we delete existing records before creating new ones (since we index based on the sha1 of the files, we weren’t able to create new records with the same sha1)

https://github.com/discourse/discourse/commit/cbf4204eccff62b6eac00932a04b827bff6cf883

4 Likes