Set up image uploads to Google Cloud Storage

Do you want to use Google Storage for your image uploads? Here are the steps:

  • Set a default Google project if you don’t have one.

  • Get a developer key.

  • Create a bucket by going here. Enter a unique name and select a default storage class (I used multi-regional within the US).

Now we need to configure the Discourse instance. Go to settings, and add a filter for S3. Then make the changes below:

  • Check the “ enable s3 backups ” checkbox if you want to activate manual or automated backups

  • Enter the desired bucket name created above in “ s3 backup bucket ” if enable s3 backups is checked

  • Paste in both “ Access Key ID ” and “ Secret Access Key ” in their respective text fields (these were generated when the developer key was created)

  • In s3_endpoint , paste in the below endpoint:

    • https://storage.googleapis.com

That’s it! Try uploading an image - it should all work magically :slightly_smiling_face:

To ‘serve’ images directly from your domain, follow the steps below:

  • Make sure your bucket is accessible publicly. It needs to set up using object level permissions though

  • Set up a load balancer if you don’t already have one. For the backend, create a backend bucket, point it to the storage bucket created above, enable CDN, and connect it

  • For the URL map, create two rules. Hosts for both need to be *; paths need to be /original/* and /optimized/*; and the backend needs to point to the backend bucket - save the settings

  • In Discourse settings, change the s3 cdn url to your domain - make sure this ties in with how your load balancer rule has been set up

That’s it - the images should now come from www.yourdomain.com/

Issues to be worked:

  • Backups - Uploading backups to a storage bucket is causing issues - will have a look at some point
10 Likes

Its working perfect.
Have tried two times.

4 Likes

On Google Storage? Are your buckets private?

2 Likes

yes on google cloud storage buckets.

Dont know about the private or public. Its on default by google as when you created.
But it provide you url for particular file publicly available. & I think thats the main use of cloud bucket to host images and make images publicly available on particular link.

But its working perfectly fine. It stores images in bucket & surf images on web from GCS bucket.

3 Likes

@akbarnoorani are you referring to storing images or backups? Images are okay but I run into issues with backups.

If you are storing backups, you should make your buckets private.

3 Likes

@P16 aaahhhaaannn… thats a good tip brother. thanks.

2 Likes

The bucket shows fine, but fails on upload, here is the log:

> [2019-12-01 01:27:30] Uploading archive...
[2019-12-01 01:27:30] EXCEPTION: Invalid argument.
[2019-12-01 01:27:30] /opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-core-3.48.6/lib/seahorse/client/request.rb:70:in `send_request'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/client.rb:995:in `create_multipart_upload'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/multipart_file_uploader.rb:52:in `initiate_upload'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/multipart_file_uploader.rb:43:in `upload'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/file_uploader.rb:32:in `upload'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/aws-sdk-s3-1.36.1/lib/aws-sdk-s3/customizations/object.rb:308:in `upload_file'
/opt/bitnami/discourse/lib/backup_restore/s3_backup_store.rb:43:in `upload_file'
/opt/bitnami/discourse/lib/backup_restore/backuper.rb:268:in `upload_archive'
/opt/bitnami/discourse/lib/backup_restore/backuper.rb:51:in `run'
/opt/bitnami/discourse/lib/backup_restore/backup_restore.rb:169:in `block in start!'
/opt/bitnami/discourse/lib/backup_restore/backup_restore.rb:166:in `fork'
/opt/bitnami/discourse/lib/backup_restore/backup_restore.rb:166:in `start!'
/opt/bitnami/discourse/lib/backup_restore/backup_restore.rb:20:in `backup!'
/opt/bitnami/discourse/app/controllers/admin/backups_controller.rb:40:in `create'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:194:in `process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/notifications.rb:168:in `instrument'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/abstract_controller/base.rb:134:in `process'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionview-5.2.3/lib/action_view/rendering.rb:32:in `process'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.0/lib/mini_profiler/profiling_methods.rb:78:in `block in profile_method'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:191:in `dispatch'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_controller/metal.rb:252:in `dispatch'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-protection-2.0.5/lib/rack/protection/frame_options.rb:31:in `call'
/opt/bitnami/discourse/lib/middleware/omniauth_bypass_middleware.rb:32:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:38:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call'
/opt/bitnami/discourse/lib/content_security_policy/middleware.rb:12:in `call'
/opt/bitnami/discourse/lib/middleware/anonymous_cache.rb:220:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/logster-2.3.2/lib/logster/middleware/reporter.rb:43:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:28:in `call'
/opt/bitnami/discourse/config/initializers/100-quiet_logger.rb:18:in `call'
/opt/bitnami/discourse/config/initializers/100-silence_logger.rb:31:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/bitnami/discourse/lib/middleware/enforce_hostname.rb:17:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-mini-profiler-1.1.0/lib/mini_profiler/profiler.rb:184:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/message_bus-2.2.0/lib/message_bus/rack/middleware.rb:57:in `call'
/opt/bitnami/discourse/lib/middleware/request_tracker.rb:163:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `public_send'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/railtie.rb:190:in `method_missing'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.4/src/ruby_supportlib/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.4/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.4/src/ruby_supportlib/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.4/src/ruby_supportlib/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads'
/opt/bitnami/discourse/vendor/bundle/ruby/2.6.0/gems/passenger-6.0.4/src/ruby_supportlib/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
[2019-12-01 01:27:30] Deleting old backups...
[2019-12-01 01:27:30] Cleaning stuff up...
[2019-12-01 01:27:30] Removing archive from local storage...
[2019-12-01 01:27:30] Removing '.tar' leftovers...
[2019-12-01 01:27:30] Marking backup as finished...
[2019-12-01 01:27:30] Refreshing disk stats...
[2019-12-01 01:27:31] Notifying 'user' of the end of the backup...
[2019-12-01 01:27:37] Finished!

When trying to upload a file through discourse manually, I get the following dialog:

The XML you provided was not well-formed or did not validate against our published schema.

If I upload the backup file through the Google Storage browser, it gets listed fine.

Any ideas?, I’m running Discourse 2.3.6 (via docker-compose)

Edit: Regular image uploads from a post fail as well, and I get a dialog saying “Invalid argument.”

1 Like

If this was installed from a bitnami package I’m afraid we can’t offer any support here. You will need to ask the package maintainer.

Either way you should be creating your own topic.

3 Likes