Problèmes de stratégie et d'autorisations avec les buckets S3 ?

Shane_Liebling · Décembre 23, 2018, 4:59

J’ai deux buckets S3, l’un pour les sauvegardes et l’autre pour les uploads. J’ai un seul utilisateur/stratégie pour les deux avec un accès S3 complet. Les uploads et les sauvegardes sont tous deux configurés pour aller vers S3. Étrangement, je peux effectuer une sauvegarde qui se retrouve bien sur S3, mais toute tentative d’upload d’image ou de pièce jointe renvoie une erreur « Accès refusé ».

J’ai essayé de reconstruire. J’ai créé de nouveaux buckets et de nouvelles stratégies. Rien de tout cela n’a aidé.

Cela semble également empêcher l’ajout d’images Gravatar. Sidekiq affiche plusieurs tâches en attente indiquant :

Jobs::HandledExceptionWrapper: Wrapped Aws::S3::Errors::AccessDenied: Accès refusé

Je peux ajouter manuellement des éléments dans le bucket uploads via le site AWS. Je peux également le faire en utilisant l’outil CLI AWS.

Avez-vous des idées ?

Shane_Liebling · Décembre 23, 2018, 5:07

Output from the logs:

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/plugins/response_target.rb:23:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/request.rb:70:in 
send_request' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/client.rb:5498:in 
put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:42:in 
block in put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:50:in 
open_file' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:41:in 
put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:34:in 
upload' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/customizations/object.rb:308:in 
upload_file' /var/www/discourse/lib/s3_helper.rb:28:in 
upload' /var/www/discourse/lib/file_store/s3_store.rb:48:in 
store_file' /var/www/discourse/lib/file_store/s3_store.rb:22:in 
store_upload' /var/www/discourse/lib/upload_creator.rb:126:in 
block (2 levels) in create_for' /var/www/discourse/lib/upload_creator.rb:125:in 
open' /var/www/discourse/lib/upload_creator.rb:125:in 
block in create_for' /var/www/discourse/lib/distributed_mutex.rb:34:in 
synchronize' /var/www/discourse/lib/distributed_mutex.rb:5:in 
synchronize' /var/www/discourse/lib/upload_creator.rb:36:in 
create_for' /var/www/discourse/app/models/user_avatar.rb:41:in 
block in update_gravatar!' /var/www/discourse/lib/distributed_mutex.rb:34:in 
synchronize' /var/www/discourse/lib/distributed_mutex.rb:5:in 
synchronize' /var/www/discourse/app/models/user_avatar.rb:14:in 
update_gravatar!' /var/www/discourse/app/jobs/regular/update_gravatar.rb:12:in 
execute' /var/www/discourse/app/jobs/base.rb:137:in 
block (2 levels) in perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in 
with_connection' /var/www/discourse/app/jobs/base.rb:127:in 
block in perform' /var/www/discourse/app/jobs/base.rb:123:in 
each' /var/www/discourse/app/jobs/base.rb:123:in 
perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in 
execute_job' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in 
block (2 levels) in process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in 
block in invoke' /var/www/discourse/lib/sidekiq/pausable.rb:81:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in 
block in invoke' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in 
invoke' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in 
block in process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in 
block (6 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in 
local' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in 
block (5 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in 
block in &lt;module:Sidekiq&gt;' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in 
block (4 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in 
stats' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in 
block (3 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_logger.rb:8:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in 
block (2 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in 
global' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in 
block in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in 
with_context' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in 
with_job_hash_context' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in 
dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in 
process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in 
process_one' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in 
run' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in 
watchdog' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in 
block in safe_thread'

merefield · Décembre 23, 2018, 5:14

If you see Access Denied, it may well be that it is indeed, denied.

I had a simliar issue recently, you need to loosen access restriction on AWS just enough to resolve it.

Shane_Liebling · Décembre 23, 2018, 5:15

The S3 user has all access to all buckets. (Uncertain why backups work but not upload.)

merefield · Décembre 23, 2018, 5:17

Mine was working fine for ages then broke, but modifying the access rights resolved it.

Somewhere there was a change.

Shane_Liebling · Décembre 23, 2018, 5:36

I can try, but this is my current policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

So the user should be able to do anything.

Shane_Liebling · Décembre 23, 2018, 6:29

Tried again with this, same issue - backups work but uploads do not:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutAccountPublicAccessBlock",
                "s3:GetAccountPublicAccessBlock",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:HeadBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::gauntlet-forums-backup",
                "arn:aws:s3:::gauntlet-forums-storage",
                "arn:aws:s3:::gauntlet-forums-backup/*",
                "arn:aws:s3:::gantlet-forums-storage/*"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::*/*"
        }
    ]
}

Shane_Liebling · Décembre 23, 2018, 7:13

Tried it with the policy in your post and in Setting up file and image uploads to S3 - still no dice.

gerhard · Décembre 23, 2018, 7:18

Looks like a typo to me.

Shane_Liebling · Décembre 23, 2018, 7:23

Fixed typo. No change. Still not working.

gerhard · Décembre 23, 2018, 7:25

Is your uploads bucket empty? Delete it and let Discourse create the bucket. Also, make sure you follow the instructions in Setting up file and image uploads to S3.

Shane_Liebling · Décembre 23, 2018, 7:28

Huzzah! Deleting the bucket fixed it! Thanks for that!

gerhard · Décembre 23, 2018, 7:33

That’s so weird. I’ve seen that outcome a couple of times in the last few weeks. I’m not sure what’s going on with manually created buckets yet.

Shane_Liebling · Décembre 23, 2018, 7:34

Strange to be sure… I swapped out that highly-permissive policy for one made from the two posts referenced and things seem good now too.

jidetheblogger · Décembre 23, 2018, 7:40

deleted the bucket from where? I have not been able to upload images on my forum too…

pfaffman · Décembre 23, 2018, 7:55

I had trouble getting uploads to work with digitalocean spaces yesterday, also with an error about the bucket already existing. I ran out of time before I could fully diagnose its. 'tis the season.

Shane_Liebling · Décembre 23, 2018, 11:16

I deleted the bucket from S3 and let Discourse create it itself.

zogstrip · Décembre 24, 2018, 10:48

I think that’s because S3 changed their default permission settings.

The settings when I create it manually

The settings when Discourse creates it

jidetheblogger · Décembre 24, 2018, 12:57

My settings are same with the discourse permission settings, yet images won’t upload.

Inspect elements shows something like "failed to load resource…

gerhard · Janvier 4, 2019, 1:40

Are you sure? You should see an “Access Denied” error if the upload fails. I thinks it’s a problem with a policy or CORS configuration. According to your screenshot the bucket permissions seem correct (both checkboxes under the “Manage public access control lists (ACLs)” are unchecked) and you “failed to load resource” looks like the browser simply isn’t allowed to load the image.

I’ve updated the instructions in Set up file and image uploads to S3. Maybe it helps you find the difference in your S3 bucket configuration.

Sujet		Réponses	Vues
S3 image upload access denied while backups upload working fine Self-hosting s3	22	2725	Mars 23, 2020
Set up file and image uploads to S3 Self-Hosting configuring , how-to	35	120043	Octobre 25, 2025
S3 Backup ... suspect access issue Support	4	1887	Octobre 16, 2018
Issues backing up to s3 Self-hosting backups , s3	5	994	Janvier 9, 2024
The bucket does not allow ACLs Support s3	5	2708	Décembre 3, 2022

Problèmes de stratégie et d'autorisations avec les buckets S3 ?

Sujets connexes