Problemas de política e permissão com buckets S3?

Shane_Liebling · Dezembro 23, 2018, 4:59pm

Tenho dois buckets S3: um para backups e outro para uploads. Configurei um único usuário/política para ambos, com acesso total ao S3. Tanto os uploads quanto os backups estão configurados para ir ao S3. Estranhamente, consigo fazer um backup e ele é enviado ao S3, mas qualquer tentativa de upload de imagem ou anexo resulta em erros de “Acesso Negado”.

Tentei reconstruir o sistema. Criei novos buckets e políticas. Nada disso ajudou.

Isso também parece impedir a adição de imagens do Gravatar. O Sidekiq apresenta várias tarefas pendentes com a seguinte mensagem:

Jobs::HandledExceptionWrapper: Wrapped Aws::S3::Errors::AccessDenied: Access Denied

Consigo adicionar itens manualmente ao bucket de uploads pelo site da AWS e também usando a ferramenta de CLI da AWS.

Alguma ideia do que possa estar acontecendo?

Shane_Liebling · Dezembro 23, 2018, 5:07pm

Output from the logs:

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/plugins/response_target.rb:23:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/request.rb:70:in 
send_request' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/client.rb:5498:in 
put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:42:in 
block in put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:50:in 
open_file' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:41:in 
put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:34:in 
upload' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/customizations/object.rb:308:in 
upload_file' /var/www/discourse/lib/s3_helper.rb:28:in 
upload' /var/www/discourse/lib/file_store/s3_store.rb:48:in 
store_file' /var/www/discourse/lib/file_store/s3_store.rb:22:in 
store_upload' /var/www/discourse/lib/upload_creator.rb:126:in 
block (2 levels) in create_for' /var/www/discourse/lib/upload_creator.rb:125:in 
open' /var/www/discourse/lib/upload_creator.rb:125:in 
block in create_for' /var/www/discourse/lib/distributed_mutex.rb:34:in 
synchronize' /var/www/discourse/lib/distributed_mutex.rb:5:in 
synchronize' /var/www/discourse/lib/upload_creator.rb:36:in 
create_for' /var/www/discourse/app/models/user_avatar.rb:41:in 
block in update_gravatar!' /var/www/discourse/lib/distributed_mutex.rb:34:in 
synchronize' /var/www/discourse/lib/distributed_mutex.rb:5:in 
synchronize' /var/www/discourse/app/models/user_avatar.rb:14:in 
update_gravatar!' /var/www/discourse/app/jobs/regular/update_gravatar.rb:12:in 
execute' /var/www/discourse/app/jobs/base.rb:137:in 
block (2 levels) in perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in 
with_connection' /var/www/discourse/app/jobs/base.rb:127:in 
block in perform' /var/www/discourse/app/jobs/base.rb:123:in 
each' /var/www/discourse/app/jobs/base.rb:123:in 
perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in 
execute_job' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in 
block (2 levels) in process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in 
block in invoke' /var/www/discourse/lib/sidekiq/pausable.rb:81:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in 
block in invoke' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in 
invoke' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in 
block in process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in 
block (6 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in 
local' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in 
block (5 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in 
block in &lt;module:Sidekiq&gt;' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in 
block (4 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in 
stats' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in 
block (3 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_logger.rb:8:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in 
block (2 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in 
global' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in 
block in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in 
with_context' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in 
with_job_hash_context' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in 
dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in 
process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in 
process_one' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in 
run' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in 
watchdog' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in 
block in safe_thread'

merefield · Dezembro 23, 2018, 5:14pm

If you see Access Denied, it may well be that it is indeed, denied.

I had a simliar issue recently, you need to loosen access restriction on AWS just enough to resolve it.

Shane_Liebling · Dezembro 23, 2018, 5:15pm

The S3 user has all access to all buckets. (Uncertain why backups work but not upload.)

merefield · Dezembro 23, 2018, 5:17pm

Mine was working fine for ages then broke, but modifying the access rights resolved it.

Somewhere there was a change.

Shane_Liebling · Dezembro 23, 2018, 5:36pm

I can try, but this is my current policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

So the user should be able to do anything.

Shane_Liebling · Dezembro 23, 2018, 6:29pm

Tried again with this, same issue - backups work but uploads do not:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutAccountPublicAccessBlock",
                "s3:GetAccountPublicAccessBlock",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:HeadBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::gauntlet-forums-backup",
                "arn:aws:s3:::gauntlet-forums-storage",
                "arn:aws:s3:::gauntlet-forums-backup/*",
                "arn:aws:s3:::gantlet-forums-storage/*"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::*/*"
        }
    ]
}

Shane_Liebling · Dezembro 23, 2018, 7:13pm

Tried it with the policy in your post and in Setting up file and image uploads to S3 - still no dice.

gerhard · Dezembro 23, 2018, 7:18pm

Looks like a typo to me.

Shane_Liebling · Dezembro 23, 2018, 7:23pm

Fixed typo. No change. Still not working.

gerhard · Dezembro 23, 2018, 7:25pm

Is your uploads bucket empty? Delete it and let Discourse create the bucket. Also, make sure you follow the instructions in Setting up file and image uploads to S3.

Shane_Liebling · Dezembro 23, 2018, 7:28pm

Huzzah! Deleting the bucket fixed it! Thanks for that!

gerhard · Dezembro 23, 2018, 7:33pm

That’s so weird. I’ve seen that outcome a couple of times in the last few weeks. I’m not sure what’s going on with manually created buckets yet.

Shane_Liebling · Dezembro 23, 2018, 7:34pm

Strange to be sure… I swapped out that highly-permissive policy for one made from the two posts referenced and things seem good now too.

jidetheblogger · Dezembro 23, 2018, 7:40pm

deleted the bucket from where? I have not been able to upload images on my forum too…

pfaffman · Dezembro 23, 2018, 7:55pm

I had trouble getting uploads to work with digitalocean spaces yesterday, also with an error about the bucket already existing. I ran out of time before I could fully diagnose its. 'tis the season.

Shane_Liebling · Dezembro 23, 2018, 11:16pm

I deleted the bucket from S3 and let Discourse create it itself.

zogstrip · Dezembro 24, 2018, 10:48am

I think that’s because S3 changed their default permission settings.

The settings when I create it manually

The settings when Discourse creates it

jidetheblogger · Dezembro 24, 2018, 12:57pm

My settings are same with the discourse permission settings, yet images won’t upload.

Inspect elements shows something like "failed to load resource…

gerhard · Janeiro 4, 2019, 1:40pm

Are you sure? You should see an “Access Denied” error if the upload fails. I thinks it’s a problem with a policy or CORS configuration. According to your screenshot the bucket permissions seem correct (both checkboxes under the “Manage public access control lists (ACLs)” are unchecked) and you “failed to load resource” looks like the browser simply isn’t allowed to load the image.

I’ve updated the instructions in Set up file and image uploads to S3. Maybe it helps you find the difference in your S3 bucket configuration.

Tópico		Respostas	Visualizações
S3 image upload access denied while backups upload working fine Self-hosting s3	22	2725	23 de Março de 2020
Set up file and image uploads to S3 Self-Hosting configuring , how-to	35	120051	25 de Outubro de 2025
S3 Backup ... suspect access issue Support	4	1887	16 de Outubro de 2018
Issues backing up to s3 Self-hosting backups , s3	5	994	9 de Janeiro de 2024
The bucket does not allow ACLs Support s3	5	2708	3 de Dezembro de 2022

Problemas de política e permissão com buckets S3?

Tópicos relacionados