Probleme mit Richtlinien und Berechtigungen bei S3-Buckets?

Shane_Liebling · 23. Dezember 2018 um 16:59

Ich habe zwei S3-Buckets, einen für Backups und einen für Uploads. Für beide habe ich einen Benutzer/eine Richtlinie mit vollem S3-Zugriff eingerichtet. Sowohl Backups als auch Uploads sind so konfiguriert, dass sie nach S3 gehen. Seltsamerweise kann ich ein Backup erstellen, das dann auf S3 landet, aber jeder Versuch, Bilder hochzuladen oder Anhänge hinzuzufügen, führt zu „Access Denied"-Fehlern.

Ich habe versucht, das System neu aufzubauen. Ich habe neue Buckets und Richtlinien erstellt. Nichts davon hat geholfen.

Dies scheint auch das Hinzufügen von Gravatar-Bildern zu verhindern. Sidekiq hat eine Reihe ausstehender Aufgaben mit folgender Meldung:

Jobs::HandledExceptionWrapper: Wrapped Aws::S3::Errors::AccessDenied: Access Denied

Ich kann manuell über die AWS-Website Elemente zum Upload-Bucket hinzufügen. Das Gleiche kann ich auch mit dem AWS-CLI-Tool tun.

Irgendwelche Ideen?

Shane_Liebling · 23. Dezember 2018 um 17:07

Output from the logs:

/var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/plugins/response_target.rb:23:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-core-3.27.0/lib/seahorse/client/request.rb:70:in 
send_request' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/client.rb:5498:in 
put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:42:in 
block in put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:50:in 
open_file' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:41:in 
put_object' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/file_uploader.rb:34:in 
upload' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/aws-sdk-s3-1.19.0/lib/aws-sdk-s3/customizations/object.rb:308:in 
upload_file' /var/www/discourse/lib/s3_helper.rb:28:in 
upload' /var/www/discourse/lib/file_store/s3_store.rb:48:in 
store_file' /var/www/discourse/lib/file_store/s3_store.rb:22:in 
store_upload' /var/www/discourse/lib/upload_creator.rb:126:in 
block (2 levels) in create_for' /var/www/discourse/lib/upload_creator.rb:125:in 
open' /var/www/discourse/lib/upload_creator.rb:125:in 
block in create_for' /var/www/discourse/lib/distributed_mutex.rb:34:in 
synchronize' /var/www/discourse/lib/distributed_mutex.rb:5:in 
synchronize' /var/www/discourse/lib/upload_creator.rb:36:in 
create_for' /var/www/discourse/app/models/user_avatar.rb:41:in 
block in update_gravatar!' /var/www/discourse/lib/distributed_mutex.rb:34:in 
synchronize' /var/www/discourse/lib/distributed_mutex.rb:5:in 
synchronize' /var/www/discourse/app/models/user_avatar.rb:14:in 
update_gravatar!' /var/www/discourse/app/jobs/regular/update_gravatar.rb:12:in 
execute' /var/www/discourse/app/jobs/base.rb:137:in 
block (2 levels) in perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/rails_multisite-2.0.4/lib/rails_multisite/connection_management.rb:63:in 
with_connection' /var/www/discourse/app/jobs/base.rb:127:in 
block in perform' /var/www/discourse/app/jobs/base.rb:123:in 
each' /var/www/discourse/app/jobs/base.rb:123:in 
perform' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:187:in 
execute_job' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:169:in 
block (2 levels) in process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:128:in 
block in invoke' /var/www/discourse/lib/sidekiq/pausable.rb:81:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:130:in 
block in invoke' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/middleware/chain.rb:133:in 
invoke' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:168:in 
block in process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:139:in 
block (6 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:98:in 
local' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:138:in 
block (5 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq.rb:36:in 
block in &lt;module:Sidekiq&gt;' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:134:in 
block (4 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:199:in 
stats' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:129:in 
block (3 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_logger.rb:8:in 
call' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:128:in 
block (2 levels) in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/job_retry.rb:73:in 
global' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:127:in 
block in dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:48:in 
with_context' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/logging.rb:42:in 
with_job_hash_context' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:126:in 
dispatch' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:167:in 
process' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:85:in 
process_one' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:73:in 
run' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in 
watchdog' /var/www/discourse/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in 
block in safe_thread'

merefield · 23. Dezember 2018 um 17:14

If you see Access Denied, it may well be that it is indeed, denied.

I had a simliar issue recently, you need to loosen access restriction on AWS just enough to resolve it.

Shane_Liebling · 23. Dezember 2018 um 17:15

The S3 user has all access to all buckets. (Uncertain why backups work but not upload.)

merefield · 23. Dezember 2018 um 17:17

Mine was working fine for ages then broke, but modifying the access rights resolved it.

Somewhere there was a change.

Shane_Liebling · 23. Dezember 2018 um 17:36

I can try, but this is my current policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

So the user should be able to do anything.

Shane_Liebling · 23. Dezember 2018 um 18:29

Tried again with this, same issue - backups work but uploads do not:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutAccountPublicAccessBlock",
                "s3:GetAccountPublicAccessBlock",
                "s3:ListAllMyBuckets",
                "s3:ListBucket",
                "s3:HeadBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::gauntlet-forums-backup",
                "arn:aws:s3:::gauntlet-forums-storage",
                "arn:aws:s3:::gauntlet-forums-backup/*",
                "arn:aws:s3:::gantlet-forums-storage/*"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::*/*"
        }
    ]
}

Shane_Liebling · 23. Dezember 2018 um 19:13

Tried it with the policy in your post and in Setting up file and image uploads to S3 - still no dice.

gerhard · 23. Dezember 2018 um 19:18

Looks like a typo to me.

Shane_Liebling · 23. Dezember 2018 um 19:23

Fixed typo. No change. Still not working.

gerhard · 23. Dezember 2018 um 19:25

Is your uploads bucket empty? Delete it and let Discourse create the bucket. Also, make sure you follow the instructions in Setting up file and image uploads to S3.

Shane_Liebling · 23. Dezember 2018 um 19:28

Huzzah! Deleting the bucket fixed it! Thanks for that!

gerhard · 23. Dezember 2018 um 19:33

That’s so weird. I’ve seen that outcome a couple of times in the last few weeks. I’m not sure what’s going on with manually created buckets yet.

Shane_Liebling · 23. Dezember 2018 um 19:34

Strange to be sure… I swapped out that highly-permissive policy for one made from the two posts referenced and things seem good now too.

jidetheblogger · 23. Dezember 2018 um 19:40

deleted the bucket from where? I have not been able to upload images on my forum too…

pfaffman · 23. Dezember 2018 um 19:55

I had trouble getting uploads to work with digitalocean spaces yesterday, also with an error about the bucket already existing. I ran out of time before I could fully diagnose its. 'tis the season.

Shane_Liebling · 23. Dezember 2018 um 23:16

I deleted the bucket from S3 and let Discourse create it itself.

zogstrip · 24. Dezember 2018 um 10:48

I think that’s because S3 changed their default permission settings.

The settings when I create it manually

The settings when Discourse creates it

jidetheblogger · 24. Dezember 2018 um 12:57

My settings are same with the discourse permission settings, yet images won’t upload.

Inspect elements shows something like "failed to load resource…

gerhard · 4. Januar 2019 um 13:40

Are you sure? You should see an “Access Denied” error if the upload fails. I thinks it’s a problem with a policy or CORS configuration. According to your screenshot the bucket permissions seem correct (both checkboxes under the “Manage public access control lists (ACLs)” are unchecked) and you “failed to load resource” looks like the browser simply isn’t allowed to load the image.

I’ve updated the instructions in Set up file and image uploads to S3. Maybe it helps you find the difference in your S3 bucket configuration.

Thema		Antworten	Aufrufe
S3 image upload access denied while backups upload working fine Self-hosting s3	22	2725	23. März 2020
Set up file and image uploads to S3 Self-Hosting configuring , how-to	35	120047	25. Oktober 2025
S3 Backup ... suspect access issue Support	4	1887	16. Oktober 2018
Issues backing up to s3 Self-hosting backups , s3	5	994	9. Januar 2024
The bucket does not allow ACLs Support s3	5	2708	3. Dezember 2022

Probleme mit Richtlinien und Berechtigungen bei S3-Buckets?

Verwandte Themen