Amazon S3 job exception -- Filter element can only be used in Lifecycle V2

[RBoy]

After a recent update I’ve started seeing these errors crop up on the log. Never seen them before. Any idea why or how to fix them or what issues they may cause?

Message (3 copies reported)

Job exception: Found two rules with same prefix ‘tombstone/’
Backtrace

/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in call’
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in call’
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in call’
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in call’
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/response_target.rb:23:in call' /var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/request.rb:70:in send_request’
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/client.rb:4340:in put_bucket_lifecycle' /var/www/discourse/lib/s3_helper.rb:72:in update_lifecycle’
/var/www/discourse/lib/s3_helper.rb:80:in update_tombstone_lifecycle' /var/www/discourse/lib/file_store/s3_store.rb:82:in purge_tombstone’
/var/www/discourse/app/jobs/scheduled/purge_deleted_uploads.rb:8:in execute' /var/www/discourse/app/jobs/base.rb:153:in block (2 levels) in perform’
Env

hostname MCEForum-app
process_id 369
application_version 127c13314a71614d71e7236fc09f8c7fb68176f1
current_db default
current_hostname discussion.mcebuddy2x.com
job Jobs::PurgeDeletedUploads
problem_db default
opts

I’m also seeing other warnings which I haven’t seen before:

Uncaught SyntaxError: Unexpected token else
Url: 
Line: 6
Column: 10
Window Location: https://discussion.mcebuddy2x.com/t/network-credentials/604
Backtrace

SyntaxError: Unexpected token else
Env

hostname	MCEForum-app
process_id	451
application_version	127c13314a71614d71e7236fc09f8c7fb68176f1
HTTP_HOST	discussion.mcebuddy2x.com
REQUEST_URI	/logs/report_js_error
REQUEST_METHOD	POST
HTTP_USER_AGENT	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Safari/537.36
HTTP_ACCEPT	*/*
HTTP_REFERER	https://discussion.mcebuddy2x.com/t/network-credentials/604
HTTP_X_FORWARDED_FOR	109.148.89.254
HTTP_X_REAL_IP	109.148.89.254
params	
message	Uncaught SyntaxError: Unexpected token else Url: Line: 6 Column: 10 Window Location: https://discuss
url	
line	6
column	10
window_location	https://discussion.mcebuddy2x.com/t/network-credentials/604
stacktrace	SyntaxError: Unexpected token else

And

Message

Uncaught SyntaxError: Unexpected token var
Url: data:application/javascript;base64,dmFyIHVyY2hpblRyYWNrZXI9ZnVuY3Rpb24oKXt9LF9nYXE9e3B1c2g6ZnVuY3Rpb24oKXt0cnkge2lmKGFyZ3VtZW50c1swXVswXT09J19saW5rJyl3aW5kb3cubG9jYXRpb24uaHJlZj1hcmd1bWVudHNbMF1bMV19Y2F0Y2goZXIpe319fSxfZ2F0PXtfY3JlYXRlVHJhY2tlcjpmdW5jdGlvbigpe30sIF9nZXRUcmFja2VyOmZ1bmN0aW9uKCl7cmV0dXJue19fbm9TdWNoTWV0aG9kX186ZnVuY3Rpb24oKXt9LF9saW5rOmZ1bmN0aW9uKG8pe2lmKG8pbG9jYXRpb24uaHJlZj1vO30sX2xpbmtCeVBvc3Q6ZnVuY3Rpb24oKXtyZXR1cm4gdHJ1ZTt9LF9nZXRMaW5rZXJVcmw6ZnVuY3Rpb24obyl7cmV0dXJuIG87fSxfdHJhY2tFdmVudDpmdW5jdGlvbigpe319fX07Y3hBcGk9e3NldENvb2tpZVBhdGg6ZnVuY3Rpb24oKXt9LGNob29zZVZhcmlhdGlvbjpmdW5jdGlvbigpe319dmFyIF9nYXQ9e19nZXRUcmFja2VyQnlOYW1lOiBmdW5jdGlvbigpe190cmFja0V2ZW50OiBmdW5jdGlvbigpe319fTs=
Line: 1
Column: 439
Window Location: https://discussion.mcebuddy2x.com/t/big-brother-season-19-conversion-issues/275/6
Backtrace

SyntaxError: Unexpected token var
Env

hostname	MCERBoyForum-app
process_id	451
application_version	127c13314a71614d71e7236fc09f8c7fb68176f1
HTTP_HOST	discussion.mcebuddy2x.com
REQUEST_URI	/logs/report_js_error
REQUEST_METHOD	POST
HTTP_USER_AGENT	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
HTTP_ACCEPT	*/*
HTTP_REFERER	https://discussion.mcebuddy2x.com/t/big-brother-season-19-conversion-issues/275/6
HTTP_X_FORWARDED_FOR	69.203.122.211
HTTP_X_REAL_IP	69.203.122.211
params	
message	Uncaught SyntaxError: Unexpected token var Url: data:application/javascript;base64,dmFyIHVyY2hpblRyYW
url	data:application/javascript;base64,dmFyIHVyY2hpblRyYWNrZXI9ZnVuY3Rpb24oKXt9LF9nYXE9e3B1c2g6ZnVuY3Rpb2
line	1
column	439
window_location	https://discussion.mcebuddy2x.com/t/big-brother-season-19-conversion-issues/275/6
stacktrace	SyntaxError: Unexpected token var

[sam (Sam Saffron)]

Are you running absolute latest? I had a regression here for a short period

[RBoy]

This was as of a few days ago. Will try it later tonight.

[RBoy]

Okay so after updating to the latest version and giving it a day, I’m still seeing the same warnings but different errors now:

This is the error:

Message
Job exception: Filter element can only be used in Lifecycle V2.
Backtrace

/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-core-3.6.0/lib/seahorse/client/request.rb:70:in `send_request'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/aws-sdk-s3-1.4.0/lib/aws-sdk-s3/client.rb:4442:in `put_bucket_lifecycle_configuration'
/var/www/discourse/lib/s3_helper.rb:109:in `update_lifecycle'
/var/www/discourse/lib/s3_helper.rb:118:in `update_tombstone_lifecycle'
/var/www/discourse/lib/file_store/s3_store.rb:77:in `purge_tombstone'
/var/www/discourse/app/jobs/scheduled/purge_deleted_uploads.rb:8:in `execute'
/var/www/discourse/app/jobs/base.rb:134:in `block in perform'
/var/www/discourse/app/jobs/base.rb:125:in `each'
/var/www/discourse/app/jobs/base.rb:125:in `perform'
/var/www/discourse/app/jobs/base.rb:180:in `perform'
/var/www/discourse/lib/scheduler/manager.rb:95:in `process_queue'
/var/www/discourse/lib/scheduler/manager.rb:37:in `block in initialize'

and I’m getting repeated warnings here:

Message (4 copies reported)
Uncaught SyntaxError: Unexpected token var
Url: data:application/javascript;base64,dmFyIHVyY2hpblRyYWNrZXI9ZnVuY3Rpb24oKXt9LF9nYXE9e3B1c2g6ZnVuY3Rpb24oKXt0cnkge2lmKGFyZ3VtZW50c1swXVswXT09J19saW5rJyl3aW5kb3cubG9jYXRpb24uaHJlZj1hcmd1bWVudHNbMF1bMV19Y2F0Y2goZXIpe319fSxfZ2F0PXtfY3JlYXRlVHJhY2tlcjpmdW5jdGlvbigpe30sIF9nZXRUcmFja2VyOmZ1bmN0aW9uKCl7cmV0dXJue19fbm9TdWNoTWV0aG9kX186ZnVuY3Rpb24oKXt9LF9saW5rOmZ1bmN0aW9uKG8pe2lmKG8pbG9jYXRpb24uaHJlZj1vO30sX2xpbmtCeVBvc3Q6ZnVuY3Rpb24oKXtyZXR1cm4gdHJ1ZTt9LF9nZXRMaW5rZXJVcmw6ZnVuY3Rpb24obyl7cmV0dXJuIG87fSxfdHJhY2tFdmVudDpmdW5jdGlvbigpe319fX07Y3hBcGk9e3NldENvb2tpZVBhdGg6ZnVuY3Rpb24oKXt9LGNob29zZVZhcmlhdGlvbjpmdW5jdGlvbigpe319dmFyIF9nYXQ9e19nZXRUcmFja2VyQnlOYW1lOiBmdW5jdGlvbigpe190cmFja0V2ZW50OiBmdW5jdGlvbigpe319fTs=

[sam (Sam Saffron)]

@RBoy

Filter element can only be used in Lifecycle V2.

yeah … is this cause your s3 bucket is old or something and needs some sort of magic Amazon upgrade?

[sam (Sam Saffron)]

Hmm looks related:

Can't set Bucket Lifecycle Configuration

[RBoy]

Okay so going through your link it looks like I need to update the bucket to support lifecycle.

BTW just for the record, this bucket was created a few months ago, I just went to S3 and created a new bucket. Does discourse need a special type of bucket?

I looked at the Amazon S3 bucket and it does have lifecycle enabled on it. I also see a purge-tombstone rule in it which I’m assuming was created by discourse (possibly related to my original error post) and lifecycle is enabled.

Here is a screenshot:

So what needs to be done here?

EDIT: Also what’s the implication of this error? Can I just delete the Tombstone rule and let discourse recreate it? This started after the latest update.

EDIT 2: I was exploring the amazon docs and according to the docs

You can use lifecycle policies to define actions you want Amazon S3 to take during an object’s lifetime (for example, transition objects to another storage class, archive them, or delete them after a specified period of time).

Looks like this rule was created by discourse. Going back to my original error, where there was a duplicate tombstone rule, do you think there was a bug in the code which caused it to create a rule of the wrong type? If so, can’t the code look at the rule it previously created and if it runs into the error, delete it and recreate it using the right type of rule? This isn’t something I created manually for sure.

[RBoy]

I’m wondering if it’s related to this commit

FEATURE: rake tasks for uploading assets to S3

Looks like discourse went from manually creating the lifecycle rules for tombstone:

 -        rules: [
 -          {
 -            id: "purge-tombstone",
 -            status: "Enabled",
 -            expiration: { days: grace_period },
 -            prefix: @tombstone_prefix
 -          }
 -        ]
 -      })

to getting it from S3

rules = s3_resource.client.get_bucket_lifecycle_configuration(bucket: @s3_bucket_name).rules

while simultaneously upgrading the gems from (to v3?)

-gem ‘aws-sdk’, require: false
+gem ‘aws-sdk-s3’, require: false

Could it be a combination of these changes that caused this error to pop up. This is probably going to impact any deployment who was on that BETA build, maybe they just haven’t seen their error logs.

[sam (Sam Saffron)]

Yes it is my change following up with the gem creators to confirm why

[RBoy]

Sam,

Any word on this?

Importantly what’s the implication of error while we wait for a fix? Does it impact operations, backup or something else?

[sam (Sam Saffron)]

Still on my list to sort out, the error has no impact if you already have the tombstone rule setup.

Worst case if you don’t then tombstone images don’t get purged from s3

[sam (Sam Saffron)]

Reported to AWS Ruby team to get guidance here:

Filter element can only be used in Lifecycle V2, when updating lifecycle policy

[RBoy]

Thanks, I’ll keep on eye on it.

[sam (Sam Saffron)]

Can you try contacting Amazon support to ask them how to upgrade your bucket to lifecycle v2 and link to the issue? It will help get more visibility.

Report back here once you hear back from them.

I confirmed there was no place in the UI to change this I searched everywhere.

[RBoy]

Apparently Amazon doesn’t provide support for regular accounts. I’m going to try to post it on the forum to see if anyone has any idea but I’m AMAZED that Amazon is providing libraries that breaks their setup and like I said this is a new setup with Amazon barely a year old.

[RBoy]

Okay here are two responses

So to clarify a bit, there is no V1 and V2 bucket concept, you just need to make sure you are using the put_bucket_lifecycle_configuration operation instead of put_bucket_lifecycle for all targeting buckets. If you still see an issue, could you try setting :http_wire_trace to true to give us a full http wire trace for debugging?

and

Forums

Hello,

The error you're experiencing is caused by using the 'Prefix' base tag and the 'Filter' base tag in a single Lifecycle policy. These tags are mutually exclusive and cannot be combined within a single policy.

You'll stop seeing this error if you wrap all of your base 'Prefix' tags in a 'Filter' tag. Here's an example of a broken policy and what it should look like in order for it to work. 

In the bad policy, Rule 1 uses a base 'Prefix' tag and Rule 2 wraps the 'Prefix' in a 'Filter' tag. In order to address this, the corrected policy has Rule 1 wrap its 'Prefix' tag in a 'Filter' tag. The tags that cause this issue are in bold for clarity.

Bad Policy
<LifecycleConfiguration>
<Rule>
<ID>Rule1</ID>
<Prefix>a</Prefix>
<Expiration>
<Days>2048</Days>
</Expiration>
<Status>Enabled</Status>
</Rule>
<Rule>
<ID>Rule2</ID>
<Filter>
<Prefix>b</Prefix>
</Filter>
<Transition>
<Days>1024</Days>
<StorageClass>STANDARD_IA</StorageClass>
</Transition>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>

Corrected Policy
<LifecycleConfiguration>
<Rule>
<ID>Rule1</ID>
<Filter>
<Prefix>a</Prefix>
</Filter>
<Expiration>
<Days>2048</Days>
</Expiration>
<Status>Enabled</Status>
</Rule>
<Rule>
<ID>Rule2</ID>
<Filter>
<Prefix>b</Prefix>
</Filter>
<Transition>
<Days>1024</Days>
<StorageClass>STANDARD_IA</StorageClass>
</Transition>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>

Please let me know if you have any further questions.

[sam (Sam Saffron)]

That is super helpful … looking at the code now to see if I can fix it!

[sam (Sam Saffron)]

Fixed per:

FIX: error setting tombstone bucket when set to old version

Testing s3 sure ain’t easy.

Also updated the original bug in the aws library, cause there is an actual underlying bug I am working around that needs fixing.

[RBoy]

Great thanks for looking into it. So for clarification, when Amazon fixes the bug in the library, your patch would still work or would it need updating?

[sam (Sam Saffron)]

My patch will continue to work even when the library is fixed. Even the test should, in theory continue to work.