Images are disappearing off of s3!


(Sam Saffron) #22

Just copying stuff out of tombstone is not enough recover an image. At that point the upload record is already deleted.

How big is this problem on your site, how many posts do you have impacted by this? Simplest thing to do if say 5 or less is just to edit the post and re-upload the image from tombstone.


(Alan Tan) #23

@davidkingham you can run the following to find out how many uploads are broke.

./launcher enter app
rake uploads:list_broken_posts

(David Kingham) #24

That command didn’t work, but uploads:list_posts_with_broken_images shows that almost 300 of the images are broken now. The files were just copied back into tombstone, so obviously the issue still remains after running the rebake. How would I properly run the rsync to copy the files back over without breaking them?


(Sam Saffron) #25

300 is a lot! Is there a pattern here? When was the first post with missing images posted? When was the last one.

We are going to work on a rake task to correct this corruption as long as the file exists either in the tombstone or in the expected location.


(Alan Tan) #26

@davidkingham Can you try out the new rake task?

./launcher enter app
RECOVER_FROM_S3=1 rake uploads:list_posts_with_broken_images

(David Kingham) #27

Sadly that did not work @tgxworld

Here is what it ran through, I checked to see if it wrote the new information to the posts and it matches up, but I still just see a transparent png

Summary
https://community.naturephotographers.network/t/what-are-these-sliders-in-the-tone-curve-for/70/1 upload://yRrAlAEE6wSM8vrx5N632swWUQE.png
https://community.naturephotographers.network/t/badlands-fog/112/1 upload://2eVsu0Kgo5YDg8mWplnqUO5nMLw.jpg
https://community.naturephotographers.network/t/crested-butte-today/145/1 upload://vMysDF5D4dHVFAxBw7FQtsRwjir.jpeg
https://community.naturephotographers.network/t/crested-butte-today/145/1 upload://r1CmRMoaB2U07JXj00pwd7GBeIQ.jpeg
https://community.naturephotographers.network/t/assets-for-the-site-design/7/2 upload://6ZtbpWbdq9a5dIpTFFmSCMO9Gf3.png
https://community.naturephotographers.network/t/assets-for-the-site-design/7/2 upload://89vl8qjUTSd52t7j1saqKCz0xnN.png
https://community.naturephotographers.network/t/assets-for-the-site-design/7/2 upload://3frdgEVjB4MUTS5pHgzMRWPNtrW.png
https://community.naturephotographers.network/t/assets-for-the-site-design/7/2 upload://sKURr24CAiTfRsoRqGd5wSa73vS.png
https://community.naturephotographers.network/t/assets-for-the-site-design/7/2 upload://f6UliEhuGKEbstcxDYFdWIzFK9X.png
https://community.naturephotographers.network/t/assets-for-the-site-design/7/2 upload://qlGOT9mTAW3JXLegtm8hUTSckSF.png
https://community.naturephotographers.network/t/assets-for-the-site-design/7/2 upload://qMBSj7Pz5yO2EQZZFQaElsJmUE3.png
https://community.naturephotographers.network/t/assets-for-the-site-design/7/2 upload://hh9zF8jOcpn1xwKmh651DginiqO.png
https://community.naturephotographers.network/t/the-wall-with-a-thousand-eyes/191/1 upload://xe1RRwdThiAGFASPaaCj3Sxv2ic.jpeg
https://community.naturephotographers.network/t/peachy/196/1 upload://42zal4rSEGXOM4qBqOLYvVyGuv.jpeg
https://community.naturephotographers.network/t/buried-in-the-sand/197/1 upload://5CsxHH02Z7cn0HR9c3V802RsoIy.jpeg
https://community.naturephotographers.network/t/rufous-hummingbird-interesting-wing-position-and-blur-from-a-multi-flash-setup/199/1 upload://fQEOOgvCSCTLbFkjSNp8exDLYm5.jpeg
https://community.naturephotographers.network/t/changes-from-the-old-npn/198/10 upload://yS3MSP0YnAQDM0W1kyjUGL0Vh3M.png
https://community.naturephotographers.network/t/peachy/212/1 upload://42zal4rSEGXOM4qBqOLYvVyGuv.jpeg
https://community.naturephotographers.network/t/summer-spires-composition-inadequacies/213/1 upload://eS7Vx4RPq8R4zYVqVyigLMIXPTj.jpeg
https://community.naturephotographers.network/t/dolphin-voyage/215/1 upload://br3R2djUq5oWaDGFKwyiIRYufA7.jpeg
https://community.naturephotographers.network/t/flourishing-life-zion-national-park-ut-2017/216/1 upload://cegUHKD8MVj4SggltagGt7Mfz5o.jpeg
https://community.naturephotographers.network/t/resurgence-patagonia-argentina-2017/217/1 upload://orJOllaZbsrtjeiQbUsOyGS1it9.jpeg
https://community.naturephotographers.network/t/dolphin-voyage/215/5 upload://wOh6LZntSjPLaEbIsfy5bqwIqjP.jpeg
https://community.naturephotographers.network/t/perfect-water-lily-2x-repost/219/1 upload://jqVkIyE28b0UedGyF0ROO1Q15Ce.jpeg
https://community.naturephotographers.network/t/about-the-people-objects-places-gallery-category/88/2 upload://aRF7Wm2ZgohtIpUtPHRgCmm6mN1.jpeg
https://community.naturephotographers.network/t/smokeys-place/221/1 upload://aRF7Wm2ZgohtIpUtPHRgCmm6mN1.jpeg
https://community.naturephotographers.network/t/summer-spires-composition-inadequacies/213/4 upload://uudEjbvB3H0y4ZPZQlsqC8dKUxb.jpeg
https://community.naturephotographers.network/t/alligator-in-the-swamp/251/1 upload://ijBnAWuq1wArDk1Z9m364s81943.jpeg
https://community.naturephotographers.network/t/lucid-dreams-lines-textures-and-light-of-the-desert-southwest/223/1 upload://uRnavyHPZ0LzKUE8GehmjEm8xTN.jpeg
https://community.naturephotographers.network/t/crazy-beautiful-banff/224/1 upload://xkHCR5pdVmYxVCACDvkBB266mYG.jpeg
https://community.naturephotographers.network/t/trichotomy-amargosa-mountains-from-badwater-basin-dvnp/227/1 upload://nmPbTU1IfEF38dUhtQBbepPqppF.jpeg
https://community.naturephotographers.network/t/desert-moon/228/1 upload://rw2fqE0IHYVTPTy12eutTC01HAV.jpeg
https://community.naturephotographers.network/t/desert-moon/228/1 upload://vuAvCnkm7ytKJsgzGSRs9lZ69F3.jpeg
https://community.naturephotographers.network/t/perfect-water-lily-2x-repost/219/5 upload://2x419g1Qn8BT78JaDvtjUSKihu7.jpeg
https://community.naturephotographers.network/t/black-and-white-critique/231/1 upload://yz4VWoHn5UANE1yUkB3x4lrJN0C.jpeg
https://community.naturephotographers.network/t/weekly-picks/238/1 upload://2eVsu0Kgo5YDg8mWplnqUO5nMLw.jpg
https://community.naturephotographers.network/t/death-valley-workshop-exploring-exposure-dec-5-9-2018/243/1 upload://oOKtj6ZKYeKmB7WvEyt9n8NmMGx.jpeg
https://community.naturephotographers.network/t/blue-dasher-obelisk/246/1 upload://l59PUwc7y5ntrCnNyNtMcXecGbc.jpeg
https://community.naturephotographers.network/t/dueling-coronets/250/1 upload://ellmoKr6Ig1wQi3HhaIeyL0ovux.jpeg
https://community.naturephotographers.network/t/cobalt-blues/254/1 upload://ajLZNEq45XKTTGCCHIf6h6JicbZ.jpeg
https://community.naturephotographers.network/t/pygmy-marmoset/255/1 upload://6qq8ZVe2dIpbbBaI3RKeYLgqVn0.jpeg
https://community.naturephotographers.network/t/hope/256/1 upload://3XDFgSdLKXDlPwcsOUHx9cFkFUO.jpeg
https://community.naturephotographers.network/t/launch-date-for-existing-npn-members/226/19 upload://eCJDTYZnfUwLEIoumfElIJPMWYB.png
https://community.naturephotographers.network/t/black-mantled-tamarins/258/1 upload://pvuSe2fM42giFDFeV1Y8TUv7ELT.jpeg
https://community.naturephotographers.network/t/a-couple-of-lazy-dogs/259/1 upload://pFSXSGt3xSSrSZy1pO9FYJmLlzk.jpeg
https://community.naturephotographers.network/t/split-falls/260/1 upload://qETVPyD1sMvY1w0EdHZFkNgE7gq.jpeg
https://community.naturephotographers.network/t/hope/256/4 upload://xkicgyEftvwF7761ZXcp7Zbk0fO.jpeg
https://community.naturephotographers.network/t/ruffled-day-lily/261/1 upload://e6LdQo0j8N7SsKB6iyCDEElpmjK.jpeg
https://community.naturephotographers.network/t/grand-teton-sunrise/264/1 upload://qgMUVJSZtVbkxjQdoXqiM6V6bvu.jpeg
https://community.naturephotographers.network/t/soft-symphony-direct-light-on-proxy-falls/265/1 upload://7bt0vIHw11t7SYluvCU9SGgiEK5.jpeg
https://community.naturephotographers.network/t/textured-take-on-a-hummingbird/266/1 upload://k8velwqhNXwE0yLgkJ3Z83cWLfd.jpeg
https://community.naturephotographers.network/t/sacred-lotus-flower-in-black-and-white/267/1 upload://qxdqpDS4HN9vSGPjgyfzDpBt5Fo.jpeg
https://community.naturephotographers.network/t/midsummers-dream/273/1 upload://g1SKUPKLj3a2rCUlwBmnlL15ijt.jpeg
https://community.naturephotographers.network/t/a-post-in-midsummers-dream-requires-staff-attention/274/3 upload://qyFHVbIveF8AR20MEtXioLE3RY4.png
https://community.naturephotographers.network/t/a-post-in-midsummers-dream-requires-staff-attention/274/3 upload://3muIHythQ2mNbAdAFMybyLyKtF3.png
https://community.naturephotographers.network/t/mount-rainier-sunrise/276/1 upload://AcnboiSPx2I5xesSeS2a2RIZetP.jpeg
https://community.naturephotographers.network/t/black-winged-kite/277/1 upload://apnoGxGBujtVRniiMjKmXeecNsR.jpeg
https://community.naturephotographers.network/t/we-thought-it-was-a-rainbow/278/1 upload://e056z5WcEWDvFZwhkRn5rCda8qx.jpeg
https://community.naturephotographers.network/t/solitude-on-a-montana-evening/280/1 upload://vvQwCLZuTxlWQ7bm0Wc9UYrXl5B.jpeg
https://community.naturephotographers.network/t/osprey-in-flight/281/1 upload://v5qzF5LOmTkutzMu8hNY7ZLTLte.jpeg
https://community.naturephotographers.network/t/sunset-at-la-sal/286/1 upload://piRWtqWm6POFX4ElTXnvqZiYejC.jpeg
https://community.naturephotographers.network/t/no-poster-names-in-the-gallery-forums/285/2 upload://fhESdVJosnuwc3tIyH9xvkZ5Rov.png
https://community.naturephotographers.network/t/this-turned-out-not-to-be-a-rainbow/288/1 upload://9wWycJ0H37GR5ZA6YyqZk0MItBY.jpeg
https://community.naturephotographers.network/t/circle-of-life-olympic-national-park-wa-2018/289/1 upload://8zoITMC02ktq4XNDcB4hxWZCKso.jpeg
https://community.naturephotographers.network/t/sunset-at-la-sal/286/6 upload://uTfaKjDqv1AMXGzSwKBNN6DEMex.jpeg
https://community.naturephotographers.network/t/remember-me-spring-at-ponytail-falls-columbia-river-gorge/292/1 upload://eHDN9Ir2AlwyK9kxItnlHZUniAd.jpeg
https://community.naturephotographers.network/t/greater-roadrunner-portrait/296/1 upload://kfOVfNLnR1yqePAgb1EGookswjl.jpeg
https://community.naturephotographers.network/t/the-look/297/1 upload://56xVfa2AzJOXpIaovmCUsz4aeSY.jpeg
https://community.naturephotographers.network/t/common-ringed-plover/299/1 upload://xkVKpVUr0nyLpERmxeW06IuSgfI.jpeg
https://community.naturephotographers.network/t/when-you-are-awakened-by-thunder-and-lightning-what-else-are-you-supposed-to-do/300/1 upload://ghHlfB97wxgzWJJf68SMYBw8lx9.jpeg
https://community.naturephotographers.network/t/denali-south-view/301/1 upload://STEzJBoR0JJ2tHVt3IQawobgPa.jpeg
https://community.naturephotographers.network/t/cinnamon-flycatcher/302/1 upload://9qYI7HYqQIIN9y74Ji69Ibdg9pT.jpeg
https://community.naturephotographers.network/t/bird-on-a-stick-can-do-the-trick/303/1 upload://ftPi8xXlZ4EUoGs1xYubeupXS0A.jpeg
https://community.naturephotographers.network/t/cinnamon-flycatcher/302/6 upload://zbM5Llsj2juYvRmnkRf9sVC2QCS.jpeg
https://community.naturephotographers.network/t/black-and-white-critique/231/9 upload://1M0Vgi1CxtFsqwwGAd2mW0NSjUH.jpeg
https://community.naturephotographers.network/t/henderson-inlet-sunset/304/1 upload://gEqhYAo2MNYiHdoABhomGavRa6w.jpeg
https://community.naturephotographers.network/t/henderson-inlet-sunset/304/1 upload://mQDFNHXPRM8ZPxI0HoyA4BEUSsb.jpeg
https://community.naturephotographers.network/t/saturated-marigold/305/1 upload://x5TnD0A71b8VwmZ5ygoFfUMNpxY.jpeg
https://community.naturephotographers.network/t/osprey-in-flight/281/8 upload://paznXN4O0jcsjlxNCSUbYrPCNkt.jpeg
https://community.naturephotographers.network/t/snow-streaks-barrier-lake/306/1 upload://87MPrjppo6id4eMzU3jzSxZjSax.jpeg
https://community.naturephotographers.network/t/splt-falls-2/307/1 upload://eySNzoSDr3lYkdjNvPBLKKMd671.jpeg
https://community.naturephotographers.network/t/sunset-at-corona-del-mar-what-would-you-do-to-improve-this-image/309/1 upload://tiR9CfaHhCQomzcZ9CsUU3VWRHe.jpeg
https://community.naturephotographers.network/t/jumping-spider/311/1 upload://j5e5iJvhbZyJq9xc5Ht7BH5Wcgw.jpeg
https://community.naturephotographers.network/t/opulence/312/1 upload://5PKXwqqGBqxgNx7oSKYpGnhzaI4.jpeg
https://community.naturephotographers.network/t/a-spiders-domain/313/1 upload://ezqAYmccDK2RkjIpNG2VPWNsi2m.jpeg
https://community.naturephotographers.network/t/overhang/315/1 upload://Atxc0bWUcjefjkEBlhmjrC7AkmB.jpeg
https://community.naturephotographers.network/t/henderson-inlet-sunset/304/8 upload://ukJWqftYoM2Iwj7OXdDVF4MTCQK.jpeg
https://community.naturephotographers.network/t/san-juan-report/318/1 upload://ddbvjWxLulK2OmlFDa0eoyM37kl.jpeg
https://community.naturephotographers.network/t/san-juan-report/318/1 upload://satVkZahsPPF5u5yC3kQn19H0kX.jpeg
https://community.naturephotographers.network/t/san-juan-report/318/1 upload://bCGdPc2tQWJ5UcTuuqjtjG2FjEj.jpeg
https://community.naturephotographers.network/t/san-juan-report/318/1 upload://g18SJQSdUd082N4fqhE1o1LTwR6.jpeg
https://community.naturephotographers.network/t/san-juan-report/318/1 upload://5MTnMgbi7U6I6ZFxVLuAG0Q8VO0.jpeg
https://community.naturephotographers.network/t/thread-waisted-wasps/325/1 upload://2kG5KG0JaHqFGNjNfmxg0xGluiL.jpeg
https://community.naturephotographers.network/t/passion-flower-repost/326/1 upload://xWh2hiF0lkf6agAIVdQswQi2xfr.jpeg
https://community.naturephotographers.network/t/customize-what-categories-you-see/328/1 upload://zVkW1Uo5aPZtNugp2jAO86KNjNz.png
https://community.naturephotographers.network/t/customize-what-categories-you-see/328/1 upload://j4mBajhv6mv0BPpYowThnE2br36.png
https://community.naturephotographers.network/t/customize-what-categories-you-see/328/1 upload://3HpJqKWoKHl6JpigQ5Hi5EMQLQ5.png
https://community.naturephotographers.network/t/dahlia-version-1/332/1 upload://jEmLR6PaaOgb22rvCiydbaUu9Lv.jpeg
https://community.naturephotographers.network/t/immature-male-ruby-throated-hummingbird-cardinal-flower/333/1 upload://t1Vjz5pLDQXKh5zcDe7nzjX7nuC.jpeg
https://community.naturephotographers.network/t/seaside-goldenrod-acadia/335/1 upload://mxY85CapZqVPjIkkHniduDrqF3Z.jpeg
https://community.naturephotographers.network/t/sedge-warbler/336/1 upload://nfQB8EvaD4NzvcnICSgY4Z6DG1i.jpeg
https://community.naturephotographers.network/t/eastern-towhee-singing/337/1 upload://7RzQfM90nA5wqicTUx1xjxhPFp3.jpeg
https://community.naturephotographers.network/t/id-help-wanted/339/1 upload://9ZjMbWBg43sUtZL7VUmqs0zFnx5.jpeg
https://community.naturephotographers.network/t/id-help-wanted/339/1 upload://oQRNsn0ouQXbsiopYL1ptD9HsDK.jpeg
https://community.naturephotographers.network/t/forest-for-the-trees/340/1 upload://oHpFjgP4twnPMCW4NlSAirsQHye.jpeg
https://community.naturephotographers.network/t/a-bit-of-patience-pays/341/1 upload://ykydyoXA5Q7pHwrmlZSz6kSaERM.jpeg
https://community.naturephotographers.network/t/gear-forum-category-about-edit-request/323/8 upload://ldoOTMjYTDwcsQlpSBTnuV3sfvR.png
https://community.naturephotographers.network/t/sunset-at-corona-del-mar-what-would-you-do-to-improve-this-image/309/10 upload://g34CQZxvdYanzV5FjyOt8x5AsrL.jpeg
https://community.naturephotographers.network/t/tree-on-the-hill-infrared/347/1 upload://k8jSN405wBUeDMtoPpDUYkT98oO.jpeg
https://community.naturephotographers.network/t/wood-stork-fishing/348/1 upload://8NybouXbKzijOe4lz86KRCWlhgG.jpeg
https://community.naturephotographers.network/t/post-tags/294/6 upload://ttBZlukakv3BqtVwFb3gUjGJiPB.png
https://community.naturephotographers.network/t/cropping-problem/350/1 upload://f4MrTvRS50QbTmVCMdE81ijsfls.jpeg
https://community.naturephotographers.network/t/cropping-problem/350/2 upload://hZAKHRQkCxaCKIEeKujnwrBAjQ6.jpeg
https://community.naturephotographers.network/t/cropping-problem/350/3 upload://cDMcFM11OValqU5bZ20wykWjbHr.jpeg
https://community.naturephotographers.network/t/the-secret-life-of-the-moon/353/1 upload://6B3vW2oMEd6GjY2e0ITX0aJrtj7.jpeg
https://community.naturephotographers.network/t/umpqua-vines/355/1 upload://9NPmpMbAa8Dzm8YiEO2rGLJNHlx.jpeg
https://community.naturephotographers.network/t/sri-lanka-leopard/356/1 upload://ijFrXWE9pYxG9UMv5KZYhYswm5U.jpeg
https://community.naturephotographers.network/t/passion-flower-repost/326/4 upload://5Pmvp63NqC8TsMeEJyfzbBKIgyC.jpeg
https://community.naturephotographers.network/t/perfect-water-lily-2x-repost/219/8 upload://hvTRahqokmtoZlppFc4wCxGn5Y3.jpeg
https://community.naturephotographers.network/t/dramatic-mount-sneffels/359/1 upload://6TjtX7Db3V2FlJS0QV08LYwILso.jpeg
https://community.naturephotographers.network/t/big-city-nights/360/1 upload://mwE7QKj8HOPJIMmDireoFagHr1U.jpeg
https://community.naturephotographers.network/t/american-white-pelican/362/1 upload://bJEPjdwRLCW4bEFrlismduwkKnQ.jpeg
https://community.naturephotographers.network/t/umpqua-vines/355/3 upload://9PSDqfuo6RxUpWuBXpjjB0owi39.jpeg
https://community.naturephotographers.network/t/umpqua-vines/355/4 upload://poVTOaJUqS1CQnf9k73NiEn2pna.jpeg
https://community.naturephotographers.network/t/through-the-smoke/366/1 upload://xlaRk9J54SNKMopebPM76a2oG6m.jpeg
https://community.naturephotographers.network/t/the-edge-of-mono/367/1 upload://AsOpk3WTPgAZOrKeJ1w1Aqalna5.jpeg
https://community.naturephotographers.network/t/casa-grande-of-big-bend/372/1 upload://yR8K4hIYoO19oDFaExBoHf6SiB6.jpeg
https://community.naturephotographers.network/t/black-and-white-critique/231/18 upload://iUakUD9S9JpK5cJZvWmHnFxDDhr.png
https://community.naturephotographers.network/t/rainbow-harbor-lighthouse-in-long-beach-ca/377/1 upload://lOch55eSIBl3xp16SJRUFeILOYh.jpeg
https://community.naturephotographers.network/t/workshop-tour-listings-signatures/375/7 upload://5PfMAvqsTXCoMQitAs0lZQEIM7S.png
https://community.naturephotographers.network/t/spider/378/1 upload://bFcl2uqIool0kDmntNp8esC16YH.jpeg
https://community.naturephotographers.network/t/cinnamon-flycatcher/380/1 upload://XyE5enOfdfxeDSC6NbAWwxf7Jm.jpeg
https://community.naturephotographers.network/t/dahlia-abstract/381/1 upload://3SuPsb32SftCQkJznXaMyBW3WEF.jpeg
https://community.naturephotographers.network/t/matanuska-morraine/382/1 upload://aA1qKlSarhUgxpmk7cU3mKsVWvE.jpeg
https://community.naturephotographers.network/t/split-falls-2-black-white/383/1 upload://n0dSqlHldcoDZ6QbivV4fQi1ZH5.jpeg
https://community.naturephotographers.network/t/macro-size-definition/364/7 upload://vVOZZXlz5f7WDSm8eYm5UF0fOWr.png
https://community.naturephotographers.network/t/macro-size-definition/364/7 upload://c5CCP1XQfjK4ASMzyHUQacpSyUf.png
https://community.naturephotographers.network/t/woodland-giraffe/384/1 upload://v99GVB2fCLtd5YeLQ6Tt5aXDUY4.jpeg
https://community.naturephotographers.network/t/sparkle/387/1 upload://g80uPG9FbAL44F1F7UHkGLFIz8J.jpeg
https://community.naturephotographers.network/t/spotted-towhee/388/1 upload://tHR5EdgWZgddc5R670PUEr86hYr.jpeg
https://community.naturephotographers.network/t/last-hummingbird-of-the-season-for-me/389/1 upload://qduqjHbRO8X88Kc0YiAqWHBJaQb.jpeg
https://community.naturephotographers.network/t/traversing-thompson-pass/390/1 upload://z0IrmKXaEtfT7meM9kB6gtudsxG.jpeg
https://community.naturephotographers.network/t/mallard-tranquility/391/1 upload://l6tx2QfXrXz6A8gj6GZ8BQ4IWj8.jpeg
https://community.naturephotographers.network/t/first-stage-fireweed/392/1 upload://dX0QHa3ysU1TEArYvHWrDRfvLgI.jpeg
https://community.naturephotographers.network/t/steadfast-fishing-pays-off/394/1 upload://2eChBxAtuZV3H4u01cWdPMqmBA6.jpeg
https://community.naturephotographers.network/t/spider/378/3 upload://ql7iaQHdaFivIqCrWdaOQcA3Uxn.jpeg
https://community.naturephotographers.network/t/great-blue-heron-in-the-tide-flats/396/1 upload://pmutQVnVZWhd4rAdUr4vE1aQDkv.jpeg
https://community.naturephotographers.network/t/the-art-of-dogwood/398/1 upload://i3iC1Fi22BssU2uZUQdoMMe8dgn.jpeg
https://community.naturephotographers.network/t/foggy-morning/402/1 upload://boYDgVOOe9ni6Qg3XwDEtOE0YwX.jpeg
https://community.naturephotographers.network/t/signs-of-fall/404/1 upload://sdEYXx3xzkPkCaJ6vxUH7wp7hfP.jpeg
https://community.naturephotographers.network/t/black-capped-chickadee-w-repost/408/1 upload://fz5MALx0idgHMkF0sh24b1SF0G8.jpeg
https://community.naturephotographers.network/t/juvenile-reddish-egret/409/1 upload://8y6j7A4zGnGCWwWWiCUAKhSq0OW.jpeg
https://community.naturephotographers.network/t/heck-why-not/412/1 upload://2zgMUjZ8ZpPicMNps7Z3NCEJc3F.jpeg
https://community.naturephotographers.network/t/heck-why-not/412/5 upload://smhHu0NACzD0iutbyugI3RdiWnL.jpeg
https://community.naturephotographers.network/t/havasuw-hagjahgeevma/415/1 upload://4ijZZJ5y2d7A83u1DlGPRIuDMQK.jpeg
https://community.naturephotographers.network/t/soggy-woods/416/1 upload://7CC9FtsR2tv2YsOIzHS55Thja7h.jpeg
https://community.naturephotographers.network/t/snowy-egret-on-solana-beach-ca/422/1 upload://exyUPEjFUNeHHU0Pf6upCchplj5.jpeg
https://community.naturephotographers.network/t/capybara-sunrise/423/1 upload://qjp4ZbO2ZZbqmmpz1OQUp2zfTgF.jpeg
https://community.naturephotographers.network/t/purple-branches/424/1 upload://aJYRWIIy4y2Rm87yJQPgxkIEQPX.jpeg
https://community.naturephotographers.network/t/search-questions-issues/425/1 upload://1QDvK1rmRhg3ZhsXYrytBvLu9i0.png
https://community.naturephotographers.network/t/pea-blue/426/1 upload://o0rUzBVPwS280ETigCfPXJQl55X.jpeg
https://community.naturephotographers.network/t/search-questions-issues/425/2 upload://gNFXgoRemJf4mtHC0ciNz1MsV7.png
https://community.naturephotographers.network/t/macgillivrays-warbler-rp/431/1 upload://mw7ZyviUrhbEwAIUyLdC38Oim7j.jpeg
https://community.naturephotographers.network/t/west-of-the-pecos-judge-roy-bean-land/433/1 upload://o4Gwi8swa76bp84XK6if8Mq9nKk.jpeg
https://community.naturephotographers.network/t/the-moody-river-bottoms/434/1 upload://3ilSaiE5o0gJGnjjtoiVcG1E949.jpeg
https://community.naturephotographers.network/t/a-glacier-in-the-clouds/435/1 upload://5kD0FR6f0wZDHa5GJZ557HXfwG1.jpeg
https://community.naturephotographers.network/t/aspen-turning-at-sand-dunes/436/1 upload://y3e5kp4pTe4870XCks8ON535fxz.jpeg
https://community.naturephotographers.network/t/in-the-pink/437/1 upload://kudoYipGxdA3xLjcOqFHfkxPrvf.jpeg
https://community.naturephotographers.network/t/macgillivrays-warbler-rp/431/6 upload://tokDpR1h0lbWOowLHpdw16BYK5G.jpeg
https://community.naturephotographers.network/t/orange-crowned-warbler/439/1 upload://zg2sBmGkRZXsaapFCnfBYlGo3ht.jpeg
https://community.naturephotographers.network/t/symbiosis/441/1 upload://wDm6GlRLH5Jm5D64SXam8VIfxkX.jpeg
https://community.naturephotographers.network/t/whinchat/443/1 upload://w4gHDFAhVkbx8deUxGbqKOuqB7z.jpeg
https://community.naturephotographers.network/t/how-to-submit-workshops/444/1 upload://5Urx3JmU2xw3L7QqmxhPF55wqFu.png
https://community.naturephotographers.network/t/how-to-submit-workshops/444/1 upload://28BA5fpQnawo3gBJVColCNHK25N.png
https://community.naturephotographers.network/t/how-to-submit-workshops/444/1 upload://xEmmwEWbsW2JAL9ULjywon53BPW.png
https://community.naturephotographers.network/t/black-capped-chickadee-w-repost/408/6 upload://rRvprKQtQgq1YkVPsLegLRLIPok.jpeg
https://community.naturephotographers.network/t/maine-sunrise/445/1 upload://hkUWLfSrqR0Eki3SwmWbvxYfWVs.jpeg
https://community.naturephotographers.network/t/pink-cosmos-with-a-hitchhiker/446/1 upload://stKC31uJueiNOi0yg0isbT6UedK.jpeg
https://community.naturephotographers.network/t/setting-moon-jtnp/448/1 upload://oMRGifKztIkzH6AsnEqmx051QVl.jpeg
https://community.naturephotographers.network/t/a-nice-wild-flower/449/1 upload://r6BMJEMfPKogcOD3UKVG4a8VEFH.jpeg
https://community.naturephotographers.network/t/eastern-bluebird/450/1 upload://7nqlYbpJR2FJCRxCdzEP1c0OZXR.jpeg
https://community.naturephotographers.network/t/milky-way-from-little-hunters-beach/452/1 upload://34Q8jWjH3QD78B6RjyZ6YvMJLvF.jpeg
https://community.naturephotographers.network/t/red-dahlia/453/1 upload://8Zz1m4GFhVPdBr1f1wbR8N4XanV.jpeg
https://community.naturephotographers.network/t/red-dahlia/453/2 upload://wGSq7mUu3mlp5Ql8Y1MAvq3x10M.jpeg
https://community.naturephotographers.network/t/another-attempt/454/1 upload://hDv4QNkKjS97ogB7KTtGUQhfp5G.jpeg
https://community.naturephotographers.network/t/field-of-fern/455/1 upload://9jdpRiDI7Oi3u7nvkY5gTGzTY3d.jpeg
https://community.naturephotographers.network/t/female-wood-duck-and-offspring/456/1 upload://4Z3HySsmRDaEmbV5fkCdary6YRb.jpeg
https://community.naturephotographers.network/t/casa-grande-in-the-clouds/457/1 upload://rjCDMz8OW3RhtYHBFhAguVPtxJa.jpeg
https://community.naturephotographers.network/t/american-white-pelican/458/1 upload://8lMSiLF79Q8z0NJuYvWcL2UQ9LL.jpeg
https://community.naturephotographers.network/t/macgillivrays-warbler/461/1 upload://bxli4Obguq466lyeECIMpzd1nyA.jpeg
https://community.naturephotographers.network/t/raincatcher-just-a-photo-of-some-wet-autumn-grasses/462/1 upload://nkWwssWIf0Pu2BSuBea3Jr594Hn.jpeg
https://community.naturephotographers.network/t/new-website-thoughts/460/2 upload://pkY1y5rnDRwd85QUb7Z170W6rc4.png
https://community.naturephotographers.network/t/new-website-thoughts/460/2 upload://Afvb4PxVBQsMREi4EFg8pONiOUP.png
https://community.naturephotographers.network/t/another-attempt/454/4 upload://kiKQzN1dbCkfWkOStOqWNm9D6aX.jpeg
https://community.naturephotographers.network/t/red-dahlia/465/1 upload://xAJhbrmt06JfZpmf6ofzx6cnjOP.jpeg
https://community.naturephotographers.network/t/in-the-fog/466/1 upload://mzR7ZjMNptj8PoU6pNfwNGPLov7.jpeg
https://community.naturephotographers.network/t/ad1ef70ad05826c2944514e28ffb62fa/154/6 upload://bPY1u7TVsmIdvllPyis1ZaXrHxo.jpeg
https://community.naturephotographers.network/t/id-help-please/472/1 upload://lAVsVNPjpBio1Y2nuHhBjYPUmfP.jpeg
https://community.naturephotographers.network/t/topic-excerpts-in-galleries-critique/473/1 upload://pWi3Cgx6nu3sF0hqzbwTxLSWfzA.png
https://community.naturephotographers.network/t/topic-excerpts-in-galleries-critique/473/1 upload://noxIvJ0NxcdJpwp8kAimyCAv0Tt.jpeg
https://community.naturephotographers.network/t/belted-kingfisher/474/1 upload://ucHEQSVdAKfP6CTyZ4gyIDFXdsV.jpeg
https://community.naturephotographers.network/t/kawela-bay-sunset-working-on-finding-the-best-compositions/477/1 upload://ucF1gH1ouda2OIkZIrVKgDJBP6O.jpeg
https://community.naturephotographers.network/t/editors-picks/476/2 upload://bdZzReOzbdr5tkWm9j4NTdgqB4H.png
https://community.naturephotographers.network/t/black-crowned-night-heron/480/1 upload://bzDaS4clhHflL96UzMxb89NCl7A.jpeg
https://community.naturephotographers.network/t/black-hawk-fishing/481/1 upload://dqxdrAOhML6YlAtb3BT3Jv3VXYR.jpeg
https://community.naturephotographers.network/t/tucked-away-falls-in-sw-colorado/483/1 upload://dbCqcjkM9RiYyBqe21zKF7KtmnW.jpeg
https://community.naturephotographers.network/t/juvenile-reddish-egret/409/11 upload://lmjzHLwIaF4FbfQAptieICsnp66.jpeg
https://community.naturephotographers.network/t/female-wood-duck-and-offspring/456/4 upload://arR1tqLOlSnHvX484rf36ayXrH6.jpeg
https://community.naturephotographers.network/t/beaver-at-turtle-bay-museum-in-redding-ca/486/1 upload://o3uCiYHHoBOW5ezMQjKegKuJTnN.jpeg
https://community.naturephotographers.network/t/tucked-away-falls-in-sw-colorado/483/4 upload://OQQWgbeRXJ9Rp0diqXFseOfgPi.jpeg
https://community.naturephotographers.network/t/chestnut-crowned-antpitta/487/1 upload://pldCsykTayqq7GbF06AYLI9flSJ.jpeg
https://community.naturephotographers.network/t/chestnut-crowned-antpitta/487/2 upload://rd3tI9SALKsErQuvRiaGZZIZcpf.jpeg
https://community.naturephotographers.network/t/weekly-picks-landscape/478/2 upload://EpKwAba5QtWGaNSBVpeNA6WDmb.jpeg
https://community.naturephotographers.network/t/alpine-bliss/490/1 upload://9AkCovNalfV1EIHv7USnrGhyJWw.jpeg
https://community.naturephotographers.network/t/brown-creeper/491/1 upload://pYrg6BLQzZSLMKngfLdlk5iP5Ie.jpeg
https://community.naturephotographers.network/t/black-capped-chickadee/492/1 upload://1aVcyKeymQhrkRvq6UyR5HCVt4G.jpeg
https://community.naturephotographers.network/t/topic-excerpts-in-galleries-critique/473/11 upload://rRisHgZfDska66rsVgnrH5WEUd4.png
https://community.naturephotographers.network/t/the-chamber-of-wonders/495/1 upload://8iJRIq0SO6cwthsi0QsXSm8KJYr.jpeg
https://community.naturephotographers.network/t/gardenia/500/1 upload://4EIhZolcX7bakQjEplsh5U2PDEw.jpeg
https://community.naturephotographers.network/t/gardenia/500/2 upload://uwg6T6oOw0Immm4QpyqpD6f6IxY.jpeg
https://community.naturephotographers.network/t/long-tailed-skipper/502/1 upload://dphT6sG21ah1b15TfZ0m2j4wpck.jpeg
https://community.naturephotographers.network/t/clapper-rail/505/1 upload://k9Jm7IRXZRKZoWI4k3aifzAFaus.jpeg
https://community.naturephotographers.network/t/foggy-creek/506/1 upload://PncO3yvNTvjDzqmftMwdpryloB.jpeg
https://community.naturephotographers.network/t/testing-notifications/497/6 upload://mHO8XnzloMNA1CLofpmkPnfjNPp.jpeg
https://community.naturephotographers.network/t/membership-renewal/504/2 upload://pHcqslYVH5tt0NAAtCMsyDdmt9S.png
https://community.naturephotographers.network/t/tree-frog-with-escape-plan/508/1 upload://y1zH8o0TTZDoOyHypByVRD6tRoH.jpeg
https://community.naturephotographers.network/t/rocky-mountain-sunset-at-rock-cut/512/1 upload://cPyqODegzoqRvaS8KsVK3nMFPLG.jpeg
https://community.naturephotographers.network/t/baja-sunrise/514/1 upload://76CEWVC9Ht7GmfZ7FpIuNsYt3GV.jpeg
https://community.naturephotographers.network/t/feeding-avocet-with-repost/515/1 upload://aU7vfwxUdwqfdNgBbajLBC5UuPo.jpeg
https://community.naturephotographers.network/t/male-pyrrholuxia/516/1 upload://8v8KNfTobpg77Y18vEQv0iexYz9.jpeg
https://community.naturephotographers.network/t/roseate-spoonbills/518/1 upload://r9n1HOBvYnfoCQg9Gj1BHAIO2SN.jpeg
https://community.naturephotographers.network/t/barred-owl-in-flight/519/1 upload://16I0DP2iKNp7dda4kAoFbQlAQEn.jpeg
https://community.naturephotographers.network/t/roseate-spoonbills/518/3 upload://h5LKhIwNiemh3MpgGphrlUo8Jox.jpeg
https://community.naturephotographers.network/t/roseate-spoonbills/518/4 upload://g3tzWJiuoGjFxGzh2p32kBR49js.jpeg
https://community.naturephotographers.network/t/feeding-avocet-with-repost/515/6 upload://k8XN9fPE8loxN2hMmqtrQIcs7vu.jpeg
https://community.naturephotographers.network/t/whinchat-on-barbed-wire/522/1 upload://ioulTDBAaqbRIimoxUveP23cZZk.jpeg
https://community.naturephotographers.network/t/asian-elephants/523/1 upload://cUYn92IPo3fV5495pUuoghist33.jpeg

and here are the posts that are still broken

Summary
https://community.naturephotographers.network/t/feeding-avocet-with-repost/515
https://community.naturephotographers.network/t/clapper-rail/505
https://community.naturephotographers.network/t/black-capped-chickadee/492
https://community.naturephotographers.network/t/gardenia/500
https://community.naturephotographers.network/t/gardenia/500
https://community.naturephotographers.network/t/purple-branches/424
https://community.naturephotographers.network/t/purple-branches/424
https://community.naturephotographers.network/t/whinchat-on-barbed-wire/522
https://community.naturephotographers.network/t/roseate-spoonbills/518
https://community.naturephotographers.network/t/barred-owl-in-flight/519
https://community.naturephotographers.network/t/foggy-creek/506
https://community.naturephotographers.network/t/a-glacier-in-the-clouds/435
https://community.naturephotographers.network/t/alpine-bliss/490
https://community.naturephotographers.network/t/baja-sunrise/514
https://community.naturephotographers.network/t/asian-elephants/523
https://community.naturephotographers.network/t/belted-kingfisher/474
https://community.naturephotographers.network/t/brown-creeper/491
https://community.naturephotographers.network/t/chestnut-crowned-antpitta/487
https://community.naturephotographers.network/t/male-pyrrholuxia/516
https://community.naturephotographers.network/t/long-tailed-skipper/502
https://community.naturephotographers.network/t/black-crowned-night-heron/480
https://community.naturephotographers.network/t/another-attempt/454
https://community.naturephotographers.network/t/soggy-woods/416
https://community.naturephotographers.network/t/snowy-egret-on-solana-beach-ca/422
https://community.naturephotographers.network/t/pea-blue/426
https://community.naturephotographers.network/t/cinnamon-flycatcher/380

@sam I don’t see any patterns, the oldest is from 7 days ago, newest is 2 days with everything in between

Also, there are a lot more thumbnails that are broken than this, but I assume once the full size images are back a rebake would bring these back.

I don’t know if this means anything, but I also have a huge list of files in the root of my bucket, I think it happened after the rebake, one of the files is called 2018-09-09-21-17-50-E2157AA437860E3B and contains this

a82e4dda19c7719237318dfc5331da22069c95d2faf5db8b9cc3f573566bb783 nature-photographers-network [09/Sep/2018:20:54:45 +0000] 94.46.151.17 - AADC401E34175DE1 REST.GET.OBJECT optimized/1X/23c8554a837491acd5b2423a80466d7ef6d61618_1_602x500.jpeg "GET /optimized/1X/23c8554a837491acd5b2423a80466d7ef6d61618_1_602x500.jpeg HTTP/1.0" 200 - 236199 236199 60 58 "https://community.naturephotographers.network/t/the-look/297" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15" -


(Alan Tan) #28

Oops you need to do a rebuild first before running the commands above.


(David Kingham) #29

Thanks, this seems to be running properly now, I am seeing results that look like this:

https://community.naturephotographers.network/t/grand-teton-sunrise/264/1 upload://qgMUVJSZtVbkxjQdoXqiM6V6bvu.jpeg
Unknown markers for jpegoptim: none
Unknown markers for jpegoptim: none

It’s running quite slow and ruby is eating up 60% of the processor, but I assume that is expected.


(David Kingham) #30

It worked! You two are life savers! Now, is it safe to assume that this also fixed the original problem that caused these images to be tombstoned?


(David Kingham) #31

Also, is it safe to delete these files from the root of my bucket?


(Sam Saffron) #32

Yes, the particular posts should be good now.

Also, long term this particular fail mode should not happen cause we associate on save, not in a job.

Regarding nuking from tombstone, don’t worry about it, there is a policy that will take care of it


(Jeff Atwood) #33

Do post saves now get extra slow as a result though? How much extra time is added to post save as a result? If “not much” then why was this put in an out of band task originally?


(Sam Saffron) #34

No noticeable difference, we look up uploads via sha (new optimisation), the document is already loaded into nokogiri at this point the extra selector is negligible


(David Kingham) #35

I have bad news, the problem is still happening. I have 10 or 15 posts this morning with no images, they are in the tombstone folder again.


(Alan Tan) #36

There is a bug in the latest release that affected uploads with id less than 16,000.

Fixed in

I’m currently working on getting a migration out which will fix the posts with missing links to the uploads automatically.


(David Kingham) #37

Still having the problem and the rake is not working anymore, can anyone help?


(Jeff Atwood) #38

Did you get absolute latest code and rebuild?


(David Kingham) #39

Yes, I updated this morning, I’m on v2.2.0.beta1 +26 and have many new tombstoned images today


(Sam Saffron) #40

@tgxworld will be around in 2-3 hours and will have a look.


(Clay Heaton) #42

FYI, we’re having a similar issue, following a rebuild, where a bunch of older images no longer are appearing in posts, in spite of no other changes. We are not using S3. I rebuilt and rebaked all posts but they’re still not appearing.

I don’t see this rake task when I run rake --tasks :

rake uploads:list_posts_with_broken_images