קריסה לאחר שדרוג ל-2.7beta7

תמיכה
16

@pfaffman Thank you! The site seems to now be functioning completely normally, but I would not turn down a quick look-over / sanity check at your convenience, if you are willing. I’ve noted your commercial contact info for any future emergencies regardless. :slight_smile:

Here is a retro, in case it is useful for anyone else

SUMMARY

After a (successful) upgrade, an incompatible unofficial theme component plugin rendered the site UI broken. This was not known at the time, so the backup restore process was initiated, but ran in to problems due to a configuration that was less than ideal.

DETAILS

  1. An upgrade to 2.7beta7 was initiated, and completed successfully.

  2. However, after the upgrade, the site UI was severely compromised: post bodies were entirely missing, top navigation (including user / login) was entirely missing, JS console reported errors

    1. The reason for this turned out to be an incompatible third-party them component for copy-pasting codeblocks, but this was not known at the time.

    2. Also not known at the time as the possibility of entering safe-mode Had that been known, the remaining problems could have been avoided

  3. Access was gained to \admin by direct navigation, and an attempted rollback was initiated. This immediately logged out the user, and it did not appear that there was any way to log back in with the broken UI

  4. Logged in to DO Droplet to initiate a manual restore with the latest backup tarball from S3.

  5. Many restore attempts failed near the final step, after uploading S3 assets, due to a single post having some error

    1. This is evidently because restores that try to re-upload assets to S3 can be flaky (several reports of this on Meta)
    2. However, did not need to be backing up upload assets, since they are already stored on S3!

  6. During one of the many restore attempts, the site was also wiped in order to try from a “clean slate” so the rollbacks after the restore failures now rolled back to an empty site.

  7. Eventually, in a desperate gamble, I ran the restore, and hard-aborted the script during the S3 upload (right before the failure and rollback)

  8. The site came back online, but exhibited the previous UI problems. However now safe-mode was known and used and the site functioned normally with plugins and themes disabled

  9. All unofficial plugins and themes were removed (include the “copy-paste” component, after which the site functioned normally

  10. Verified that previously uploaded images were still loading, and from the S3 CDN

I suspect the final upload and “rebake” was not needed since the assets were already still on S3, and the posts did not need to be updated to use new URLs.

I am not certain what if any remaining restore steps were missed after the script was aborted, but so far no-one has encountered any issues with the site in its current state.

LESSONS / ACTIONS

  • Start with safe-mode for diagnosing problems in the future
  • Turn off the setting to include uploads in backups (will suggest to Discourse to warn about this situation)
  • Remove all unofficial plugins and theme components
  • Suggest adding Turn on new built-in codeblock copy feature
  • Enable Digital Ocean weekly image backups as a backstop for recovery