Embeds are cached, yes. I’ve not dug into the code recently to determine what happens if an embed is removed once cached and the original is removed.
On platforms such as WordPress the default is for social media embeds to be preserved. The embed text loses its styling, but the wording is preserved. It makes sense that the content sticks around for context, if it didn’t people would stop embedding and start screenshotting.
The tweet above suggests that once an embed URL becomes invalid due to deletion that it falls back to the URL. I’m going to experiment with this on one of my instances.