A bug was reported where, after creating a bookmark and dismissing the dialog with esc, the delete icon for the bookmark does not appear upon reopening the dialog. schleifer described the issue, noting that the icon only appears if the dialog is interacted with via a click. codinghorror directed the report to @martin, who acknowledged the issue and fixed it in a pull request (#10088). The fix adds the ability to save a bookmark by pressing enter in the focused text box, though esc still does not save the bookmark.
When a bookmark is created but the dialog is dismissed with the keyboard, you cannot delete the bookmark by re-visiting the dialog again. The icon is simply not present.
Pressing Escape will not save the bookmark @schleifer but I added back in the functionality to save when you press enter inside the focused text box, so if you press f enter it will save a topic bookmark.