If there’s a way to remove an unintended link, it would be really cool to have some way to add a URL back again (e.g. if you accidentally clicked the x button)
How about when there’s no link, there’s a little 
button which displays a text field when clicked?