You can assign a topic without writing a note:

Or you can assign it with a note:
You can edit the note later, and it pops up as if you’re editing a whisper. It doesn’t let you save an empty whisper, therefore, effectively, you cannot remove the note:
I understand that for a normal whisper this makes sense: if you want to empty a whisper, just delete it! But in this case it does make sense to want to empty the assignment note. After all, “empty assignment note” is a valid state for an assignment entry (as seen in the 1st screenshot). But it’s not a state that you can return to via editing.
Deleting the assignment entry is not a good workaround. You’re losing track of the assignment action. The topic remains assigned even if this entry is deleted.

