I indeed cannot reproduce with a tag alone.
I made a small mistake in my OP:
Tag.find_by_name is case insensitive (good) and Tag.find_by(name: ) is case sensitive (bad).
This is because David already fixed it for tags 7 years ago by overriding the find_by_name method.
For TagGroup, that fix was never applied.