Yes, it's a privacy leak. A tiny one, but it does exist, and can be exploited.
If this concerns you, salt your email with random stuff. For instance google mail allows you to add a plus and anything after to your email and before the @.
So if your email is firstname.lastname@example.org then emails to email@example.com goes to your inbox. So does firstname.lastname@example.org, etc.
This allows you to slightly change your email address for every gravitar site you use, defeating the tiny privacy hole that exists in gravitar usage.