I think it’s the other way around. If there is a user id, you at least had the opportunity to ask for consent. I think the IP addresses without user ID are more problematic, since they belong to unknown people who were just passing by, and never gave permission to store any personal data at all.
To answer your question, I don’t think it will cause issues. Don’t forget to clean up your access logs as well.