What is the difference between custom invite mailer, invite forum mailer and custom invite forum mailer?

Product Support
2

invite mailer

This template is used when inviting a user to a topic.

This template is used when inviting a user to a topic with custom message.

This template is used when inviting a user to a forum.

This template is used when inviting a user to a forum with custom message.

In this case you should edit invite forum mailer.