I’m not particularly familiar with Google Custom Search, but my understanding is that it would only be able to search public topics. Assuming that’s true, that means your users won’t be able to search their PMs, topics in non-public categories, tags/users/groups (we block indexing for those pages), etc. If there’s a way to give Google admin access to your site, that may solve some issues, but opens up other (privacy) issues.
We do support Algolia, see Add Algolia search to your Discourse.