Subfolder is better in my opinion (if your set up is a main site with forum section) from a purely SEO perspective. Especially if it’s an old established site, not so much if it is an entirely new site. If I were setting that configuration up though, I’d probably put it on a subdomain anyway as it seems cleaner. Makes it easy to put the forum on a separate server if wanted as well.
Each subdomain to some extent, is treated as a separate site. Some trust is passed between them it seems. However, it’s not as much as a subfolder which doesn’t require any transfer of authority.
For example, it wouldn’t make sense for the authority of wordpress, wix, blogspot, tumblr etc. to transfer to sites made by users who haven’t attached a custom domain to their site. i.e. https://ponygeddon.wordpress.com/. The search engines seemingly can somewhat differentiate between these uses it seems. But I honestly think we often overestimate how intelligent search algos are.
That said, it really seems like Google is having a crusade against UGC content (except Youtube). I rarely get any UGC results for my searches unless I specifically append a site name. Even to the extent of needing to apply ‘inurl:example.com’ and still getting minimal results. Lots of friends have started needing to do this as well. Maybe we’re just going crazy though. 
So if that really is true, then subdomain might be better for SEO too. To allow your main site subdomain to not be directly considered UGC.
Some other possible differences:
- SE updates might affect your subdomains differently - hedged bets between updates.
- Some or most search engines are limiting the amount of search results from the same site/subdomain. So having separate subdomains may give you more opportunity to rank multiple pages for the same queries.