To modify the composer or as you called it the new topic dialog, you’d need to write a plugin. You can see all the plugin guides here: #plugin-guides, but I believe the specific one you want to look at is Beginner’s Guide to Creating Discourse Plugins Part 2: Plugin Outlets - howto / developers - Discourse Meta.
That said, it would help you if you went through all of them from the first one (part 1) to the last one (part 7).