This appears to be a significant security downgrade. MD5 is unsuitable for password hashing. Not only would this change affect a critical part of Discourse, but you would also need to maintain it for all future releases.
Perhaps we can suggest a better approach if you could explain what you’re trying to achieve.
Regarding customizing Discourse, please take a look at Developing Discourse Plugins - Part 1 - Create a basic plugin. The plugin system allows you to safely extend core features.