How do you customize hot topic score algorithm with a plugin?

Easiest but might be limited depending on what you want to do.

Doable but more complex. Have a look on how TopicQuery implements lists

and specifically

  def list_hot
    create_list(:hot, unordered: true, prioritize_pinned: true) do |topics|
      topics = remove_muted(topics, user, options)
      topics.joins("JOIN topic_hot_scores on topics.id = topic_hot_scores.topic_id").order(
        "topic_hot_scores.score DESC",
      )
    end
  end

You can add an extra one relatively easily although there are quite some moving parts you need to take into account. I did that in my ‘homepage-filter’ plugin and that might be a good starting point.