The limit(10) should do that already 
This kind of thing should do it: (but not tested)
Topic.where("category_id in (?)", [cat_id])
.where("closed = ?", false)
.joins("LEFT JOIN topic_custom_fields import_tcf ON import_tcf.topic_id = topic.id AND topic_custom_fields.name = 'importedSortDate'")
.order("TIMESTAMP import_tcf.value")
.order("created_at desc")
.limit(10)
The problem is, asking postgres to cast the date string into a timestamp will be very inefficient. It will have to do the conversion for every single topic first, then pick the first 10.
There are a couple of options I can think of here - you could store the dates as “seconds since epoch”, rather than ISO8601. That will make them much easier for postgres to sort. Or you might be able to use a database migration to add an index to the topic_custom_fields table for TIMESTAMP value.