It seems that as posts are processed for attachments, there are a number of sidekiq jobs that get spun up to deal with some other processing on these posts. As a result, even a single process working on attachment imports manages to slowly drive the server to over 40 load average even with 8 cores. (I increased the number of sidekiq workers to handle the load.)
I might be able to stop the unicorn service until the import is complete, but that is just shifting the load to a later time. It seems like the processing has to be done one way or the other.