Requiring an additional gem in your plugin for testing

To require an additional gem in your plugin you should add a gem "gem-name" ... to your plugin.rb. But what if I need an additional gem during testing?

I only need it during test execution, so I do not want to add it to my plugin.rb.

I can’t do the same gem "gem-name" ... trick in the spec file or plugin_helper.rb, as that produces the error

Gem::LoadError:
  rspec-collection_matchers is not part of the bundle. Add it to your Gemfile.

I want to add the rspec-collection_matchers gem. Sure I could work around it and produce less nice expectations. I am wondering if there is a construction which will allow me to include additional gems during testing.