Yes, it is generally safe to upgrade multiple major releases at once. Rails migrations handle that for you.
However, it is always a good idea to take a full backup before major changes like this, so you have the ability to restore to a known working state in the unlikely event something goes very wrong.