This would be a great improvement. My forum currently has some kind of DNS error, but if your service workers are working then the front page will work even though you can’t really do anything except maybe view some pages that were still in cache. And if you click one that wasn’t, then the forum goes spinning, and keeps spinning if you try to go back to the front page. It would be very confusing if you didn’t know the cause.
A text banner explaining the site is offline would be better than an icon you might miss. TBH, this offline mode may be useful in limited situations, but right now I’d prefer it not to load at all.