Raw_email post field stores images as Base64 strings

AI-generated summary

The raw_email field in Discourse stores images as Base64 strings, which can result in large file sizes. Canapin discovered this while searching for CDN URLs in their forum’s posts and found that the downloaded data was nearly 100 MB in size. supermathie explained that this is because images are sent over email in Base64 format and storing the raw_email field preserves this format. While this may take up disk space, it is unlikely to impact performance unless the field is actively referenced. supermathie suggested that stripping large Base64 content from old raw_emails could be considered, but noted that this would mean no longer storing the original raw_email.

Hi,

Since I’m thinking of changing my CDN (Stackpath) and I know that Discourse dynamically sets the CDN URLs in links, I did nonetheless a search for the string “stackpath” in my forum’s posts to see how many occurrences there would be.

It returned between 300 and 400 posts.

I downloaded the data, and I was surprised that it’s almost a 100 MB file!

Apparently, it’s because images in posts sent by email seem to be present as base64 strings in the raw_email field.

So… As I see it, it doesn’t seem to be an issue really, but I’d like to have some clarifications about this.

Why are images stored like this in the raw_email field? Is there no better way to do so?
Can it impact performances in any way? What about forums where a lot of people reply by email and with images involved? Could we consider that it takes too much space for not much info?

Any explanation or additional information about that is welcome. :slight_smile:

This is going to sound a bit reductive, but that’s how images are sent over email. And since we’re storing the raw_email… that’s what we store.

Think of it like a 100MB Inbox. Which is a drop in the bucket for most sites.

You can find the storage size with:

select sum(pg_column_size(raw_email)) / 1048576 as raw_email_MB from posts;

We would no longer be storing the raw_email in that case… I suppose there’s an argument to made for stripping large base64 content from raw_emails over a certain age…

Not likely; it’ll sit on disk but should not be referenced unless you look at it.

:thinking:

Thank you for the detailed answer :slight_smile: