Learning the hard way: AWS EC2 /tmp/ folder is always loaded into memory

We fixed a memory leak in our codebase

It's rare for companies to talk about backend changes.. but this one is super important

We kept having issues where our customers' tasks would randomly die. We had no idea what was going on

1. We did running profiles of our EC2 instances and noticed that memory was spiking right before a worker would die... how interesting
2. We also knew one small quirk about our product: we would save all screenshots / llm calls / downloaded files to the /tmp/ directory as a users' task ran
3. We checked some AWS documentation and found out... that all data inside the /tmp directory is automatically loaded into memory (https://www.reddit.com/r/linuxquestions/comments/xowiye/a_stupid_question_but_is_tmp_actually_located_on/)

So our servers would write these very useful artifacts until we ran out of memory and blew up

We quickly started writing to another directory and our memory issues vanished.

Fortunately for our users, this means we can re-enable "fast" workflow runs! You should see your workflows starting instantly in the UI again