Varv

S3 multipart uploads, explained simply

Every tool that moves big files to S3 uses multipart uploads under the hood. Knowing roughly how they work explains a lot: why resume is possible at all, why some uploads fail at 99 percent, and why a broken upload can quietly cost money for months.

Why parts exist

A single upload request to Amazon S3 tops out at 5 GB, and one giant request is fragile anyway: one network hiccup at 99 percent and everything starts over. So S3 lets a client split a file into parts, upload each part as its own request, and ask S3 to assemble the object at the end. Objects up to 5 TB are built this way.

What resume actually means

Because every part stands alone, a failed part costs you that part, not the whole file. A client that remembers which parts were confirmed can continue after a crash, a reboot or a dead connection by re-sending only what is missing. That memory has to survive the client itself crashing, which is why Varv journals every part to disk as it completes; the transfer queue picks up mid-file even after a force quit.

The hidden cost of broken uploads

Here is the part most people learn from a bill. If an upload dies and is never finished or aborted, the parts already uploaded stay in the bucket, invisible in normal listings, and you pay storage for them indefinitely. Buckets that receive lots of uploads accumulate these fragments for years.

The fix is a lifecycle rule that aborts incomplete multipart uploads after a few days. It is one setting, it costs nothing, and almost every bucket should have it. Varv's bucket settings expose lifecycle rules directly, so adding it does not require the web console.

Part size, briefly

Parts can range from 5 MB to 5 GB, with at most 10,000 parts per object. Bigger parts mean fewer requests; smaller parts mean finer resume and better parallelism on unstable connections. Good clients pick this for you based on file size, and some services prefer larger parts than the usual defaults, which is what the provider presets are for.

Big files without the anxiety

Varv splits, parallelizes, resumes and verifies large uploads automatically, and its queue survives anything short of a power cut. Actually, that too.

Get notified at launch