A silent checksum that broke every S3-compatible upload

Problem

We had a small Go service writing generated files to an S3-compatible object store backed by a self-hosted, Ceph-based gateway. It used a popular cloud-portable blob storage abstraction library - the kind that lets you swap s3://, gs://, or file:// behind one interface, so application code never has to know which backend it’s talking to.

Every write failed. Not intermittently — every single PutObject call came back with a 400 and an error along the lines of:

api error XAmzContentSHA256Mismatch: UnknownError

That error means the object store computed a hash of the request body and it didn’t match what the client claimed to have sent. On paper this points at data corruption in transit: a proxy mangling the body, a broken TLS terminator, bytes changing between signing and sending.

We ruled all of that out, methodically:

  • Forcing a specific AWS region made no difference, tried both a placeholder region and the store’s real one.
  • Forcing the SDK to sign the payload hash properly (instead of using the HTTPS default of an “unsigned payload” sentinel) made no difference.
  • Deleting the target object before every write, to rule out some kind of overwrite/versioning confusion, made no difference — the store just as happily rejected a PutObject into a key that didn’t exist yet.
  • The error was 100% reproducible, not load- or size-dependent.

None of that matched a genuine in-transit corruption story. So we stopped guessing and turned on the SDK’s raw request/response logging to see exactly what left the process.

That’s when it turned up: every outgoing PutObject carried an x-amz-checksum-crc32 header, and the request body was framed as a chunked, trailer-signed upload. We had explicitly configured the client to only compute checksums when the operation required one — not by default — via the SDK’s standard “when required” checksum-calculation setting. That setting was being read correctly and stored on our bucket handle. It just wasn’t reaching the code path that actually performed the write.

The blob abstraction library used a modern, streaming-capable “transfer manager” component internally to perform uploads, wrapping the plain SDK client. That transfer manager has its own copy of the checksum-calculation setting, entirely separate from the one on the underlying SDK client, and the library’s glue code that wires the two together only copied over a couple of unrelated options (buffer size, concurrency). The checksum setting was silently dropped. Left at its zero value, it resolved to the SDK’s own default — “checksum whenever supported” — which meant a CRC32 trailer got attached to every request whether we asked for it or not.

Our self-hosted gateway didn’t like that chunked, trailer-checksummed request format and rejected it, generically, as a content-hash mismatch. A real S3-compatible backend from a different vendor, or a newer version of the same one, might have handled it fine — which is exactly why this kind of gap goes unnoticed for a long time: it only bites you against implementations that are stricter, or just different, about a rarely-exercised code path.

Solution

Once the actual mechanism was clear, the fix was narrow and unglamorous: stop going through the library’s high-level write path for this one operation, and call the plain, low-level PutObject operation on the underlying SDK client directly.

The high-level path (call it WriteAll) is convenient — buffering, retries, multipart handling, all for free — but it’s also where the broken option-forwarding lived. The low-level client, constructed once and cached, respects the checksum-calculation setting correctly, because that logic lives directly in the officially maintained SDK, not in the third-party glue on top of it.

In sketch form, the shape of the fix looked like this:

// Before: goes through the library's writer abstraction, which
// silently ignores our checksum-calculation preference.
err := bucket.WriteAll(ctx, key, data, nil)

// After: bypass the abstraction for this one call and use the
// underlying client directly. Reads and deletes still go through
// the normal abstraction — only writes were affected.
_, err := client.PutObject(ctx, &s3.PutObjectInput{
    Bucket: aws.String(bucketName),
    Key:    aws.String(key),
    Body:   bytes.NewReader(data),
})

A few things made this the right level of fix rather than a hack:

  • It targets the actual mechanism, not a symptom. Retrying, adding backoff, or catching the specific error code and swallowing it would have “fixed” the visible failure while leaving every write silently un-persisted, or at best flaky. We tried something like that first (tolerate the specific error and continue) and it made the symptom — a 502 to the caller — go away, without making the write succeed. That’s a trap: it looks fixed because your smoke test passes, but the data never lands.
  • It’s scoped tightly. Only the write path changes. Reads, deletes, and every other bucket operation still go through the library exactly as before, because they were never in the broken code path to begin with. No need to abandon a useful abstraction wholesale over one gap in it.
  • It was verified end to end, not just by “no error returned.” After the change, a write immediately followed by a read of the same key came back byte-identical, confirmed against the real backend, not a mock.

The broader lesson: when a well-known, well-typed SDK error shows up (“checksum mismatch”) against infrastructure you don’t fully control, resist the urge to treat the error message as the whole story. It’s an accurate description of what the server observed, but it says nothing about why the client sent that. Reproducing the failure while varying every plausible cause we could configure - region, signing mode, object existence — and getting no change was itself a strong signal that the actual cause lived somewhere upstream of our configuration entirely. Raw wire-level logging settled it in minutes once we reached for it; we probably should have reached for it several wrong guesses earlier.