Lesson 1 of 5

Hashing & Integrity

"If you think cryptography is the solution to your problem, then you don't understand cryptography and you don't understand your problem." — Peter G. Neumann

Part IV ended with permission. This Part is about protection, and it opens with the primitive engineers use most and understand least.

Hashing is where cryptographic misunderstanding starts, because a hash function looks like one tool and is actually a family of tools with incompatible purposes. The same three words — "we hash it" — are used to describe verifying a download, storing a password, signing a webhook and generating a cache key, and at most one of those is served well by any given hash function. SHA-256 is excellent for three of them and catastrophic for the fourth. bcrypt is right for the fourth and useless for the others. The mistake is not choosing a weak algorithm; it is choosing a strong algorithm designed for a different job.

Speed makes the point concrete. A hash function's speed is a feature when you are checksumming a 4GB file and a vulnerability when you are storing passwords, and it is the identical property in both cases. Part III made this argument for credential storage; here we generalise it, because the pattern recurs across every primitive in this Part — a property that is a strength in one context is the weakness in another, and cryptographic failure is almost always a context error rather than an algorithm error.

Then there is the second axis, which is the one people miss entirely: a plain hash proves nothing about who produced it. If an attacker can change the message, they can recompute the hash, and integrity without authenticity is a checksum against corruption rather than a defence against an adversary. That is what keys are for, and it is the whole reason HMAC exists.

The big question

You publish a file alongside its SHA-256 hash so users can verify the download. An attacker who has compromised your web server replaces the file — and also replaces the published hash with the hash of their file. Every user who checks gets a perfect match. What property does a plain hash actually provide, what does it not, and what would you have needed to publish instead?

One word, four different jobs

Tests — Run & Watch Them Pass

Run to execute the tests…

The last test is worth dwelling on, because "we hashed the identifiers so the data is anonymous" appears in real privacy documentation. A hash is only one-way when the input is unguessable. There are roughly 10 billion possible phone numbers; hashing all of them is minutes of work.

Everyday analogy

A tamper-evident seal on a package tells you whether it was opened in transit, and it tells you nothing about who applied it. If the person who wants to alter the contents can also buy the same seals, the seal proves only that somebody sealed it — which is exactly the situation of a hash published on the same server as the file. A wax seal with a signet ring is the fix, and note what makes it work: not that the wax is harder to break, but that the ring is something only one person holds. The security has moved from the seal into the key, which is the move every remaining chapter in this Part makes in some form.

Collisions, and why MD5 is not merely old

Tests — Run & Watch Them Pass

Run to execute the tests…

From the archives

The Flame malware, discovered in 2012, used an MD5 collision to forge a code-signing certificate that chained to Microsoft's own certificate authority — which meant it could present itself to Windows Update as legitimate Microsoft software. The collision attack involved was novel enough that cryptographers reconstructing it concluded it had been developed independently of the published academic work. What makes Flame the definitive cautionary tale is the timeline: MD5 collisions had been demonstrated publicly in 2004, and a rogue CA certificate built on them was demonstrated at a conference in 2008. Four years of clear public warning, and the deprecation had still not reached every corner of a system as scrutinised as Windows Update. Cryptographic migrations take a decade, which is why the time to move off a primitive is when it is theoretically weakened, not when it is practically broken.

Keys turn integrity into authenticity

Tests — Run & Watch Them Pass

Run to execute the tests…

The fourth and fifth tests together explain why you must never build your own MAC. SHA256(secret + message) looks obviously correct and is forgeable without the key — an attacker appends &role=admin and produces a valid tag, because of a property of how the hash processes blocks that has nothing to do with the secret's strength.

Engineer's notebook

The webhook verification bug I have seen most often is not a missing check; it is a check performed against re-serialised JSON. The framework parses the body, the handler stringifies it again to compute the HMAC, and the tag never matches because key order or whitespace changed. Under deadline, the fix that ships is to log a warning and continue, and now the endpoint accepts anything. If you take one operational habit from this chapter: capture the raw body before parsing, verify against those exact bytes, and make verification failure a hard rejection with no code path that proceeds anyway. The second habit is a timestamp check — a signature with no freshness bound means a request captured once is valid forever, which turns an old log entry into a live credential.

Common Mistakes

Using a fast hash for passwords. SHA-256 is the right tool for integrity and the wrong one for credentials, and the property that makes it right is the property that makes it wrong.
Publishing a hash next to the file it protects. Anyone who can replace one can replace the other. That needs a signature and an out-of-band public key.
Treating a hash as anonymisation. Low-entropy inputs — emails, phone numbers, identifiers — are recovered by enumerating candidates.
Rolling your own MAC as hash(secret + message). Length extension forges valid tags without the key. Use HMAC.
Comparing digests with ==. The early return leaks the position of the first mismatch, turning an infeasible guess into a byte-at-a-time search.
Verifying a webhook against re-serialised JSON. The bytes change, verification fails, and the pressure is always to proceed anyway.
Signing without a freshness bound. A valid signature with no timestamp is a credential that never expires.
Defending an aging primitive because no preimage exists. Signatures break on collisions, and collision resistance is the half that falls first.

Interview Corner

Interview Corner — the same idea at three levels. Answer yours; recognise the one above it.

Warm-up

What is the difference between hashing and encryption?

Model answer — Encryption is reversible by design — there is a key, and with it you get the original back — so it is for confidentiality, for data you need to read later. Hashing is one-way: it produces a fixed-size digest and there is no operation that recovers the input, so it is for verifying rather than for storing. That is why passwords are hashed rather than encrypted; you never need the original, you only need to check whether a submitted value produces the same digest, and encrypting them would mean a key exists somewhere that turns your database back into a plaintext password list. Two qualifications I would add. Hashing is only one-way when the input is unguessable — hashing an email address or a phone number is not anonymisation, because an attacker just hashes every candidate and compares, and a phone number space is about ten billion, which is minutes of work. And not all hashes are for the same job: a fast hash like SHA-256 is right for checking a file's integrity and wrong for passwords, where you want something deliberately slow like argon2id.

Intermediate

A colleague verifies webhooks with sha256(secret + body) and compares with ==. What is wrong?

Model answer — Two separate bugs, either of which is enough to forge a request. First, the construction is vulnerable to length extension. SHA-256 is a Merkle–Damgård hash, so its internal state after processing a message is exactly the digest — which means an attacker who has a valid digest for some body, plus the body's length, can continue hashing from that state and produce a valid digest for body + anything without ever knowing the secret. If the body is something like user=ada&role=viewer and the parser takes the last value wins, they append &role=admin and have a validly signed admin request. HMAC exists specifically to prevent this — its nested two-pass construction means the output is not resumable state. Second, comparing with == returns as soon as bytes differ, so the time taken leaks how many leading characters were correct. That turns guessing a 64-character hex signature from an infeasible search into roughly 64 × 16 guesses, one byte at a time. The fix is a constant-time comparison — crypto.timingSafeEqual or hmac.compare_digest — which examines every byte regardless. I would also check two things people miss: that the HMAC is computed over the raw body bytes rather than re-serialised JSON, because re-serialising changes bytes and the usual "fix" is to stop verifying; and that there is a timestamp check, because a signature with no freshness bound means a captured request stays valid forever.

Advanced

Design integrity protection for a system that distributes software updates to a million devices.

Model answer — The governing constraint is that a hash published alongside the artefact proves nothing, because whoever can replace the artefact can replace the hash — so the root of the design is a signature over the artefact digest, with the verifying key distributed out of band and embedded in the device rather than fetched at update time. Concretely: SHA-256 for the digest, Ed25519 for the signature — fast to verify on constrained hardware, small, and no parameter choices to get wrong. The signature covers a manifest rather than a single file, and the manifest includes the version, the target device class and an expiry, so a signed-but-old update cannot be replayed to downgrade a device to a version with a known vulnerability. Rollback protection is the requirement people forget: without a monotonic version counter enforced on the device, an attacker with a valid signature from last year can pin you to last year's bugs forever. For key management, the signing key lives in an HSM, never on a build machine, and I would separate a rarely-used root key from an online signing key so that compromise of the pipeline does not compromise the trust anchor — which means designing key rotation and revocation up front, because a million devices with a hardcoded key and no rotation path is an unrecoverable position. I would look at TUF, The Update Framework, rather than inventing this: it addresses threats that are not obvious from first principles — freeze attacks where a device is served a stale-but-valid manifest indefinitely, mix-and-match attacks combining files from different versions, and compromise of individual role keys with graceful recovery. And I would make verification fail closed on the device with no override flag, because the operational pressure during a bad rollout is always to add one, and that flag becomes the vulnerability.

Company lens — how the same topic gets asked

micro1Concept: hashing vs encryption, and why passwords use a slow hash.

GoogleDepth: length extension, collision vs preimage bounds, timing side channels.

MetaDesign: signed update manifests, rollback protection, key rotation and revocation.

AmazonReality: a webhook that stopped verifying because re-serialised JSON broke the tag.

Hands-on

Find every place your codebase computes a hash and write down which of the three jobs it is doing: integrity, authenticity, or password storage. Any fast hash used on a credential is a finding. Then check every signature verification for two things — whether the comparison is constant-time, and whether the HMAC is computed over raw bytes rather than re-parsed data. If you have a webhook receiver, try sending it a request with a deliberately wrong signature and confirm it is rejected rather than logged; then send a valid request twice and see whether replay is caught. Finally, if anything in your system publishes a hash for users to verify, ask where the hash lives relative to the file, and what an attacker who controls that location would need to do.

Tests — Run & Watch Them Pass

Run to execute the tests…

Summary

"Hash" names three incompatible jobs — integrity, authenticity, password storage — and the same speed property is a feature for one and a vulnerability for another.
A bare hash detects corruption, not tampering. If the attacker can change the message they can recompute the digest, so a hash published beside the file it protects proves nothing.
Keys are what turn integrity into authenticity, and a signature plus an out-of-band public key is what the download-verification case actually needs.
Hashing is one-way only for unguessable inputs. Emails, phone numbers and identifiers are recovered by enumeration — hashing is not anonymisation.
Collision resistance is half the bit length and falls first. MD5's practical collision cost is around 2^18 against a theoretical 2^64, and Flame turned that into a forged Microsoft-chaining certificate four years after the public warning.
Never build a MAC as hash(secret + message). Length extension forges valid tags without the key; HMAC's nested construction exists to stop exactly that.
Compare digests in constant time, over raw bytes, with a freshness bound — three separate failures that each turn a correct-looking verification into no verification at all.
Four defaults cover nearly everything: SHA-256, HMAC-SHA-256, argon2id, Ed25519. The right amount of cryptographic creativity in application code is none.

The next chapter moves from proving data is unchanged to keeping it secret — and to the discovery that encryption without authentication leaves an attacker able to change your plaintext without ever reading it.

Editorial reflection

The reason cryptography defeats good engineers is that it inverts the normal relationship between code and correctness. Everywhere else, code that produces the right answer on the inputs you tried is probably right, and testing is a reasonable proxy for correctness. Cryptographic code produces the right answer on every input you will ever try while being completely broken — the naive MAC verifies genuine messages perfectly, rejects random garbage perfectly, and is forgeable by anyone who knows what Merkle–Damgård means. The == comparison returns the correct boolean every single time; its flaw is in how long it takes to do so, which no test asserts and no review catches by reading. This is why the discipline here is unlike the rest of software engineering: you cannot reason your way to a correct construction from the properties that seem relevant, because the properties that matter are the ones you have not heard of. The practical consequence is not "learn more cryptography" — it is to recognise the class of problem and reach for the primitive someone has already analysed. Nearly every cryptographic disaster in application code began with an engineer who understood the maths well enough to feel confident and not well enough to know which question they had failed to ask.

Enrol to track progress

Q&A · 0

Enrol to ask questions and join the discussion.

No questions yet — be the first to ask.