Message Digest

A message digest (a.k.a. cryptographic hash function) is another cryptographic algorithm, but it is very different from encryption/decryption. It normally has no key (although there is a variant of this concept that does use a key, called an HMAC, or Hash-based Message Authentication Code).

Since there is no key with a basic message digest algorithm, a given algorithm will always produce the same digest for a given message. With encryption, if you have the correct key, you can recover the plaintext - so the entire plaintext is included in the ciphertext, just hidden. A digest is usually much shorter than the message it characterizes, so there is no way it could contain the entire message, even in hidden form (except for very short messages). In fact, by intent, it is (virtually) impossible to recover the message or any part of it, from the digest - creating a digest is a one-way (irreversible) process. Every step in encryption must be reversible ("one to one onto"). With a digest, it mostly does operations that can't be reversed, such as "many-to-one" mappings.

When someone publishes a file for others to download, they often include a message digest of the file, so that you can recreate the digest on your downloaded copy to verify that it is the same file they uploaded (and hasn't had some other file substituted, or an error occurred during downloading). Of course, if someone substituted a bogus file, they could just update the published digest to be that of the substituted file (anyone can easily create a digest of any file).

A basic message digest algorithm can accept any input data (textual or binary), of any size (including an entire book), and produce as output a fixed length (128 to 512 bits, depending on algorithm) cryptographic residue of that input data. This residue, or digest, characterizes the input data in some way. If there is any change at all to the input data (change one character of the book "War and Peace") you get a completely different digest. However, for a given message, a particular message digest algorithm (e.g. SHA-1) will always reliably produce the same identical digest. This makes it a very sensitive detector of change in files.

With a message digest, the object is to mix the information thoroughly, so that every bit of the message is figured into the digest somehow. It is also important that there is no way to recover any of the original message from the digest. Since digests can be made of any size message, there are far more messages than there are possible digests. This means that there exist multiple messages that will produce the same digest. In practice, with even 160 bit digests, if you were to produce a digest of every book ever published it is unlikely that any two would have the same digest. The number of possible digests (while small compared to all possible messages) is still vastly larger than the number of books ever published. Finally, it must be extremely difficult to make some malicious change to a message and figure out "offsetting" changes elsewhere in the message that would result in the same digest (this is trivial to do with a 16 bit checksum).

Over time, as computing power has continued to increase, and research into message digests is done, attacks have been found that make some digest algorithms fail one or more of the above requirements. MD2 and MD5 were "compromised" years ago, and now SHA-1 is considered "weak". New applications should use SHA-2 or SHA-3. SHA-1 should be used only for backward compatibility with old applications.

Common message digest algorithms are MD5, SHA-1, RIPEMD, SHA-2 and SHA-3. Note that some of these are actually families of algorithms, with different output lengths. SHA-1 comes only in a 160 bit flavor. SHA-2 actually supports digest lengths of 224, 256, 284 and 512 bits. These are often called SHA224, SHA256, SHA384 and SHA512. MD5 and SHA-1 are now obsolete (because of successful attacks against them) and should only be used for compatibility with old products. SHA-3 is not intended to replace SHA-2 (at least currently) because no successful attacks are known against SHA-2 - it is just an alternate technology in the event attacks against SHA-1 can be applied against SHA-2 someday (SHA-2 is based on the SHA-1 design, just with longer digests). SHA-3 has just recently been approved and code implementing it is not widely available.

For comparison, it took the following times for the various message digest algorithms to read and process a 2.4 GB file, with the corresponding speed:

MD5 = 7.89 sec, 294.4 MB/sec

SHA1 = 8.924 sec, 260.3 MB/sec

RIPEMD160 = 42.561 sec, 54.6 MB/sec

SHA256 = 35.206 sec, 66.0 MB/sec

SHA384 = 139.587 sec, 16.6 MB/sec

SHA512 = 140.38 sec, 16.6 MB/sec

Note that SHA256 is both considerably stronger and faster than RIPEMD-160. Also note that SHA256 is about 4 times slower than SHA1, but SHA384 and SHA512 are much slower than SHA256. These timings were on an AMD FX-8350 running at 4 GHz (using C# managed code).

 

Here is message digest in mathematical notation:

M = Message

SHA1 = Message Digest algorithm

MD = Message Digest

 

Creating a Message Digest:

MD = SHA1(M)

A common use of message digest is to produce a digest of some message with a particular algorithm (e.g. SHA-1), then encrypt the digest with an asymmetric key algorithm (e.g. RSA) and the private key of the person doing this. The resulting encrypted digest is known as a digital signature.

 

Message Digest Demo App

The link below allows you to download a Windows .Net application that lets you explore how message digests work. You can type (or paste) any text in the message window and the message digest is automatically calculated and displayed (every time the message changes). You can also choose a file, which produces and displays the message digest of that file. You can select any of the following message digest algorithms: MD5, SHA1, RIPEMD-160, SHA56, SHA384 or SHA512. For files, it will display the time required and the speed (MB/sec).

For "safe computing", download the app to your computer, and run a virus check on it, before executing it. Once you have run it, there will be launch icons on both your desktop and in your Start Menu (under Sixscape Communications / MD_Demo). You can remove the application by running the installer again and choosing remove, or via Control Panel / Programs and Features.

Install MD_Demo App