Hashing a File in Elixir

April 10, 2019

X Follow Button

A hash function is a function that converts a variable size sequence of bytes (a string, a file content etc.) to a fixed size sequence of bytes, called digest. This means that hashing a file of any length, the hash function will always return the same unique sequence of bytes for that file. It’s a sort of digital fingerprint, usually represented by an hexadecimal string of length between 32 and 128 characters.

The hash of a file is useful, for example, to check if the content of two files is identical, or if the content was corrupted during the download.

There are different hash functions, MD5, SHA-1, SHA-2, SHA-3 etc. , many of them available in Elixir.

Update 👨‍💻: I had initially written the examples below using MD5 algorithm (which is the weakest in the list), just because I thought to be the fastest one. @Hauleth pointed out that SHA-1 and SHA-256 should be faster on new CPUs due to Intel SHA Extensions, so I rewrote the examples using SHA-256

Hashing a string

Let’s start hashing a string using the SHA-256 algorithm.

 iex> :crypto.hash(:sha256,"I love Elixir")
<<164, 35, 167, 235, 69, 224, 253, 77, 180, 92, 77, 172, 37,...>>
iex> :crypto.hash(:sha256,"I love Elixir!")
<<209, 119, 188, 230, 168, 124, 98, 212, 119, ...>>

Elixir

Copy

We’ve used the hash/2 function in the :crypto Erlang module.

The first argument is the name of the hash algorithm we want to use, in this case :sha256, the second argument is the sequence of bytes we want to hash, in this case a string. It returns a sequence of bytes.

We see how the output changes just by appending a “!” character.

We can use Base.encode16/1 to get the hexadecimal string representation

iex> :crypto.hash(:sha256,"I love Elixir!") \
...> |> Base.encode16() \
...> |> String.downcase()
"d177bce6a87c62d4772f404fcad2f8c2d9606c04f99942b71d7c521eb79c4c3b"

Elixir

Copy

If you are on a Linux or Mac machine, you can use a command line tool like sha256sum to see that the digest corresponds

$ echo -n 'I love Elixir!' |  sha256sum
d177bce6a87c62d4772f404fcad2f8c2d9606c04f99942b71d7c521eb79c4c3b -

Bash

Copy

Hashing a file

Calculating the hash of a file is conceptually the same as calculating the hash of a string. A file is a sequence of bytes and we could use the same :crypto.hash(:sha256, file_content_binary) function. But we saw that most of the time is not a good idea to load the whole file into memory!

We can use File.stream! and a different set of functions available in :crypto to read and process a file in chunks.

Let’s see first a simple example using the same string we’ve used before, divided into chunks

iex> [chunk_1, chunk_2] = ["I love ", "Elixir!"]
iex> hash_ref = :crypto.hash_init(:sha256)
#Reference<...36636>
iex> hash_ref = :crypto.hash_update(hash_ref, chunk_1)
#Reference<...36647>
iex> hash_ref = :crypto.hash_update(hash_ref, chunk_2)
#Reference<...36655>
iex> digest = :crypto.hash_final(hash_ref)
<<209, 119, 188, 230, 168, 124, 98, 212, 119, ...>>

iex> digest |> Base.encode16() |> String.downcase()
"d177bce6a87c62d4772f404fcad2f8c2d9606c04f99942b71d7c521eb79c4c3b"

Elixir

Copy

We process the sequence in chunks getting the same result we’ve gotten previously, and we can obviously do the same with files:

hash_ref = :crypto.hash_init(:sha256)

File.stream!(file_path)
|> Enum.reduce(hash_ref, fn chunk, prev_ref->
  new_ref = :crypto.hash_update(prev_ref, chunk)
  new_ref
end)
|> :crypto.hash_final()
|> Base.encode16()
|> String.downcase()

Elixir

Copy

We get a hash reference from :crypto.hash_init(:sha256), which is passed to Enum.reduce as the first accumulator.
We use Enum.reduce to read each chunk from the file and add it to the calculation. The :crypto.hash_update/2 returns a new reference which is then set as the new accumulator.
Once processed all the chunks the final reference is then piped into the :crypto.hash_final/1 function which returns the SHA-256 digest of the file.

We can write the reduce function in a nicer and more compact way

File.stream!(file_path)
|> Enum.reduce(:crypto.hash_init(:sha256),&(:crypto.hash_update(&2, &1)))
|> :crypto.hash_final()
|> Base.encode16()
|> String.downcase()

Elixir

Copy

`File.stream!` chunks vs lines

By default File.stream! emits lines instead of just chunks. Emitting lines is slower than emitting chunks, I think because the stream needs to look for newlines while splitting the chunks in strings.

To force the stream to emit chunks we use File.stream!/3

iex> File.stream!(file_path, [], 2_048)
%File.Stream{
  line_or_bytes: 2048,
  modes: [:raw, :read_ahead, :binary],
  path: file_path,
  raw: true
}

Elixir

Copy

setting a chunk size of 2048 bytes.

I made a quick benchmark ( you can find on this gist) where we see that streaming chunks is faster and also better memory wise.

Name             ips        average  deviation         median         99th %
chunks       23.29 K       42.93 μs    ±63.44%       41.98 μs       83.98 μs
lines         9.21 K      108.54 μs    ±42.52%       93.98 μs      275.98 μs

Comparison:
chunks       23.29 K
lines         9.21 K - 2.53x slower +65.61 μs

Memory usage statistics:

Name      Memory usage
chunks         2.11 KB
lines         20.84 KB - 9.88x memory usage +18.73 KB

Bash

Copy

Wrap up

We’ve seen what a hash function is and how to easily calculate the hash of a file using Elixir.

In the past (unfortunately I think still in the present 😅), hash functions were used to store passwords in the database. If you need to securely handle and store passwords, please use the bcrypt_elixir library!

tempest.services.disqus.com is blocked

This page has been blocked by an extension

Try disabling your extensions.

ERR_BLOCKED_BY_CLIENT

Reload

This page has been blocked by an extension

Disqus Comments

We were unable to load Disqus. If you are a moderator please see our troubleshooting guide.

Start the discussion…

Comment

Log in with

or sign up with Disqus or pick a name

Disqus is a discussion network

Don't be a jerk or do anything illegal. Everything is easier that way.

Read full terms and conditions

This comment platform is hosted by Disqus, Inc. I authorize Disqus and its affiliates to:

Use, sell, and share my information to enable me to use its comment services and for marketing purposes, including cross-context behavioral advertising, as described in our Terms of Service and Privacy Policy, including supplementing that information with other data about me, such as my browsing and location data.
Contact me or enable others to contact me by email with offers for goods or services
Process any sensitive personal information that I submit in a comment. See our Privacy Policy for more information

Acknowledge I am 18 or older