Inspecting Binary Data in Bash with xxd and od: Hexdumps, Magic Bytes, Format Detection

1. Why command-line binary analysis stays indispensable in practice

Deployment scripts, upload handlers and backup routines constantly deal with files whose actual content cannot be trusted based on the file name or declared extension. A file with a .jpg extension might actually be a PNG, an uploaded archive might have been cut off mid-transfer, and a configuration script might receive a supposed text file that is actually UTF-16 encoded and therefore looks like gibberish.

In all these cases, a direct look at the raw bytes at the start of a file answers the question faster than any guess based on the file name. Tools like xxd and od have been standard equipment on every Linux system for decades, need no graphical interface, and drop straight into Bash scripts to automatically validate files before further processing.

2. xxd: the fast, readable hexdump

xxd produces a hexdump view with three columns per line: the offset within the file, the bytes in hexadecimal, and next to that the same bytes as printable ASCII characters, with non-printable bytes shown as a dot. This combined view makes xxd especially pleasant to read, because text sections inside a binary file show up immediately in the right-hand column, without having to translate hex values one by one.

For most debugging tasks, xxd -l N file is enough to show only the first N bytes, because the information relevant for format detection almost always sits right at the start of the file. That avoids dumping a potentially huge file in full when only the header actually matters.

#!/usr/bin/env bash
set -euo pipefail

# Show the first 32 bytes as a classic hexdump
xxd -l 32 upload.bin

# 00000000: 8950 4e47 0d0a 1a0a 0000 000d 4948 4452  .PNG........IHDR
# 00000010: 0000 0100 0000 0100 0806 0000 0072 b83b  .............r.;

# xxd can also reverse the process: turn a hexdump back into raw bytes
xxd -r -p <<< "89504e470d0a1a0a" > header.bin

3. od: POSIX-portable and flexible in output formatting

od (octal dump) predates xxd, is guaranteed to be present in practically every POSIX environment, and is therefore the more reliable choice when a script needs to run on as many systems as possible where xxd might not be installed. The name is a historical artifact, since od can output far more than just octal, including hexadecimal byte by byte with the option -t x1, just like xxd.

A practical advantage of od over xxd is the fine-grained control over the output format through the -t option: -t x1 for individual hex bytes, -t d4 for signed 32-bit integers, or -t c for a pure character view. That flexibility makes od especially useful when a binary file format has known numeric fields at fixed offsets that should be read directly as numbers rather than as hex bytes.

#!/usr/bin/env bash
set -euo pipefail

# Same first bytes, POSIX-portable hexdump with od
od -A x -t x1z -v upload.bin | head -n 2

# 000000 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52  >.PNG........IHDR<

# Read a 4-byte little-endian integer at a fixed offset as a decimal number
od -A none -j 16 -N 4 -t u4 upload.bin

4. Reading magic bytes: how file formats identify themselves at the start

Most binary file formats begin with a fixed byte sequence, the so-called magic number, that unambiguously points to the format, regardless of how the file is named. PNG files always start with 89 50 4e 47, ZIP archives with 50 4b 03 04, and PDF documents with the ASCII characters %PDF. These sequences are part of the respective format specification and practically never change between versions of the same format.

A Bash script can read these magic bytes directly with xxd or od and compare them against known signatures to determine whether an uploaded file actually is what it claims to be, entirely independent of the submitted file extension or the HTTP content-type header, both of which a client is free to manipulate. This check matters especially for upload handlers, which should never blindly trust client-supplied information.

#!/usr/bin/env bash
set -euo pipefail

detect_format() {
  local file="$1"
  local magic
  magic=$(xxd -l 4 -p "$file")

  case "$magic" in
    89504e47) echo "PNG image" ;;
    ffd8ffe*|ffd8ffdb) echo "JPEG image" ;;
    504b0304) echo "ZIP archive (or DOCX/XLSX/JAR)" ;;
    25504446) echo "PDF document" ;;
    1f8b0800|1f8b0808) echo "gzip archive" ;;
    *) echo "Unknown format: $magic" ;;
  esac
}

detect_format "upload.bin"

5. The file command as a fast first check, with limits

The file command internally uses exactly this kind of magic-byte database and delivers an immediate, readable answer for most common formats, without a script having to maintain its own signatures. For a quick first check in a deployment script, file --mime-type -b file is therefore often the most pragmatic starting point, because it reliably recognizes common image formats as well as archives, executables and text encodings.

The limits of file show up with proprietary or very specific binary formats that have no signature in the bundled database, or when a script needs to evaluate not just the broad format but a concrete field at a fixed offset, for example a version number inside a custom binary format. In those cases, reaching directly for xxd or od with a self-defined offset remains the only reliable solution.

#!/usr/bin/env bash
set -euo pipefail

# Quick MIME-type check before deciding how to process an upload
mime=$(file --mime-type -b uploaded_file)
case "$mime" in
  image/png|image/jpeg) echo "Processing as image" ;;
  application/zip)      echo "Processing as archive" ;;
  *) echo "Rejected: unsupported type $mime" >&2; exit 1 ;;
esac

6. Comparing byte ranges and locating corruption

When debugging a corrupted file, for example a backup that no longer unpacks after transfer, a direct byte-by-byte comparison against a known good copy helps. cmp -l lists every differing byte offset along with the differing values in octal, which, combined with a targeted xxd call on exactly that offset, lets you see the deviation in the context of the surrounding bytes.

This technique is especially valuable for distinguishing between purely random corruption, for example from a failing storage medium, and a structured deviation, for example a misconfigured character-set converter. Random corruption shows scattered single-byte differences, while a structured deviation often shows up at the same relative positions in a recurring pattern.

#!/usr/bin/env bash
set -euo pipefail

# List every differing byte offset between two files
cmp -l original.bin corrupted.bin | head -n 5

# Zoom into a specific offset (e.g. byte 1024) with 16 bytes of context
xxd -s 1016 -l 32 original.bin
xxd -s 1016 -l 32 corrupted.bin

7. Automating binary analysis in deployment and upload pipelines

In an automated deployment or upload pipeline, it pays off to place magic-byte validation as a standalone, reusable function at the very start of every processing step, instead of only running it manually when needed. Such a function validates a file before any further step, like unpacking, uploading to a storage bucket, or forwarding it to another tool, and aborts in a controlled way on an unexpected signature instead of handing a broken file to a downstream tool.

This validation should, just like any other input check, run as early as possible in the pipeline and produce a clear, human-readable error message stating both the expected and the actual value found. That considerably shortens troubleshooting when an automated run fails later and someone needs to figure out exactly why a specific file was rejected.

8. Not reading huge files in full: checking only the relevant bytes

Both xxd and od support offset and length limits (-l/-s for xxd, -N/-j for od) that prevent a script from reading a multi-gigabyte file in full just to check the first few bytes. For pure format detection, the first 16 to 64 bytes are almost always enough, and a script that consistently uses these limits stays at constant, barely measurable runtime even on very large files.

Anyone who instead accidentally pipes the entire file through xxd or od and only limits the output afterward with head still reads the complete file off disk, because the limit only kicks in after full processing has already happened. That unnecessary I/O load quickly adds up to a noticeable performance problem once automated checks run frequently across many files.

9. xxd, od and file compared directly

For day-to-day debugging, xxd is usually the most pleasant choice thanks to its readable, combined hex-and-ASCII output, while od, as a POSIX standard tool, is the more reliable option for portable scripts that also need to run on minimal systems without xxd. The file command remains the fastest route to a rough format statement, but it does not replace a targeted byte check once a script needs to validate a concrete, self-defined binary format.

10. Summary

11. FAQ: Inspecting Binary Data in Bash: The Essentials at a Glance