Sort, uniq and comm for Comparing Datasets

1. Why sort, uniq and comm are unbeatable for dataset comparisons

The sort uniq comm trio is among the oldest Unix tools of all and still remains the fastest solution for many dataset comparisons. The reason lies in the clear division of labor: sort brings lines into a defined order, uniq detects and reduces directly adjacent duplicates, and comm compares two already sorted files line by line and splits the result into three columns, exclusive to file one, exclusive to file two, and shared lines.

Unlike a database query or a script in a higher level language, sort uniq comm needs no additional software installation, no intermediate imports, and no connection setup delay. The tools come preinstalled on practically every Linux and Unix system and work stream based, which makes them ideal for ad hoc comparisons in deployment scripts and cron jobs.

Typical use cases for sort uniq comm include comparing two user lists from different systems, detecting newly added or removed product IDs between two exports, or deduplicating email addresses from several source files. The following sections show how the three tools work individually and how they are combined to build reliable comparisons between datasets.

2. Using sort correctly: keys, options, stable sorting

The basic requirement for every sort uniq comm comparison is correct sorting, because both uniq and comm expect already sorted input and silently return wrong results on unsorted data, without printing an error. The -k option defines the sort key for multi column data, for example sort -t',' -k2,2 to sort by the second column of a CSV file separated by commas.

For numeric sorting, -n is mandatory, because the default sort works lexicographically and would place the string "10" before "9". With -u, sort removes duplicates already during sorting, which in many cases makes a subsequent call to uniq unnecessary. For reproducible sort uniq comm pipelines across different environments, LC_ALL=C should also be set, to keep sort order consistent regardless of the system language.

#!/usr/bin/env bash
set -euo pipefail

# Numeric sort by second column, comma-separated file
LC_ALL=C sort -t',' -k2,2n customers.csv

# Sort and deduplicate in a single pass
LC_ALL=C sort -u email_list_raw.txt > email_list_unique.txt

# Multi-key sort: primary by department, secondary by salary descending
LC_ALL=C sort -t',' -k2,2 -k3,3nr employees.csv

# Stable sort keeps original relative order for equal keys
LC_ALL=C sort -s -t',' -k1,1 export.csv

3. uniq for duplicates and frequencies: -c, -d, -u

uniq only works on directly adjacent lines, which is why combining it with a preceding sort is essential for sort uniq comm tasks. Without sorting, uniq only detects duplicates that happen to already sit next to each other in the file, all others go unnoticed. The -c option prefixes every line with its occurrence count, which is ideal for frequency analyses like "how often does each IP address appear in this access log".

With -d, uniq outputs only lines that occur at least twice, useful for specifically hunting down duplicates in a list that should actually be unique, such as email addresses or order numbers. Conversely, -u shows only lines that occur exactly once, which helps separate truly unique entries from recurring ones. For sort uniq comm workflows that need to be restricted to certain columns, uniq also offers -f N to ignore the first N fields during comparison.

#!/usr/bin/env bash
set -euo pipefail

# Count occurrences per IP address in an access log, sorted by frequency
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

# Find duplicate email addresses that should be unique
sort emails.txt | uniq -d

# Show only truly unique entries (appearing exactly once)
sort ticket_ids.txt | uniq -u

# Ignore the first field when comparing (e.g. timestamp prefix)
sort logs.txt | uniq -f 1

4. comm for line by line comparison of two sorted files

While uniq works within a single file, comm compares two separate, each sorted files line by line and splits the result into three columns: lines only in file one, lines only in file two, and lines in both files. Exactly this behavior makes sort uniq comm the ideal combination for set operations between two lists, without needing a database or a more complex tool.

With the options -1, -2 and -3, individual columns can be hidden on purpose. comm -23 file_a file_b suppresses columns two and three and shows only lines that exist exclusively in file_a, while comm -13 file_a file_b shows only lines that exist exclusively in file_b. These two variants are the most common uses of sort uniq comm in everyday work, because they directly answer what got added or removed between two snapshots.

#!/usr/bin/env bash
set -euo pipefail

# Prepare sorted, deduplicated files first — mandatory for comm
sort -u yesterday_ids.txt > /tmp/yesterday.sorted.txt
sort -u today_ids.txt > /tmp/today.sorted.txt

# Full three-column output: only in yesterday | only in today | in both
comm /tmp/yesterday.sorted.txt /tmp/today.sorted.txt

# Only new entries (present today, absent yesterday)
comm -23 /tmp/today.sorted.txt /tmp/yesterday.sorted.txt

# Only removed entries (present yesterday, absent today)
comm -23 /tmp/yesterday.sorted.txt /tmp/today.sorted.txt

# Only entries present in both (intersection)
comm -12 /tmp/yesterday.sorted.txt /tmp/today.sorted.txt

5. Combined pipelines: diffs between exports, lists, snapshots

In practice, sort, uniq and comm are rarely used in isolation, but combined into pipelines that model several processing steps in a single command. A typical pattern for sort uniq comm: a raw data source is first reduced with awk or cut to the relevant column, then sorted and deduplicated, before being compared with a second, similarly prepared file via comm.

For snapshot comparisons over time, for example daily exports of a product database, it pays off to archive the prepared intermediate files with a timestamp, so later sort uniq comm runs enable not just the most recent comparison, but also historical comparisons between any two days. This intermediate layer costs hardly any extra storage, because sorted, deduplicated ID lists are usually much smaller than the raw data they were built from.

6. Large datasets: sort performance with --parallel and -T

On very large files in the gigabyte range, sort itself becomes the bottleneck of the whole sort uniq comm pipeline. GNU sort supports the option --parallel=N to distribute the external merge sort algorithm across several CPU cores, which can significantly reduce sort time on multi core systems. Additionally, -S controls the amount of memory sort may use before spilling to disk, for example -S 2G for two gigabytes.

When sort operations have to spill to disk because the file does not fit entirely in memory, sort creates temporary files. With -T /path/to/fast/storage, this directory can be explicitly redirected to a fast SSD volume or a tmpfs, instead of using the system wide default under /tmp, which can itself become a bottleneck on heavily loaded servers. For sort uniq comm pipelines that run regularly on large exports, this fine tuning is often the decisive factor between a runtime of seconds and several minutes.

#!/usr/bin/env bash
set -euo pipefail

# Use 4 parallel threads and a fast tmpfs for temporary sort files
sort --parallel=4 -S 2G -T /dev/shm -u huge_export.csv > huge_export.sorted.csv

# Measure the effect of parallel sort on a large file
time sort --parallel=1 huge_export.csv > /dev/null
time sort --parallel=4 huge_export.csv > /dev/null

7. Practical example: checking two CSV exports for new, removed and shared entries

A concrete example for sort uniq comm in daily deployment work: two daily CSV exports of a product catalog need to be checked for which product IDs got added, which were removed, and which stayed unchanged in both exports. The process first extracts the ID column from both files, sorts and deduplicates them, and then applies comm with the matching suppression flags.

This pattern can be extended arbitrarily, for example by additionally fetching the affected rows from the original file with grep -Ff, so that not just the IDs, but the complete records of the new or removed entries are printed. That turns three simple sort uniq comm commands into a complete, automatable report for product catalog changes.

#!/usr/bin/env bash
set -euo pipefail

# Extract, sort and deduplicate product IDs from both exports
awk -F',' 'NR>1{print $1}' export_yesterday.csv | sort -u > /tmp/ids_yesterday.txt
awk -F',' 'NR>1{print $1}' export_today.csv     | sort -u > /tmp/ids_today.txt

echo "New product IDs:"
comm -23 /tmp/ids_today.txt /tmp/ids_yesterday.txt

echo "Removed product IDs:"
comm -23 /tmp/ids_yesterday.txt /tmp/ids_today.txt

echo "Unchanged product IDs (present in both):"
comm -12 /tmp/ids_yesterday.txt /tmp/ids_today.txt | wc -l

# Fetch full rows for the new IDs from the current export
grep -Ff <(comm -23 /tmp/ids_today.txt /tmp/ids_yesterday.txt) export_today.csv

8. Pitfalls: locale, case sensitivity, field separators

The most common mistake in sort uniq comm comparisons is an inconsistent locale between the sort calls involved. Without LC_ALL=C, sort uses the system wide locale, which orders accented characters, case, and special characters differently than a pure byte based sort. If one file is sorted with a different locale than the other, comm returns seemingly wrong results, because the supposedly sorted order actually is not identical.

A second pitfall concerns case sensitivity: sort treats "Apple" and "apple" as different strings by default, which can lead to unexpected results in sort uniq comm comparisons of names or email addresses, if the same address appears once in lowercase and once in uppercase. The -f option in sort ignores case during sorting, while uniq separately can also detect case insensitive duplicates with its own -i option. The third pitfall, differing field separators between two CSV files, should be checked before every comparison, to make sure both sources actually use the same separator and the same column order, otherwise comm compares values that are not related at all in content.

9. Comparison: comm vs. diff vs. join for dataset comparison

Alongside sort uniq comm, there are two other standard tools, diff and join, that at first glance solve similar tasks but differ in important details.

For pure set operations between two sorted lists, sort uniq comm remains the most direct solution, because the result is immediately split into three clearly separated columns. diff, on the other hand, is the right choice when actual text changes within lines matter, for example for config files or source code. join shows its strength once additional columns from both files need combining on a shared key, similar to a SQL join.

10. Summary

Sort uniq comm together form a lean, universally available solution for comparing two datasets, without a database and without additional software. sort establishes the mandatory sorted input, uniq detects duplicates and frequencies within a file, and comm compares two sorted files line by line and delivers the result directly in three clearly separated columns.

For reliable results, a consistent locale with LC_ALL=C across all involved sort calls is decisive, as is a uniform field separator between the files being compared. On very large datasets, --parallel and -T lead to significantly shorter runtimes. For pure set comparisons, sort uniq comm remains the most direct solution, while diff is better for text changes and join for column wise combinations.

11. FAQ: Sort, uniq and comm for dataset comparison