What this handles
Cleaning up a list is one of those tasks that appears constantly and has no convenient home. Email lists with duplicates, log lines needing deduplication, CSV columns pasted out of a spreadsheet, keyword lists, import files — all need the same handful of operations.
Everything runs locally in your browser. Nothing is uploaded, which matters when the list contains addresses, identifiers, or anything else you would not paste into an unknown server.
Why alphabetical sorting is not straightforward
Naive sorting compares character codes, which produces results that look wrong to humans.
In ASCII, all uppercase letters come before all lowercase, so Zebra sorts before apple. Accented characters sort after all unaccented ones, so éclair ends up after zebra. Neither matches what anyone expects.
Proper alphabetical ordering requires locale-aware collation, which JavaScript provides through Intl.Collator. It handles case folding, accent equivalence, and language-specific rules — in Swedish, for example, å sorts after z rather than with a.
This tool uses locale-aware comparison, which is why its output differs from a naive sort and matches what a spreadsheet would produce.
The numeric sorting trap
Sorting numbers as text produces a well-known and frequently shipped bug: 1, 10, 100, 2, 20, 3.
The cause is straightforward — string comparison proceeds character by character, and "1" is less than "2", so "10" sorts before "2". It looks correct in small samples and only becomes obvious once values cross a digit boundary.
The same problem affects version numbers, where v1.10 must sort after v1.9, and file names like image10.png. Handling these correctly needs natural sort ordering, which JavaScript supports via Intl.Collator with the numeric: true option.
Deduplication and what counts as identical
Exact matching is what this tool performs, and it is worth knowing what that misses:
- Trailing whitespace.
appleandappleare different strings. Trimming first is usually what you want, which is why it is the default here. - Case.
Appleandappleare distinct. For email addresses the local part is technically case-sensitive but virtually never treated as such in practice. - Unicode normalisation. An accented character can be encoded as a single code point or as a base letter plus a combining mark. These look identical and compare as different. Normalising with
normalize('NFC')resolves it. - Invisible characters. Non-breaking spaces, zero-width joiners, and byte-order marks all survive copy-paste and break exact matching invisibly.
Doing this from the command line
For files large enough that a browser struggles, the Unix tools are decades old and extremely fast:
sort file.txt | uniq— sort then remove adjacent duplicates.uniqonly compares neighbouring lines, so sorting first is required.sort -u file.txt— the same thing in one step.sort -n— numeric sort, avoiding the string-ordering trap.uniq -d— show only the duplicated lines.uniq -c— prefix each line with its count.awk '!seen[$0]++' file.txt— remove duplicates while preserving the original order, whichsort -ucannot do.
Frequently asked questions
Why does my alphabetical sort put uppercase first?
Naive sorting compares character codes, where all uppercase letters precede all lowercase. Locale-aware collation via Intl.Collator handles case and accents the way a human would expect.
Why do my numbers sort as 1, 10, 2?
Because they are being compared as text, character by character. Use numeric sorting, or Intl.Collator with the numeric option, which also handles version numbers and file names correctly.
Why are two lines that look identical not deduplicated?
Usually trailing whitespace, differing case, or Unicode normalisation - an accented character can be one code point or a base letter plus a combining mark. Invisible characters from copy-paste are another common cause.
How do I remove duplicates while keeping the original order?
In this tool, choose remove duplicates without sorting. On the command line, awk '!seen[$0]++' file.txt does the same, which sort -u cannot.