Remove Duplicate Lines
Remove, find or count duplicate lines in a list, with options to ignore case, trim whitespace and keep either the first or last occurrence.
—
Which spelling survives is decided by which you typed first
With "ignore case" on, the tool groups lines by their lowercased form and keeps the line as you wrote it — the first time you wrote it. So the same three lines in three orders give three different answers:
| Input order | Survivor | Unique count |
|---|---|---|
Apple, APPLE, apple | Apple | 1 |
apple, APPLE, Apple | apple | 1 |
APPLE, apple, Apple | APPLE | 1 |
The count is the same all three times, and that count is what most people are after. The surviving text is not a property of the data; it is a property of the order the data arrived in. That is the only sensible choice — there is no principled way to pick between three spellings — but it is worth knowing before pasting a list whose casing you care about.
The options run in a fixed order, and the order matters
Trim runs first, then the blank filter, then the deduplication. Which means trimming
creates blanks for the filter to remove. With the input
"a", " ", "b":
| Options | Output |
|---|---|
| Neither | "a", " ", "b" |
| Trim only | "a", "", "b" |
| Ignore blanks only | "a", " ", "b" |
| Both | "a", "b" |
A line of three spaces is not blank until something trims it. "Ignore blank lines" on its own leaves it alone, which reads as a bug and is a sequencing consequence — the two options together do more than either alone.
Trim also merges lines that were genuinely distinct strings:
"a", " a ", "a " is
3 different values and 1 trimmed one. Whether
that is what you want depends entirely on where the text came from.
The three modes are one grouping viewed three ways
| Mode | On a, b, a, c, b, a |
|---|---|
| Unique lines | a, b, c |
| Only duplicates | a, b |
| Appears exactly once | c |
The last two are a clean partition of the first — 2 plus
1 is 3, with no line in both halves, and that
holds on any input whatever the options are set to. So these are not three different
algorithms; they are one grouping shown three ways. If the duplicate list and the
appears-once list ever failed to add up, something would have gone wrong upstream of all
three.
How to use
- Paste your list, one item per line.
- Trim whitespace first if the data came from a spreadsheet.
- Choose whether to remove duplicates or just show them.
- Copy the cleaned list.
Frequently asked questions
Why are my duplicates not being removed?
Almost always invisible differences. Trailing spaces, a mix of Windows and Unix line endings, a non-breaking space pasted from a web page, or differing capitalisation all make two apparently identical lines distinct. Trimming and ignoring case usually resolves it.
Should I keep the first or the last occurrence?
Depends on the data. For a log or an append-only export, the last is often the most current record. For a manually curated list, the first is usually the intended one. Where the lines are truly identical it makes no difference at all.
Does removing duplicates preserve the original order?
Here, yes — the surviving lines stay in their original sequence. Some tools sort as a side effect of deduplicating, which is convenient sometimes and destructive when the order carries meaning, as in a log or a ranked list.
How do I find duplicates without removing them?
Use the option that lists them instead. Seeing which entries repeat, and how often, is frequently more useful than a cleaned list — it tells you whether you have a data-entry problem or a legitimate one-to-many relationship.
What counts as a duplicate for case?
By default, an exact match. With case ignored, Apple and apple are duplicates, and the version kept is whichever occurrence you chose to keep. This matters for email addresses, where the domain is case-insensitive and the local part technically is not, though in practice almost every provider treats it as such.
Can I deduplicate on part of a line?
Not with a plain line comparison — that needs a field-aware tool. If your lines are comma-separated records and you want to deduplicate on the first column only, a CSV tool will do it properly, whereas a line-based one will treat every distinct full row as unique.
🔒 This tool runs entirely in your browser. Nothing you enter is uploaded, logged, or stored.