The operations are easy. The order is what gets it wrong.
Sorting a list and removing duplicates are both about four lines of code, which is why there are a thousand tools that do them. What almost none of them tell you is that doing the same operations in a different order gives a different answer.
Trim then dedupe removes " apple" and "apple" as duplicates. Dedupe then trim keeps both, because at the moment of comparison they genuinely were different strings. Same two operations, same input, two different lists out.
So the order here is fixed, and this is it:
- Trim, so that lines are comparable before anything compares them.
- Remove blanks, because trimming turns lines that held only spaces into genuinely empty ones.
- Dedupe, now that identical-looking lines really are identical strings.
- Sort, on the set that survived rather than the set that arrived.
- Number, last.
Numbering last is the one that bites
Number first and sort second and you sort by the numbers you just added. They are text, so 10 comes before 2, and the result is a mess that looks completely deliberate: 1, 10, 11, 2, 3. Numbering last means the sort saw the content and the numbers come out in order.
Sorting is not alphabetical by default
A plain sort compares character codes rather than letters, and every capital has a lower code than every lower case letter. So the default gives you Zebra before apple, which nobody has ever wanted.
The same comparison walks character by character, which is why item10 lands before item2. It is the same reason photo1, photo2 and photo10 come out wrong in so much software.
This uses a locale collator with numeric ordering, which handles both: case-insensitive letter order, and runs of digits compared as numbers.
The invisible one: line endings
Text pasted from Windows carries a carriage return before every newline. Split on the newline alone and every line keeps a trailing character you cannot see, and then lines that look identical do not compare equal, dedupe leaves duplicates behind, and trim appears not to work.
Everything here normalises line endings before it does anything else. It is not tidiness, it is the difference between the tool working on pasted text and quietly not.
Building a list for somewhere else
The prefix and suffix turn a plain list into something another program will take. A quote as the prefix and a quote plus comma as the suffix gives you a quoted comma-separated list ready for a SQL INclause or a spreadsheet formula.
The affix is applied before numbering, so if you use both, the number sits outside the quotes where you want it.
Common questions
Why does removing duplicates leave duplicates behind?
Almost always trailing whitespace. A line ending in a space is a different string from the same line without one, and you cannot see the difference. It happens constantly in lists pasted out of a spreadsheet or an email. Trimming before deduplicating fixes it, which is why the trim runs first here and is on by default.
Why does sorting put capitals first?
Because a plain sort compares character codes rather than letters, and every capital has a lower code than every lower case letter, so "Zebra" sorts before "apple". This tool uses a locale collator instead, which sorts the way a person means, and also puts "item2" before "item10" rather than after it.
Why is item10 sorting before item2 elsewhere?
Because a character-by-character comparison sees "1" before "2" and stops there. It is the same reason files called photo1, photo2 and photo10 come out in the wrong order in a lot of software. Numeric collation compares runs of digits as numbers, which is what this tool does.
Does the order of operations matter?
A great deal, and it is why the order here is fixed rather than following the order you tick the boxes. Trim then dedupe removes " apple" and "apple" as duplicates; dedupe then trim keeps both. The pipeline runs trim, remove blanks, dedupe, sort, then number, and each step is in that position for a reason.
Why is numbering applied last?
Because numbering first and sorting second sorts by the numbers you just added, as text, so line 10 lands before line 2. Numbering last means the sort worked on the content and the numbers come out in order. It is a small thing that produces a mess which looks deliberate.
Is my text uploaded anywhere?
No. Everything runs in your browser on your own machine. There is no server involved, no account, and nothing is stored, so you can safely paste a list that has customer names or addresses in it.
Can I use this to build a SQL or CSV list?
Yes, that is what the prefix and suffix are for. Set the prefix to a quote and the suffix to a quote and a comma, and a plain list becomes a quoted, comma-separated one ready to drop into a query or a spreadsheet formula. The affix is applied before numbering, so the number sits outside the quotes.
Does the shuffle give the same result twice?
It gives the same result until you ask for another shuffle, which is deliberate: changing an unrelated option should not silently reorder your list. Press "shuffle again" for a new order.