Remove Duplicate Lines

Duplicate line remover is an online text mechanic to find and remove duplicate lines of text. Lines can even be compared in a case insensitive way. Empty or blank lines can be deleted & whitespace can be trimmed. All the removed lines can be seen or checked along with their line numbers using this tool. You can also see a count of how many times a line appeared. Finally, you can sort the deduped lines in ascending or descending order.

Input: Paste text below

Settings

Background Information

We all use text files everyday. Be it a list of file paths, names of people or things to do. There is often a need to remove duplicate lines of text. This tool allows you to do just that. Read more at wikipedia.

Settings Explained
• 1. Case Sensitive

Use this to optionally perform a case-insensitive matching when trying to find duplicate lines. This is useful when you are comparing things like windows file paths where case does not matter.

The output stays the same as the original line.
Case Sensitive On

She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
Case Sensitive Off

She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
• 2. Case Conversion

Case is changed using one of the following styles before duplicate lines are removed

The output will be changed depending on the case conversion rule selected.
None

No case Change
Lower Case

lower case change
Upper Case

UPPER CASE CHANGE
Camel Case

camelCaseChange
Pascal Case

PascalCaseChange
Title Case

Title Case Change
Snake Case

snake-or-kebaberized-case-change
Sentence Case

Sentence case change
• 3. Remove Empty Lines

When turned on, empty lines are removed from the output.

Remove Empty Lines On

She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
Remove Empty Lines Off

She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog

She sells sea Shells on the sea shore
• 4. Trim Type

The following trim types are available

None

In this mode no whitespace is trimmed

She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
Start

In this mode only the starting whitespace is trimmed or removed

She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
End

In this mode only the ending whitespace is trimmed or removed

She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
Both

In this mode both the starting and the ending whitespace is trimmed or removed

She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
• 5. Show Removed Lines

If selected, removed lines are shown along with their corresponding line numbers

• 6. Show Occurrences

If selected, all unique lines along with a count of how many times they appeared is shown

• 7. Remove Double Space

If selected, contiguous or multiple spaces are replaced by a single space before removing the duplicates

• 8. Sort Type

This is the main setting as it governs the sorting logic/strategy. The following strategies are available:-

Alphabetical

This is the default sorting order which utilizes a case insensitive & culture neutral comparison between items. Numbers are treated as normal ASCII characters. For more information, checkout the wikipedia article on alphabetical order.

007, James Bond
09, John Doe
1, Jane Doe
Natural

This is similar to alphabetical sorting, but treats numbers differently. Consecutive numbers are grouped together and treated as a single character. For example, "1, Jane Doe" will appear before "007, James Bond". For more information, checkout the wikipedia article on natural sort order.

1, Jane Doe
007, James Bond
09, John Doe
Length

In this type of sorting, lines are sorted on the basis of their lengths in characters. So, you can make the smaller ones bubble to the top or use the descending sort order to make them sink to the bottom with the larger ones taking the top spots.

1, Jane Doe
09, John Doe
007, James Bond
Reverse

This is useful mainly if you want to reverse the order of the lines in the original input. Sort Order is not supported in this mode.

Original
09, John Doe
1, Jane Doe
007, James Bond
Sorted
007, James Bond
1, Jane Doe
09, John Doe
Random

This sort type randomizes the lines. Sort Order is not supported in this mode.

• 9. Sort Order

Some sort types support ascending & descending modes. Setting descending order reverses the order of the items while still applying the sorting logic.
Random & Reverse sort types do not support sort order.