1. Home
  2. Tools
  3. Text
  4. Line
  5. Duplicate Line Remover
Share

Remove Duplicate Lines

Updated: Jan 24, 2019
TEXT   DEDUPLICATION   DATA CLEANSING  
Duplicate line remover is an online text mechanic to find and remove duplicate lines of text. Lines can even be compared in a case insensitive way. Empty or blank lines can be deleted & whitespace can be trimmed. All the removed lines can be seen or checked along with their line numbers using this tool. You can also see a count of how many times a line appeared. Finally, you can sort the deduped lines in ascending or descending order.


Input: Paste text below



Settings





Background Information

We all use text files everyday. Be it a list of file paths, names of people or things to do. There is often a need to remove duplicate lines of text. This tool allows you to do just that. Read more at wikipedia.

Settings Explained

Case Sensitive

Use this to optionally perform a case-insensitive matching when trying to find duplicate lines. This is useful when you are comparing things like windows file paths where case does not matter.

The output stays the same as the original line.
Case Sensitive On
She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
Case Sensitive Off
She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
Case Conversion

Case is changed using one of the following styles before duplicate lines are removed

The output will be changed depending on the case conversion rule selected.
None
No case Change
Lower Case
lower case change
Upper Case
UPPER CASE CHANGE
Camel Case
camelCaseChange
Pascal Case
PascalCaseChange
Title Case
Title Case Change
Snake Case
snake-or-kebaberized-case-change
Sentence Case
Sentence case change
Remove Empty Lines
When turned on, empty lines are removed from the output.
Remove Empty Lines On
She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
Remove Empty Lines Off
She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog

She sells sea Shells on the sea shore
Trim Type
The following trim types are available
None
In this mode no whitespace is trimmed
She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
   She sells sea Shells on the sea shore
Start
In this mode only the starting whitespace is trimmed or removed
She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
End
In this mode only the ending whitespace is trimmed or removed
She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog   
   She sells sea Shells on the sea shore
Both
In this mode both the starting and the ending whitespace is trimmed or removed
She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
Show Removed Lines
If selected, removed lines are shown along with their corresponding line numbers
Show Occurrences
If selected, all unique lines along with a count of how many times they appeared is shown
Remove Double Space
If selected, contiguous or multiple spaces are replaced by a single space before removing the duplicates
Sort Type
This is the main setting as it governs the sorting logic/strategy. The following strategies are available:-
Alphabetical
This is the default sorting order which utilizes a case insensitive & culture neutral comparison between items. Numbers are treated as normal ASCII characters. For more information, checkout the wikipedia article on alphabetical order.
007, James Bond
09, John Doe
1, Jane Doe
Natural
This is similar to alphabetical sorting, but treats numbers differently. Consecutive numbers are grouped together and treated as a single character. For example, "1, Jane Doe" will appear before "007, James Bond". For more information, checkout the wikipedia article on natural sort order.
1, Jane Doe
007, James Bond
09, John Doe
Length
In this type of sorting, lines are sorted on the basis of their lengths in characters. So, you can make the smaller ones bubble to the top or use the descending sort order to make them sink to the bottom with the larger ones taking the top spots.
1, Jane Doe
09, John Doe
007, James Bond
Reverse
This is useful mainly if you want to reverse the order of the lines in the original input. Sort Order is not supported in this mode.
Original
09, John Doe
1, Jane Doe
007, James Bond
Sorted
007, James Bond
1, Jane Doe
09, John Doe
Random
This sort type randomizes the lines. Sort Order is not supported in this mode.
Sort Order
Some sort types support ascending & descending modes. Setting descending order reverses the order of the items while still applying the sorting logic.
Random & Reverse sort types do not support sort order.
 

Comments


History

Jan 24, 2019
Option to show number of times each line occurred
Jan 23, 2019
Sorting support
Jul 29, 2018
Trim type, Double Space Removal & Case Conversion setting
Dec 30, 2017
Show a count of total lines removed

Sep 16, 2017
Tool Launched
Created: Sep 16, 2017
Online Tool Designed For: Windows, OS X, Android, iOS, Linux
➤