Remove Duplicate Lines

Duplicate line remover is an online text mechanic to find and remove duplicate lines of text. Lines can even be compared in a case insensitive way. Empty or blank lines can be deleted & whitespace can be trimmed. All the removed lines can be seen or checked along with their line numbers using this tool. You can also see a count of how many times a line appeared. Finally, you can sort the deduped lines in ascending or descending order.

All processing happens on the client-side, locally in your browser. No data is uploaded.

Paste text below

Deduped Lines

Settings

How the input is converted. Each is explained below.

Case Sensitive

If selected, case sensitive matching is used

Remove Empty Lines

If selected, empty lines are removed

Show Removed Lines

If selected, removed lines are shown along with their corresponding line numbers

Show Occurrences

If selected, all unique lines along with a count of how many times they appeared is shown

Remove Double Space

If selected, contiguous or multiple spaces are replaced by a single space before removing the duplicates

Case Conversion

Case is changed using one of the following styles before duplicate lines are removed

Trim Type

Select the trim type to remove whitespace (space and tab) in output

Sort Type

The type of sort you want to perform

Sort Order

The order of the sort you want to perform

History

2019-01-24Option to show number of times each line occurred
2019-01-23Sorting support
2018-07-29Trim type, Double Space Removal & Case Conversion setting
2017-12-30Show a count of total lines removed
2017-09-16Tool Launched

Settings explained

Case Sensitive: Use this to optionally perform a case-insensitive matching when trying to find duplicate lines. This is useful when you are comparing things like windows file paths where case does not matter. The output stays the same as the original line.
Case Conversion: Case is changed using one of the following styles before duplicate lines are removed. The output will be changed depending on the case conversion rule selected.
Remove Empty Lines: When turned on, empty lines are removed from the output.
Trim Type: None: no whitespace is trimmed. Start: only the starting whitespace is removed. End: only the ending whitespace is removed. Both: both the starting and the ending whitespace is removed.

Background

We all use text files everyday. Be it a list of file paths, names of people or things to do. There is often a need to remove duplicate lines of text. This tool allows you to do just that. Read more at wikipedia.