Remove Duplicate Lines

Jan 24, 2019
0 Comments
Duplicate line remover is an online text mechanic to find and remove duplicate lines of text. Lines can even be compared in a case insensitive way. Empty or blank lines can be deleted & whitespace can be trimmed. All the removed lines can be seen or checked along with their line numbers using this tool. You can also see a count of how many times a line appeared. Finally, you can sort the deduped lines in ascending or descending order.


Input: Paste text below

Settings

We all use text files everyday. Be it a list of file paths, names of people or things to do. There is often a need to remove duplicate lines of text. This tool allows you to do just that. Read more at wikipedia.

Settings Explained
  • 1. Case Sensitive

    Use this to optionally perform a case-insensitive matching when trying to find duplicate lines. This is useful when you are comparing things like windows file paths where case does not matter.

    The output stays the same as the original line.
    Case Sensitive On

    She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
    Case Sensitive Off

    She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
  • 2. Case Conversion

    Case is changed using one of the following styles before duplicate lines are removed

    The output will be changed depending on the case conversion rule selected.
    None

    No case Change
    Lower Case

    lower case change
    Upper Case

    UPPER CASE CHANGE
    Camel Case

    camelCaseChange
    Pascal Case

    PascalCaseChange
    Title Case

    Title Case Change
    Snake Case

    snake-or-kebaberized-case-change
    Sentence Case

    Sentence case change
  • 3. Remove Empty Lines

    When turned on, empty lines are removed from the output.

    Remove Empty Lines On

    She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
    Remove Empty Lines Off

    She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog

She sells sea Shells on the sea shore
  • 4. Trim Type

    The following trim types are available

    None

    In this mode no whitespace is trimmed

    She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
   She sells sea Shells on the sea shore
    Start

    In this mode only the starting whitespace is trimmed or removed

    She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
    End

    In this mode only the ending whitespace is trimmed or removed

    She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog   
   She sells sea Shells on the sea shore
    Both

    In this mode both the starting and the ending whitespace is trimmed or removed

    She sells sea shells on the sea shore
Quick brown fox jumps over the lazy dog
She sells sea Shells on the sea shore
  • 5. Show Removed Lines

    If selected, removed lines are shown along with their corresponding line numbers

  • 6. Show Occurrences

    If selected, all unique lines along with a count of how many times they appeared is shown

  • 7. Remove Double Space

    If selected, contiguous or multiple spaces are replaced by a single space before removing the duplicates

  • 8. Sort Type

    This is the main setting as it governs the sorting logic/strategy. The following strategies are available:-

    Alphabetical

    This is the default sorting order which utilizes a case insensitive & culture neutral comparison between items. Numbers are treated as normal ASCII characters. For more information, checkout the wikipedia article on alphabetical order.

    007, James Bond
09, John Doe
1, Jane Doe
    Natural

    This is similar to alphabetical sorting, but treats numbers differently. Consecutive numbers are grouped together and treated as a single character. For example, "1, Jane Doe" will appear before "007, James Bond". For more information, checkout the wikipedia article on natural sort order.

    1, Jane Doe
007, James Bond
09, John Doe
    Length

    In this type of sorting, lines are sorted on the basis of their lengths in characters. So, you can make the smaller ones bubble to the top or use the descending sort order to make them sink to the bottom with the larger ones taking the top spots.

    1, Jane Doe
09, John Doe
007, James Bond
    Reverse

    This is useful mainly if you want to reverse the order of the lines in the original input. Sort Order is not supported in this mode.

    Original
    09, John Doe
1, Jane Doe
007, James Bond
    Sorted
    007, James Bond
1, Jane Doe
09, John Doe
    Random

    This sort type randomizes the lines. Sort Order is not supported in this mode.

  • 9. Sort Order

    Some sort types support ascending & descending modes. Setting descending order reverses the order of the items while still applying the sorting logic.
    Random & Reverse sort types do not support sort order.

Comments 0

History
Jan 24, 2019
Option to show number of times each line occurred
Jan 23, 2019
Sorting support
Jul 29, 2018
Trim type, Double Space Removal & Case Conversion setting
Dec 30, 2017
Show a count of total lines removed

Sep 16, 2017
Tool Launched