Remove Duplicates From List In Python

In this short guide, I’ll show you four different techniques to remove duplicates from list in python. So without further ado let’s get started.

1. By Using set()

set() is the builtin data type which on initialize automatically excludes the duplicate elements from it.

So in this code example, we first convert the original list to set data type removing its duplicates then convert the set back to the list.

#python remove duplicates from list
def remove_duplicates(original):
    # convert original list to set
    # set automatically removes duplicates
    # and then convert back set to list and return to caller
    set_ = set(original)
    removed = list(set_)
    return removed


original = [1,2,2,1,0,8,9,0]
removed = remove_duplicates(original)

print("Original : ",original)
print("Duplicates Removed :",removed)

Save and run the above code you should see the following output.

Output:

Output of the program to remove duplicates from list in python

2. By Using For-Loop

In the following code, we will use for loop to remove duplicates from the list. here is the algorithm :

  • We first define a new empty list.
  • Then we iterate through the original list of values.
  • Then on each iteration, we add the element to the new list if it does not exist.
  • Then we return the new list to the caller.
#program to remove duplicates from list in python
def remove_duplicates(original):
    result = [] # define an empty list

    for i in original: # iterate through original list
        if i not in result: # check if element already exists in result list
            result.append(i) # if not then add it
    return result # return list to caller


original = [1,2,2,1,0,8,9,0]
removed = remove_duplicates(original)

print("Original : ",original)
print("Duplicates Removed :",removed)

3. By Using Ordered Dict

In the following code snippet, we will use OrderedDict data type from the collection module to remove duplicates from list in python.

from collections import OrderedDict

def remove_duplicates(original):
    return list(OrderedDict.fromkeys(original))

original = [1,2,2,1,0,8,9,0]
removed = remove_duplicates(original)

print("Original : ",original)
print("Duplicates Removed :",removed)

That’s wrap for the tutorial on how to remove duplicates from list in python.

Also read about check if file exists in python

Leave a Reply