How to Remove Duplicates From List in Python?

Summary: In this tutorial, we will learn different ways to remove duplicates from a list data structure in Python.

Unlike, the set data structure, the list data structure in Python allows to have duplicate values.

But, there may come situations, that we need to have only unique elements in the list. In such cases, we can easily remove duplicate values from the pre-populated list by using any of the following methods.

Method 1: Manually Iterating through the List

old_list = [5, 7, 8, 8, 5, 1]
new_list = []

for element in old_list:
    if element not in new_list:
        new_list.append(element)
        
print("Old List: ",old_list)
print("New List: ",new_list)

Output:

Old List: [5, 7, 8, 8, 5, 1]
New List: [5, 7, 8, 1]

In this method, we traverse the old list using a for loop and fill the new list with only unique elements.

We fill the new list with those elements which are not already present in the list.

This is a naive method and is not so efficient. There are better ways to remove duplicates from the list in Python.

Method 2: Using Set()

old_list = ['a', 'b', 'c', 'a', 'b']

new_list = list(set(old_list))
        
print("Old List: ",old_list)
print("New List: ",new_list)

Output:

Old List: [‘a’, ‘b’, ‘c’, ‘a’, ‘b’]
New List: [‘c’, ‘a’, ‘b’]

In this method, using the set() constructor we first construct a set from the old list and then convert it back to the list using the list() constructor.

By doing so, the duplicates are removed. This happens because the set data structure in Python doesn’t allow duplicate, thus auto removes them.

This is the easiest method to remove duplicates from a Python list.

The only drawback of this method is that the order of elements is not preserved.

Method 3: Using collections.OrderedDict.fromkeys()

from collections import OrderedDict

old_list = [7, 8, 5, 5, 3, 7]

new_list = list(OrderedDict.fromkeys(old_list))
        
print("Old List: ",old_list)
print("New List: ",new_list)

Output:

Old List: [7, 8, 5, 5, 3, 7]
New List: [7, 8, 5, 3]

In this method, we first convert the list into a dictionary using OrderedDict.fromkeys() method and then we convert it back to list using list() constructor method.

The OrderedDict.fromkeys() method eliminates the duplicates from the list because the dictionary in Python cannot have duplicate keys.

This is the fastest method in python to remove duplicates values from a list.

In this tutorial, we learned how to remove duplicates from a list using the python programming language.