Find & Remove Duplicates in List of Dictionaries in Python (3 Examples)

Hi! This tutorial will show you how to detect and eliminate duplicates in a list of dictionaries in the Python Programming language.

Here is an overview:

Let’s get into the Python code!

Create Demo List of Dictionaries

We will here create a demo Python list of dictionaries whose duplicates we will get and delete.

Therefore, in your preferred Python coding IDE, run the code below to create the demo list of dictionaries:

dictionaries = [
    {"name": "John", "age": 30},
    {"name": "Alice", "age": 25},
    {"name": "John", "age": 30},
    {"name": "Bob", "age": 35},
    {"name": "Alice", "age": 25}
]
 
print(dictionaries)
 
# [{'name': 'John', 'age': 30}, 
#  {'name': 'Alice', 'age': 25}, 
#  {'name': 'John', 'age': 30}, 
#  {'name': 'Bob', 'age': 35}, 
#  {'name': 'Alice', 'age': 25}]
 
print(type(dictionaries))
 
# <class 'list'>

With the demo list of dictionaries created, we will now examine three ways to detect and delete duplicates in the list.
 

Example 1: Detect & Eliminate Duplicates in List of Dictionaries Using for Loop & set() Function

In this first example, we will use a for loop and the set() function to get and remove duplicates in the list of dictionaries:

unique_dictionaries = []
 
unique_set = set()
 
for dictionary in dictionaries:
    dictionary_tuple = tuple(dictionary.items())
    if dictionary_tuple not in unique_set:
        unique_set.add(dictionary_tuple)
        unique_dictionaries.append(dictionary)
 
print(unique_dictionaries)
 
# [{'name': 'John', 'age': 30}, {'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 35}]
 
print(type(unique_dictionaries))
 
# <class 'list'>

This example uses a set to find and remove duplicates from the list of dictionaries.

It iterates through each dictionary in the input list using a for loop. For each dictionary, it converts the key-value pairs into a tuple using the items() method.

This tuple representation is used to check if it already exists in unique_set. If the tuple is not found in the set, it means the dictionary is unique and it is added to both the set and unique_dictionaries.

By utilizing the set’s property of storing unique elements only, duplicate dictionaries are effectively filtered out, and the resulting unique_dictionaries list contains only the unique dictionaries from the original list.
 

Example 2: Detect & Eliminate Duplicates in List of Dictionaries Using List & set Comprehension

In this next example, we will use list comprehension and set comprehension to get and eliminate duplicates in the list of dictionaries:

unique_dictionaries = [dict(t) for t in {tuple(sorted(d.items(), key=lambda x: (x[0] != 'name', x))) for d in dictionaries}]
 
print(unique_dictionaries)
 
# [{'name': 'Alice', 'age': 25}, {'name': 'John', 'age': 30}, {'name': 'Bob', 'age': 35}]
 
print(type(unique_dictionaries))
 
# <class 'list'>

Here, a set comprehension is used to create a set of tuples representing the dictionaries. The sorted() function is used to sort the items in each dictionary.

The key parameter is set to a lambda function that checks if the key is “name” (x[0] != 'name').

If it is not, it returns False, and if it is “name”, it returns True. This ensures that the dictionaries are sorted in such a way that the ones with the “name” key come first.

Additionally, within each group, the key-value pairs are sorted by their natural order (x). The resulting set only contains unique tuples representing unique dictionaries.

Finally, a list comprehension is used to convert each unique tuple back to a dictionary, resulting in unique_dictionaries that contains the dictionaries with the “name” key placed first, and duplicate dictionaries removed.

Example 3: Detect & Eliminate Duplicates in List of Dictionaries Using pandas

In this last example, we will use Python’s pandas library to detect and delete duplicates in the list of dictionaries.

First, install and import pandas:

# install pandas
pip install pandas
 
# import pandas
import pandas as pd

Now, we can use pandas to eliminate duplicates in the list like so:

df = pd.DataFrame(dictionaries)
 
df.drop_duplicates(inplace=True)
 
unique_dictionaries = df.to_dict('records')
 
print(unique_dictionaries)
 
# [{'name': 'John', 'age': 30}, {'name': 'Alice', 'age': 25}, {'name': 'Bob', 'age': 35}]
 
print(type(unique_dictionaries))
 
# <class 'list'>

In the above example, we first create a DataFrame using the input list of dictionaries.

Then, the drop_duplicates() method is applied to the DataFrame, which removes any duplicate rows based on all columns.

The inplace=True parameter ensures that the modifications are made directly to the DataFrame. After dropping the duplicates, the resulting DataFrame contains only unique rows.

Finally, the to_dict() method is used to convert the DataFrame back into a list of dictionaries, where each dictionary represents a unique row in the original data.

Thus, the resulting unique_dictionaries list contains only the unique dictionaries from the original list, with duplicates removed.
 

Video, Further Resources & Summary

Do you need more explanations on how to find and remove duplicates in a list of dictionaries in Python? Then you should have a look at the following YouTube video of the Statistics Globe YouTube channel.

In the video, we explain how to find and remove duplicates in a list of dictionaries in Python.

The YouTube video will be added soon.

Furthermore, I encourage you to check out other interesting Python list tutorials on Statistics Globe, starting with these ones:

• Difference Between List & Set in Python
• Check if List of Lists is Empty in Python (2 Examples)
• Convert List to Matrix & Vice-Versa in Python (Examples)
• Transpose 2D List in Python (3 Examples)
• Introduction to Python Programming

This post has shown, using three examples, how to find and remove duplicates in a list of dictionaries in Python. Your use case will determine which solution to adopt.

I hope you found this tutorial helpful! In case you have further questions, you may leave a comment below.

This page was created in collaboration with Ifeanyi Idiaye. You might check out Ifeanyi’s personal author page to read more about his academic background and the other articles he has written for the Statistics Globe website.