5 Methods to Remove Duplicate Items from Python Lists

On this tutorial you’ll find out how take away duplicate gadgets of Python lists.

When working with lists in Python, generally you simply must work with distinctive gadgets within the checklist – by eradicating the duplicates.

There are a number of other ways you are able to do this. On this tutorial, we’ll focus on 5 such strategies.

Fundamentals of Python Lists

Let’s begin our dialogue by reviewing the fundamentals of Python lists.

Python lists are mutable. So you may change them in place by including and eradicating parts from the checklist. As well as, Python lists are collections of parts that aren’t essentially distinctive.

So how do you retain solely the distinctive parts And to delete the duplicate or repeating parts?

Properly, you are able to do this in a number of methods. You possibly can create a brand new checklist that incorporates solely the distinctive parts within the unique checklist. Or you may select to vary the unique checklist and take away the duplicate gadgets.

We are going to be taught these intimately on this tutorial.

Strategies to take away duplicates from Python lists

Let’s take an actual world instance. Suppose you’re at your buddy’s birthday celebration.🎊🎉

Within the assortment of candies proven, you may see that there are some gadgets which might be repeated. You now need to take away these duplicate gadgets from the candies checklist.

Let’s make one sweets checklist of all gadgets within the picture above.

sweets = ["cupcake","candy","lollipop","cake","lollipop","cheesecake","candy","cupcake"]

Within the above sweets checklist, the gadgets ‘sweet’ and ‘cupcake’ are repeated twice. Let’s use this instance checklist to take away the duplicate gadgets.

Iterate over Python lists to take away duplicates

The best technique is to create a brand new checklist that incorporates every merchandise precisely as soon as.

Learn via the code cell under:

unique_sweets = []
for candy in sweets:
  if candy not in unique_sweets:
    unique_sweets.append(candy)

print(unique_sweets)

# Output
['cupcake', 'candy', 'lollipop', 'cake', 'cheesecake']

We initialize an empty checklist unique_sweets.
As you undergo the sweets checklistwe now have entry to every candy.
If candy shouldn’t be but current within the unique_sweets checklist, we add it to the top of unique_sweets checklist utilizing the .append() technique.

Suppose you come throughout a repeating merchandise, say the second prevalence of “sweet” within the sweets checklist. This isn’t added to the unique_sweets checklist because it already exists: candy not in unique_sweets evaluates to False for the second prevalence of ‘cupcake’ and ‘sweet’.

Subsequently, on this technique, every merchandise happens precisely as soon as within the unique_sweets checklist – with none repetition.

Use checklist comprehension to take away duplicates

You too can use checklist comprehension to view the unique_sweets checklist.

Wish to brush up on the fundamentals of checklist comprehension?

▶️ Watch the checklist comprehension tutorial in Python.

Let’s use the checklist comprehension expression: [output for item in iterable if condition is True] to succinctly rewrite the loop above.

unique_sweets = []
[unique_sweets.append(sweet) for sweet in sweets if sweet not in unique_sweets]
print(unique_sweets)

# Output
['cupcake', 'candy', 'lollipop', 'cake', 'cheesecake']

Even when you create a brand new checklist, you don’t populate the created checklist with values. It is because the output is the .append() operation on the unique_sweets checklist.

To take away duplicate gadgets from Python lists, you may as well use built-in checklist strategies, and we’ll cowl this within the subsequent part.

Use built-in checklist strategies to take away duplicates

You should utilize the Python checklist strategies .depend() And .take away() to take away duplicate gadgets.

– With the syntax checklist.depend(worth)the .depend() technique returns the variety of instances worth occurs in checklist. Thus, the quantity akin to repeating gadgets is larger than 1.

– checklist.delete(worth) removes the primary prevalence of a worth from the checklist.

Utilizing the above, we now have the next code.

for candy in sweets:
  # test if the depend of candy is > 1 (repeating merchandise)
  if sweets.depend(candy) > 1:
  # if True, take away the primary prevalence of candy
    sweets.take away(candy)

print(sweets)

# Output
['cake', 'lollipop', 'cheesecake', 'candy', 'cupcake']

For the reason that .take away() take away technique solely the primary prevalence of a worth can’t be used to take away gadgets that seem greater than twice.

If a specific entry is duplicated (occurring precisely twice), this technique removes the primary entry.
If a sure merchandise is repeated Ok instances, and after operating the above code, Ok-1 there’ll nonetheless be reruns.

However normally, after we say duplicates, we normally check with all repeats.

To deal with this case, you may modify the above loop to take away all iterations besides one. As a substitute of a if conditionally to test the amount of a specific merchandise you should use a whereas loop to take away duplicates repeatedly till the depend of every merchandise within the checklist is 1.

The checklist sweets now consists of 2 reps of “cupcake” and three reps of “sweet.”

sweets = ["cupcake","candy","lollipop","cake","lollipop","candy","cheesecake","candy","cupcake"]

You possibly can one whereas loop to take away repeats, as proven under. The whereas loop continues to run so long as the variety of candies in candies is larger than 1. If there is just one extra prevalence, then the situation candies.depend(candy) > 1 turns into False and the loop jumps to the subsequent merchandise.

for candy in sweets:
  # test if the depend of candy is > 1 (repeating merchandise)
  whereas(sweets.depend(candy) > 1):
  # repeatedly take away the primary prevalence of candy till one prevalence stays.
    sweets.take away(candy)

print(sweets)
# Output
['cake', 'lollipop', 'cheesecake', 'candy', 'cupcake']

However utilizing nested loops will not be very environment friendly, so that you would possibly need to think about using one of many different strategies mentioned when you’re working with giant lists.

Up to now we now have discovered the next:

Strategies to take away duplicate gadgets from Python lists – by creating new lists – containing solely distinctive gadgets
Constructed-in checklist strategies .depend() And .take away() to vary the prevailing checklist

There are some built-in information buildings in Python that require the values to be all distinctive, with no repetition. Subsequently, we are able to forged a Python checklist to one in all these information buildings to take away duplicates. After which convert them again to an inventory. We’ll discover ways to do that within the sections to return.

Forged Python checklist right into a set to take away duplicates

Python units are collections of parts which might be all distinctive. Subsequently, the variety of gadgets within the set (given by len(<set-obj>) is the same as the variety of distinctive parts current.

You possibly can forged any Python iterable right into a set utilizing the syntax: set(iterable).

Now let’s forged the checklist candies right into a set and see the output.

set(sweets)
# Output
{'cake', 'sweet', 'cheesecake', 'cupcake', 'lollipop'}

From the output within the code cell above, we see that every merchandise seems precisely as soon as and the duplicates have been eliminated.

Additionally observe that the order of things shouldn’t be essentially the identical as their order within the unique checklist of candies. It is because a Python set object shouldn’t be solely a group of distinctive parts, however a unordered assortment.

Now that we have eliminated the duplicates by casting the checklist right into a set, we are able to flip it again into an inventory, as proven under.

unique_sweets = checklist(set(sweets))
print(unique_sweets)

# Output
['cake', 'cheesecake', 'candy', 'cupcake', 'lollipop']

Use checklist gadgets as dictionary keys to take away duplicates

Python dictionary is a group of key-value pairs the place the keys uniquely establish the values.

You possibly can create a Python dictionary utilizing the .fromkeys() technique with the syntax: dict.fromkeys(keys, values). Right here, keys And values are iterables containing the keys and values of the dictionary respectively.

keys is a required parameter, and it may be any Python iterable that matches the keys of the dictionary.
values is a non-obligatory parameter. If you don’t specify the iterable values, the default worth of None has been used.

With out specifying the values, dict.fromkeys(sweets) returns a Python dictionary with the values set None – the default worth. The code cell under explains this.

dict.fromkeys(sweets)

# Output
{'cake': None,
 'sweet': None,
 'cheesecake': None,
 'cupcake': None,
 'lollipop': None}

As with the earlier part, we are able to flip the dictionary into an inventory once more, as proven under.

unique_sweets = checklist(dict.fromkeys(sweets))
print(unique_sweets)
# Output
['cupcake', 'candy', 'lollipop', 'cake', 'cheesecake']

From the above output, we are able to see that the duplicate gadgets have been faraway from the checklist sweets.

In abstract👩‍🏫

Here is a abstract of the varied strategies you should use to take away duplicates or repetitions from Python lists.

Use the Python checklist technique .add() so as to add non-repeating gadgets to a brand new checklist. The brand new checklist incorporates every merchandise within the unique checklist precisely as soon as and removes all repetitions. You too can do that utilizing checklist comprehension.
Use built-in .depend() And .to delete() strategies to take away gadgets that seem precisely twice. The identical may be positioned in a single whereas loop to take away all additional occasions.
Forged a Python checklist right into a set to maintain solely the distinctive parts.
Utilization dict.fromkeys(checklist) to take away any duplicates from the checklist, as there ought to be no repetition keys of the dictionary.

Then try Python tasks to apply and be taught. Or discover ways to discover the index of an merchandise in Python lists. Have enjoyable studying!