Find Common Keys Of A Number Of Dictionaries

I have around 20000 dictionaries.I want find all the keys which are common in all the dictionary(that key need to be present in all 20000) dictionary.How can I achieve this.Current

Solution 1:

You could create the set of the keys in the first dictionary and then in a loop form the intersection with the remaining keys:

>>>d1 = {'a':1, 'b':2, 'c':3}>>>d2 = {'b':4, 'c':7,'d':5}>>>d3 = {'b':5, 'c':8,'e':6}>>>dicts = [d1,d2,d3]>>>common_keys = set(d1.keys())>>>for d in dicts[1:]:
    common_keys.intersection_update(set(d.keys()))

>>>common_keys
{'b', 'c'}

Solution 2:

I would use the Counter from the collections module For instance:

from collections import Counter
count = Counter()
for dictionary in listOfAllDictionaries:
    count = count+Counter(dictionary)
uniquelist = [keyforkeyin count if count[key] == 1]
commonlist = [keyforkeyin count if count[key] > 1]

Solution 3:

This might not be the most efficient way about it, but it uses only common, easy to understand operators and methods. Simply create a list of all the keys, and count the ones that show up as many times as you have dictionaries

>>>d1 = {1: 'a', 2: 'b'}>>>d2 = {3: 'c', 2: 'b'}>>>dictionaries = [d1, d2]>>>for d in dictionaries:
    keys += list(d.keys())

>>>keys
[1, 2, 2, 3]

>>>commonkeys = []>>>for k in keys:
    if k not in commonkeys and keys.count(k) == len(dictionaries):
        commonkeys += [k]

>>>commonkeys
[2]

voila, the common key among the dictionaries was 2.

Solution 4:

Pinching John Coleman's example dictionaries

d1 = {'a':1, 'b':2, 'c':3}
d2 = {'b':4, 'c':7,'d':5}
d3 = {'b':5, 'c':8,'e':6}
d = [d1, d2, d3]
d = iter(d)
reduce(set.intersection, d, set(next(d)))
set(['c', 'b'])

Solution 5:

A Pythonic way, not necessarily the most understandable. Neverthheless, here is the one liner:

>>>d1 = { 'a' : 1, 'b' : 1 }>>>d2 = { 'a' : 1, }>>>d3 = { 'a' : 1, 'c' : 1 }>>>set.intersection(*tuple(set(d.keys()) for d in [d1, d2, d3]))

set(['a'])

Unlike the solutions that are using Counters and Lists, this should be pretty efficient and low overhead going through so many dicts. This does build a large tuple, however, and for that reason, I would suspect that the more efficient code would be to loop through it.

It was mentioned that you might want "unique" keys as well, but I'm not sure what means. Does that mean, just not common, or does that mean, only in one dict? If it is the former, then just also keep a set that is the union, not the intersection, then take the difference of the two when you are done. Depending on the total number of unique keys we are talking about, I still think this is a relatively cheap operation.

Edit: To answer question below:

from collections import defaultdict

all_dicts = [d1, d2, d3]
common = set.intersection(*tuple(set(d.keys()) for d in all_dicts))

common_key_values = defaultdict(list)
for d in all_dicts:
    for key in common:
        common_key_values.append(d[key])