Deduplication: The Unique Human Problem

by | Feb 21, 2020 | Identity |

Secret codes and symbols have been part of communities of people for millennia. They are still in wide use today. They are signals we belong to the same club.

Data sharing comes with trade offs between privacy, security, and utility. And in the humanitarian space, privacy almost always loses out. However, perhaps the idea of a secret (somewhat) symbol might help us.

When organisations work in a consortium, there is often a desire to share data to improve efficiency. There is also a power struggle about which organisation ‘holds’ the master list, the source of truth. We tend to also pass around sensitive data about individuals like kids trading cards.

Alternatively, we, as a group, could agree on a method of using attributes of a person to create a unique code. This unique code could be the only thing exposed to the other agencies to check against for duplicates. This code would be a like a community code or a community symbol. It would communicate the person is part of the community, while also communicating a few basic things to the other community members who can ‘read’ it.

When working in a consortium, we are working in a specific location with a specific people group. We are not working with every person on the planet. Therefore our deduplication or unique human problem is only for the people with whom we work. The secret symbol (effectively a hash) might allow us to deduplicate without creating a single, centralised list of people.

Photo by Noah Näf


Submit a Comment

Your email address will not be published. Required fields are marked *