Deduplication between two different digital solutions is not overly complex. At its most basic, it is a comparison. Is the data in this box, the same as the data in this box? Yes or No.

And yet, deduplication is a complex endeavour due to humans and the organisations they are part of. The traditional way to do deduplication is to get everyone to use the same digital solution. This does make the task easier, but it certainly increases the agnst. As digital transformation has taken root in humanitarian organisations, each organisation has its own data management solution.  Therefore, each organisation wants to use the technology it knows, trusts, and has capacity in.  It doesn’t want to be told to use another system. And in the era of ever improving data protection regulations, organisations also don’t want to lose control of the data they collect.

But even when we still keep organisations at the centre, but encourage each organisation to use their preferred digital solution of choice, there are still challenges:

• The need to have the same conversations hundreds of times.  And this continues on months after an agreement is finalised.  It is impossible to involve everyone in the discussions, therefore repeat conversations are the only way forward.
• Scope creep is one of the demons to battle.  Being clear on purpose is the only way to win the battle.  Early on be clear on a few things:
  • The purpose is deduplication of individuals at a country level (nothing else)
  • Collect data digitally
  • Share as little data as possible and
  • Get the ‘system’ to flag potential duplicates, but humans verify them ‘offline’
• Beyond the data sharing agreement, you need an agreed proceess for verifying and resolving potential duplicates
• And lastly, building digital literacy is always required

But what happens to the tensions between organisations (and staff), if instead of keeping organisations at the centre, we put the people we seek to help at the centre? What happens then?

We’re starting a project to find out. And there will be significant challenges to resolve and it might not work. But it is time to try. If you are interested in being involved or hearing more, please let me know.

Photo by Victor