# Record Linkage and Entity Resolution

**URL:** <https://forum.openrefine.org/t/record-linkage-and-entity-resolution/924>\
**Category:** Development & Design\
**Created:** [August 20, 2023, 9:44pm UTC](https://forum.openrefine.org/t/record-linkage-and-entity-resolution/924 "2023-08-20T21:44:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dqlearner](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.openrefine.org/dqlearner/32/232_2.png) [@dqlearner](https://forum.openrefine.org/u/dqlearner)\
**Post date:** [August 20, 2023, 9:44pm UTC](https://forum.openrefine.org/t/record-linkage-and-entity-resolution/924/1 "2023-08-20T21:44:11Z")

</div>

I noticed that Open Refine would be more complete as Data Management and Cleaning tool if we can implement functions to solve common data management/cleansing problem taking advantage of it's powerful clustering algorithm:  
-Entity Resolution or Deduplication.  
-Record Linkage between two datasets

---

<div class="post-metadata">

**Author:** ![olea](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.openrefine.org/olea/32/231_2.png) [@olea](https://forum.openrefine.org/u/olea)\
**Post date:** [August 23, 2023, 5:43pm UTC](https://forum.openrefine.org/t/record-linkage-and-entity-resolution/924/2 "2023-08-23T17:43:24Z")

</div>

> [@dqlearner](#):
>
> - Entity Resolution or Deduplication.

Interesting. Could you tell more about this?

---

<div class="post-metadata">

**Author:** ![dqlearner](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.openrefine.org/dqlearner/32/232_2.png) [@dqlearner](https://forum.openrefine.org/u/dqlearner)\
**Post date:** [November 16, 2023, 11:55pm UTC](https://forum.openrefine.org/t/record-linkage-and-entity-resolution/924/3 "2023-11-16T23:55:16Z")

</div>

Entity Resolution or Deduplication: This means to detect and group (clustering) duplicate records for example in a customer table (written in different forms). Current clustering method works great when we want to compare a single column, but not multiple columns (Name, Address, City etc).  
Also the current method does not allow to export the clustering result. It will be useful if it can group the clusters and give a number for each of them and be able to export as csv for managing in excel later.

Record Linkage between two data sets: Means beasically the concept of re-conciliación of data. But what I really mean is to to match similar records between two data sets.  
Customer table A, Customer table B and get the mathcing results with matching score.

These two implementation/function would give a great boost for Data Management use case.

---

<div class="post-metadata">

**Author:** ![Martin](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.openrefine.org/martin/32/7_2.png) [@Martin](https://forum.openrefine.org/u/Martin)\
**Post date:** [November 21, 2023, 8:09pm UTC](https://forum.openrefine.org/t/record-linkage-and-entity-resolution/924/4 "2023-11-21T20:09:25Z")

</div>

@dqlearner, thanks for the details.

**Entity Resolution or Deduplication**  
You can export the cluster results in JSON for a separate analysis (see my screenshot below).

 ![option to export OpenRefine cluster](https://europe1.discourse-cdn.com/flex017/uploads/openrefine/original/1X/79c72babb8ee927faf7ba28b592debdaefeb1fef.png) ![preview of the exported cluster in json format in OpenRefine](https://europe1.discourse-cdn.com/flex017/uploads/openrefine/original/1X/b446ac1b7f26c1d0fd0a379b752e17d8b30c04a7.png)

It could be interesting to see how clustering on multiple columns could work. The current workaround is to concatenate the columns, but we are losing some granularity with this approach. I think this is something worth exploring further.

Regarding **Record Linkage** , OpenRefine support of [reconciliation services](https://openrefine.org/docs/manual/reconciling) should help a lot. If you want to reconcile against your own dataset you can start by looking at csv base reconciliation service:

- [GitHub - gitonthescene/csv-reconcile: A reconciliation service for OpenRefine serving data from a given CSV file.](https://github.com/gitonthescene/csv-reconcile)
- [Reconcile-csv - join dirty data - Open Knowledge Foundation Labs](https://okfnlabs.org/reconcile-csv/)