# Concurrency of long-running operations

**URL:** <https://forum.openrefine.org/t/concurrency-of-long-running-operations/1009>\
**Category:** Development & Design\
**Created:** [October 4, 2023, 6:26pm UTC](https://forum.openrefine.org/t/concurrency-of-long-running-operations/1009 "2023-10-04T18:26:07Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![antonin\_d](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.openrefine.org/antonin_d/32/32_2.png) [@antonin\_d](https://forum.openrefine.org/u/antonin_d)\
**Post date:** [December 3, 2023, 9:43pm UTC](https://forum.openrefine.org/t/concurrency-of-long-running-operations/1009/8 "2023-12-03T21:43:09Z")

</div>

Here is a quick demo of what the user experience could look like for row-wise concurrency:

Beyond the scenario of reconciliation followed by data extension presented in the video, I think something like this could be useful in a fairly wide range of scenarios. For instance, the URL-fetching operation is generally very slow if you choose a long wait time between requests, to be polite to the web server. Generally, the HTTP response is processed further, for instance via GREL/Python or reconciliation (see for instance [this tutorial](https://www.wikidata.org/wiki/Wikidata:Tools/OpenRefine/Editing/Tutorials/Working_with_APIs)). This feature would allow the user to experiment with those downstream processing steps without having to wait for the URL-fetching operation to complete.

After experimenting with it a little, it has become quite clear that the current restriction to row-wise operations is limiting the reach of the feature: records mode support would be really useful. For instance, the data extension operation always runs in records mode, because if the reconciliation service returns multiple values then a record structure is created. I would keep this task on the back burner for now, as I think it's worth validating the basic principle with users first, before investing more time on such a feature, which is not central to the reproducibility topic anyway (it is already an extension of column-based concurrency).

But still, I am quite enthusiastic about the potential effects such a feature can have. Intuitively, it should:

- make working with slow reconciliation services much less frustrating, because the time it takes for reconciliation to complete isn't such a big deal anymore. Similarly with scraping or working with restrictive APIs.

- reduce the need to artificially restrict long-running operations on subsets of the dataset to make them tractable. At the moment, this is something I see done very often. The problem is that once you have done many other operations following your original costly operation, there isn't really a good way to relax the original filter and run the whole workflow on a larger part of the dataset:

- it brings us closer to the ability of executing entire workflows in a streaming fashion, from the import stage to the export. This would be particularly interesting in a CLI environment, where you'd be able to be able to do something like this (with a purely fictional and non-vetted syntax): `cat source_data.csv | refine my_workflow.json > cleaned_data.tsv`  
assuming the file `my_workflow.json` encodes the import metadata, series of operations and export format. If all the operations are row/record wise, then the entire processing could be done in a streaming mode.

---

_[View the full topic](https://forum.openrefine.org/t/concurrency-of-long-running-operations/1009)._