Apply operation history from json file not working

Support and Helpdesk
21

However given this is how OpenRefine has operated for as long as I’ve used it, and I’ve found it occasionally useful (as others have clearly based on this thread), I think it’s reasonable to ask for someway to recreate the previous behaviour - whether that’s by an ‘override’ for particular errors in the Apply history, or by some other mechanism.

That sounds like the option I mentioned:

We could probably introduce an option to treat errors as warnings, but removing the error checking altogether and going back to the previous situation seems like a bad idea to me.

with the exception of "recreate previous behavior," when that previous behavior is to silently drop errors on the floor without notifying the user. I think anyone who values their data wants us to continue to improve our error handling and reporting.

Tom

22

I have bundled a sample json operation history and a sample .csv with relevant data in to a zip file, attached here.

OpenRefine-ApplyOpHist-BugReport-2025-10-16.zip (39.5 KB)

Thanks for the data to easily reproduce the problem.

The json operation works fine in OpenRefine 3.8, but runs into the validation problem in 3.9 due to missing columns in the csv data.

Actually, when I run this pair of files with OpenRefine 3.8, I get dozens of errors, but they are silently swallowed (actually, they're dumped on the console, but unless you start the Refine server from the command line, you'll never see them). Note that this isn't limited to missing columns, (almost?) NO errors are being reported to the user. While this may be fine for your use case (and Julia's) where you are confident in your post-export QC measures, for the vast majority of users' use cases, silently ignoring errors is going to be a real problem.

We can probably change these errors to warnings without too much difficulty, but I'll flag that, as we continue to improve error reporting, you'll likely to continue to uncover hidden problems like this one which have been masked by bugs in the past.

I'm curious where this Undo History was extracted from. Was there originally an uber spreadsheet that contained all these columns before subsets were created?

When the 3.10 beta is available (or now using one of the snapshot builds), I'd encourage you to check out the column mapping functionality, because I suspect it was built with a different world view than the one you operate under (again, I wasn't involved).

Lastly, I'll say that this probably should have received a prominent mention in the 3.9 release notes. I'll update them to highlight the compatibility issue.

Tom

23

Hello, all. I’m just chiming in to say that I have workflows that are similar to the ones shared above by @timothy-mendenhall. I apply project history with the knowledge that not all of the steps will be relevant for my particular project. Having that functionality is also essential for my work. I would be okay with having to ‘confirm’ that I would like to apply a formula that does not actually relate to the data at hand.

24


| |

  • | - |

I can write up an issue to track this.

I would encourage users who run into problematic behavior with new releases to create their own issue(s) here:
https://github.com/OpenRefine/OpenRefine/issues/new
There's no reason this responsibility needs to fall solely on Rory's, or other developers', shoulders.

This forum is great for seeking peer support, asking questions, and getting clarifications, but it's not a substitute for the bug tracking system. There's too much danger of things getting lost in freeform discussions.

Tom

25

Ah, I just wrote one up and your response came up just as I was posting it to this channel. The issue is here: Allow users to partially apply histories · Issue #7500 · OpenRefine/OpenRefine · GitHub

I second the encouragement to create issues and will go that route in the future.

26

Thank you @Rory ! The bug report looks good.

1 Like