I wanted to share something I've been working on, and ask for feedback from people who know OpenRefine far better than I do.
OxRefine is a reimplementation of OpenRefine in Rust. [why you started, e.g. the data size you were working with, in your own words]. It isn't meant to replace OpenRefine or to compete with this project, and it wouldn't exist without the years of work that went into it. I've tried hard to follow OpenRefine's behaviour rather than invent my own.
opens OpenRefine projects with their undo history, and writes projects that OpenRefine opens
text, numeric, timeline and custom facets, records mode
key collision clustering with every keyer, nearest neighbour with Levenshtein and PPM
reconciliation against Wikidata and other W3C Reconciliation services
most exports, including Excel, ODS, SQL and templating
How I check it: every case is compared with output from a real OpenRefine 3.10.1. Right now 2,122 of 2,136 checks match. The 14 that don't are listed here: Differences and what's missing - OxRefine
What's missing: RDF, MARC and wikitext import, the scatterplot facet, Jython and Clojure expressions, the database, Wikibase and Google extensions, and translations.
I'd really appreciate it if anyone tried it on a project or recipe of their own and told me where the results differ. And if there are behaviours you know are easy to get wrong, I'd love to hear about them.
This is a wonderful project and I really appreciate your work.
One thing that I like about OpenRefine is being able to share my project with coworkers, using simply a browser on port 3333. Is it possibile with OxRefine? Can I start a project on a Linux machine and continue on another machine simply using the browser?
I see in the road map that you are going to to implement the interface to databases. Could it be possible to create a sibling table to share a "workbed" with the SQL tools without exporting and importing table contents back again? This is a compromise between operating on the original table (bad!) and working with files out of the database.
Cool. I presume that you had a coding agent/harness write this for you. Are there any details available on what agent/harness, what models, what prompting strategy, etc?
Can you say more about how legacy extensions will work with this, if at all?
What is the long term goal, if not to replace or compete?
To present myself, I am so far just a user, but I am looking into attracting funding in order to help extending openRefine in some respects. As my job
I find that a Rust version does not need to be a competition. Building plugins for JAVA is probably easier. And openRefine probably does not want every plugin to be in two languages, right? But what would be awfully useful for people doing complex mappings would be to have an openRefine version that does not have to include the whole JAVA machinery to work.
Imagine data steward Alice interactively building a data migration bit for a pipeline maintained by data analyst Bob. Maybe Bob wants this to be really fast and to be part of his mixed Python+C++ pipeline. Currently, the way to go would be using the API, meaning to maintain a JAVA on the machine running the pipeline. In this case, a reliable Rust version would be really handy and would be a real addition to what openRefine offers.
Hi Wolfgang. Welcome to the project. What is your relationship to ilyaux and the 0xRefine project?
I would hope that folks with funding available would consider funding improvements in the core project. I am happy to discuss this with anyone who is interested. I have a prototype of a column oriented implementation which shows many times improvements in memory and processing efficiency while maintaining 100% compatibility, but work on it is currently stuck behind addressing some reported security vulnerabilities.
To the first approximation, supporting two implementations is going to be twice the work with the added burden of ensuring compatibility between the two. The reason that OpenRefine doesn't have a supported API and we recommend using the Undo histories only on the same file or an identical file is that there has been almost no investment in testing for these things, so the first thing I'd address with additional funding on the automation front is closing the testing gap so that we have a supported API and can support reusable recipes with proper error checking & reporting.
I'm happy to work with folks who have automation needs to address them. Please get in touch.
I do not have funding. For more than one year I am trying to build a proposal where together with other bioinformatics data managers we try to extend OpenRefine (sorry for mis-casing) in a way that benefits the existing OpenRefine community, as well as becoming more useful for scientists.
There is no funding as of yet, but there is (I think) a realistic perspective to get some funding in a German research infrastructure funding scheme this way. And (without having liaised with the other two partner who are likely to build this proposal for German research infrastructure funding)
We would be interested in contributing to the main branch of OpenRefine.
And of course it would be crucial that the ideas are welcome.
I am optimistic that under these conditions the funders would also fund contribution to the core, as the benefits of contributing to OpenRefine would be clear.
The basic idea would be to combine the strengths of three currently separate tools/concepts around workbooks, roll them all into OpenRefine and make it possible to use OpenRefine as a backend. There will be also extensions, e.g. for the scientists I know it is problematic that OpenRefine operates on tables as opposed to stacks of tables. One of the tools uses hidden sheets etc... Some other things are already on your roadmap, like e.g. being able to handle large tables.
I am long enough in this “business” to know that a this setting (new person comes into existing project, new person has very concrete plans) has plenty of opportunities for conflict, so I am very motivated to get things sorted in a way that people who have been contributing to OpenRefine in the past hope that the project gets funding.
How would you best go about it? How about a zoom call, who should be in that call in your opinion? Or first a 1:1 zoom and then a zoom with everyone interested?
Oh, and yes, I did not want to highjack the discussion about OxRefine, just saying that there may be a worthy use case. But I also see your point: Better put the finite resources into the main branch.
Use the Linux machine’s address on port 3333, together with the access token printed at startup. Projects and their undo history stay on the server, so you can reopen the same project from another computer.
There are some limitations today:
all browsers share one active project/session, there are no separate user permissions, and changes are not automatically synchronized between browser windows. For now, I would recommend one active editor at a time. My aim as OxRefine approaches 1.0 is to fully support the browser-based workflow you describe, with a smooth handover between computers and coworkers.
Your suggestion about databases is interesting too. A separate working table alongside the original could give OxRefine and SQL tools a useful shared workspace while preserving the source data. I’ll investigate this in more detail, particularly how edits made through SQL would interact with OxRefine’s undo history and how changes could be written back safely. That deserves some careful design, and I appreciate you bringing up the use case.
Yes, I’ve been using Claude Fable throughout development. It wrote most of the user interface and contributes to implementation elsewhere in the project. It also helps with GitHub and commit workflows, developing tests, and working through architectural decisions. I set the goals and priorities and guide the work through individual features, problems, and revisions.
An important part of that workflow is checking behavior against fixtures generated from a real OpenRefine 3.10.1 instance. Those comparisons provide a concrete reference for the implementation, and the remaining differences are documented.
The project started as a personal experiment: could I make these data-cleaning workflows faster?
The results on the workflows I measured encouraged me to keep developing it.
My goal now is to bring OxRefine to a stable release that people can comfortably use for everyday work: fast processing, reasonable memory use, reliable project persistence, and clear errors when something goes wrong. Compatibility with OpenRefine projects and recipes is a central part of that goal, so people can try it with work they already have.
@wolfgang example is also a useful application: preparing a cleanup interactively and then running it efficiently inside a pipeline without requiring a Java runtime.
Existing OpenRefine extensions do not currently run in OxRefine. They depend on OpenRefine’s Java classes, Butterfly modules, and often its client-side interface hooks. Compatibility with project archives and recipes does not provide compatibility with those extension APIs.
There are two separate questions here: implementing equivalent functionality in OxRefine, and running existing extensions unchanged. Reimplementing particular features looks technically feasible, but each would need its own implementation, tests, and ongoing maintenance. I have not established a practical, tested approach to running arbitrary OpenRefine extensions unchanged.