I've been working on this project with almost a quarter million records for months, and all was fine, but after I had to shut it down when it got stuck doing a parseJson operation on all records, it no longer wants to load. Whenever I try to open the project it just hangs in the loading screen forever:
I restarted OpenRefine several times and even my PC, but to no avail. Using Version 3.7.4 [05e9de9] on Windows 10...
I took a look at where the projects are stored, but the project file is not marked as ".corrupted", yet it still won't load.
What can I do? I tried importing some of the zip files from the history folder but those are not accepted as being in the right format (gzip). Is there anyway to restore the project?
Zipped project file can be downloaded here: WeTransfer - Send Large Files & Share Photos Online - Up to 2GB Free
Sorry to hear that you're having problems!
What can I do? I tried importing some of the zip files from the history folder but those are not accepted as being in the right format (gzip). Is there anyway to restore the project?
The Project Import expects a gzipped file in the format created by the Project Export, which is different than the files that you found.
The first thing to do is make a backup of your data directory (if you've been working on this for months, hopefully you have LOTS of backups).
Next, try starting the OpenRefine server from a terminal/shell so that you can see any error messages that it logs. It might also be worth checking the Javascript console of your browser to see if any errors are logged there, although the server messages are likely to be more informative. Let us know what you find and we can go from there.
Tom
Thanks for replying and lending a helping hand!
The normal console window that always appears when running OpenRefine only gives me the following:
17:46:28.722 [ refine] POST /command/core/load-language (22234ms)
17:46:28.731 [ refine] GET /command/core/get-preference (9ms)
17:46:28.731 [ refine] GET /command/core/get-preference (0ms)
17:46:28.736 [ refine] POST /command/core/load-language (5ms)
17:46:28.741 [ refine] POST /command/core/load-language (5ms)
17:46:28.763 [ refine] POST /command/core/load-language (22ms)
17:46:28.767 [ refine] POST /command/core/load-language (4ms)
17:46:28.816 [ refine] GET /command/core/get-project-metadata (49ms)
So it hangs on trying to get the project metadata, but not getting any error messages otherwise.
I will try to do as you write and try to run it directly from a terminal window etc and keep an eye on the JS console.
So this is what happens when running OR from a terminal window:
c:\Applications\OpenRefine\openrefine-3.7.4>openrefine.exe
17:51:11.842 [ refine_server] Starting Server bound to '127.0.0.1:3333' (0ms)
17:51:11.899 [ refine_server] Initializing context: '/' from 'c:\Applications\OpenRefine\openrefine-3.7.4\webapp' (57ms)
17:51:18.465 [ refine] Starting OpenRefine 3.7.4 [05e9de9]... (6566ms)
17:51:18.465 [ refine] initializing FileProjectManager with dir (0ms)
17:51:18.465 [ refine] C:\Users\vsb582\AppData\Roaming\OpenRefine (0ms)
17:51:19.623 [ FileProjectManager] Failed to recover project in directory 1794569537280.project (1158ms)
17:51:19.625 [ FileProjectManager] Failed to recover project in directory 1830783842220.project (2ms)
17:51:19.629 [ FileProjectManager] Failed to recover project in directory 2358457692672.project (4ms)
17:51:43.758 [ refine] POST /command/core/load-language (24129ms)
17:51:43.786 [ refine] GET /command/core/get-preference (28ms)
17:51:43.836 [ refine] POST /command/core/load-language (50ms)
17:51:43.844 [ refine] POST /command/core/load-language (8ms)
17:51:43.852 [ refine] POST /command/core/load-language (8ms)
17:51:43.933 [ refine] GET /command/core/get-importing-configuration (81ms)
17:51:43.963 [ refine] GET /command/core/get-all-project-tags (30ms)
17:51:43.997 [ refine] GET /command/core/get-all-project-metadata (34ms)
17:51:44.141 [ refine] GET /command/database/saved-connection (144ms)
17:51:44.318 [ refine] GET /command/core/get-csrf-token (177ms)
17:51:44.341 [ refine] GET /command/core/get-languages (23ms)
17:51:44.460 [ refine] GET /command/core/get-version (119ms)
17:52:05.647 [ refine] POST /command/core/load-language (21187ms)
17:52:05.661 [ refine] GET /command/core/get-preference (14ms)
17:52:05.664 [ refine] GET /command/core/get-preference (3ms)
17:52:05.675 [ refine] POST /command/core/load-language (11ms)
17:52:05.689 [ refine] POST /command/core/load-language (14ms)
17:52:05.698 [ refine] POST /command/core/load-language (9ms)
17:52:05.714 [ refine] POST /command/core/load-language (16ms)
17:52:05.783 [ refine] GET /command/core/get-project-metadata (69ms)
The projects failing recovery I am not interested in. Mine has ID: 1740187251626
The JS Console also isn't very informative as far as I can tell:
Hmmmm...
Thanks for the additional information. I don't see anything obvious there, so let me have a look at the project data that you made available.
Tom
OK, I think you just need more memory. I got an OutOfMemory error with the default 1400M memory setting, but it opened without issue using 4000M (it probably doesn't need that much more, but I didn't want to play guessing games).
Unfortunately, Java is kind of pathological about trying really hard to make due with the memory that it has, spending more and more time garbage collecting and less and less time doing real work, so it can kind of grind to a halt when it should really just give up. I added code to measure memory (and report) usage during project creation, but we don't have anything similar for opening an existing project. Generally, if it was open before, it should be able to be opened again, but I think you hit an edge condition where your last operation pushed up the memory requirements enough that you can no longer open the project with the same amount of memory.
Try increasing the amount of memory that you give to the OpenRefine server (either using refine.ini or on the command line) and let us know if that fixes things for you.
Tom
p.s. And remember to keep making regular backups
1 Like
Thank you! This is really giving me hope that the problem can be solved.
I've adjusted settings like this, but nothing is happening yet.
#Maximum JVM heap (memory) and max form size allocations
#REFINE_MAX_FORM_CONTENT_SIZE=1048576
REFINE_MEMORY=4096M
#Initial (and minimum) size of Java heap
REFINE_MIN_MEMORY=4096M
I'll try some other methods of increasing memory and report back when I'm succesful.
Success!!! 
Editing openrefine.l4j.ini:
#Launch4j runtime config
#initial memory heap size
-Xms1024M
#max memory memory heap size
-Xmx4096M
Thank you for the tips!
Yay! Glad you figured it out. For anyone else who needs this in the future, I should have included this link to directions on increasing memory:
https://openrefine.org/docs/manual/installing#increasing-memory-allocation
Tom
1 Like
