Importing 300MB SQLite fails (freezes, no errors)

Support and Helpdesk
1

Hello everyone,

I'm trying to import a 300MB SQLite database into OpenRefine. The table has about 50 columns and 1 million rows – which is not small, but also not huge in my opinion.

I'm using OpenRefine 3.9.3 on macOS, and I've started it via the Terminal, so I can see the Java log output. I entered the path to the SQLite file, the preview looked fine (I didn't change anything) and pressed proceed, to start the import.

Everything looks good, the status bar progresses. But it never finishes. The memory usage rises to 100% eventually (after 2-3min). And after about 8min, all log outputs in the Terminal freeze (no more regular POST /command/core/get-importing-job-status (27ms) messages). Nothing happens anymore, and there are no errors anywhere.

EDIT: The POST /command/core/get-importing-job-status log outputs eventually came back. They're still coming in, but the process is running now for 60min, so I doubt that it's actually still doing something.

Any ideas what I could try to fix this?

2

Raise the Java engines memory from the default 1400M to 16G or whatever your system can provide. The log should show something helpful during startup. See the refine.ini config file to set the REFINE_MEMORY env variable.

2 Likes
3

Thanks! Sorry, I only now found that this is mentioned in the docs, that one should increase the memory limit for files larger than 50 MB. I did not read carefully enough. I increased the limit to 8 GB and the import was done within 2 minutes.

1 Like