Parser Settings

Support and Helpdesk
1

Hi,

How to change the maximum number of characters defined in the parser settings?

IMG_20240523_183444
IMG_20240523_183444734×361 109 KB

Thx in advance
Lenn

2

Hi Lenn,

Did you find where to make your changes in our code below? For finding things in code, you can just do a code search from GitHub or your IDE for something like "CsvParserSettings".
Also if you need further detailed help on their library, you can probably ask uniVocity here: uniVocity/univocity-parsers: uniVocity-parsers is a suite of extremely fast and reliable parsers for Java. It provides a consistent interface for handling different file formats, and a solid framework for the development of new parsers. (github.com)

3

How to change the maximum number of characters defined in the parser settings?

Before we dive into changing things, can you describe what it is that you were trying to do when you got this error? It may just be a bug or some other issue.

Was it a CSV file that you were trying to load? Did it have especially long lines or large text strings in the cells?

Tom

4

Same question as OP. Parsing a log file with JSON in one column that could be pretty large, but not unbounded.

Is this fixed here? OpenRefine/main/src/com/google/refine/importers/SeparatorBasedImporter.java at master · OpenRefine/OpenRefine · GitHub

(I'm curious how 32K maps to 256K shown in the error; four bytes for unicode?)

Would be great if this was configurable.

1 Like
5

Same question as OP. Parsing a log file with JSON in one column that could be pretty large, but not unbounded.

Is this fixed here? OpenRefine/main/src/com/google/refine/importers/SeparatorBasedImporter.java at master · OpenRefine/OpenRefine · GitHub

(I'm curious how 32K maps to 256K shown in the error; four bytes for unicode?)

Would be great if this was configurable.

Hi Paul. Glad you found your way from the old mailing list. The usual questions apply: what version of OpenRefine, CSV or TSV, what error are you getting, do you have example data that demonstrates the problem, etc.

When we increased the max column count from 512 to 16K, we also reduced the maximum cell size from 256K to 32K (the default is 4K), to balance memory usage a little. If you'd like this value to be configurable, please open a feature request. Including a PR with your feature request would be awesome.

What size fields are you looking to be able to work with?

Tom

6

@paulmakepeace Hi long lost Paul!