RDF, JSON and CSV parsers have already been implemented based on RecordReaderGenericBase, but XML is handled by two separate implementations which to the best of my knowledge are not capable of ingesting data in parallel (is that still correct @patrickwestphal @Hebaallahibrahim ?)
For XML documents that essentially contain a collection of items (and only the header and footer need to be ignored) the parser should be easy to adapt: ignore the wrapper tags until the collection element is reached and emit the items. The record offset seek procedure has to take care of <item> tags nested in other <item> tags. The parser framework already supports reading ahead records for validation purposes. The validation rule is something like: When finding an <item> tag at depth N then reading ahead records must finds tags at a lower depth (which would be the case if we started at an <item> below an <item>) unless it is </collection></wrapper>EOF.
<preamble>
<wrapper>
<collection>
<item>1</item>
<! -- ... -->
<item>n</item>
</collection>
</wrapper>
RDF, JSON and CSV parsers have already been implemented based on
RecordReaderGenericBase, but XML is handled by two separate implementations which to the best of my knowledge are not capable of ingesting data in parallel (is that still correct @patrickwestphal @Hebaallahibrahim ?)For XML documents that essentially contain a collection of items (and only the header and footer need to be ignored) the parser should be easy to adapt: ignore the wrapper tags until the collection element is reached and emit the items. The record offset seek procedure has to take care of
<item>tags nested in other<item>tags. The parser framework already supports reading ahead records for validation purposes. The validation rule is something like: When finding an<item>tag at depth N then reading ahead records must finds tags at a lower depth (which would be the case if we started at an<item>below an<item>) unless it is</collection></wrapper>EOF.