bugGNU Octave - Bugs: bug #59245, Implementation of an XML interface...

 
 

bug #59245: Implementation of an XML interface without Java dependency

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Fri 09 Oct 2020 07:22:10 PM UTC
   
 
Category:  Octave Function Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Feature Request
Status:  None Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 07 Nov 2020 09:52:13 PM UTC, comment #14: 

@Markus: I'm at your disposal for helping with spreadsheet I/O internals. With io and mapping pkg releases out of the way I was planning to devote more time to other things so more active development is s/th for maybe later.
The OOXML file format has meta info distributed over a lot of separate XML files and subdirs in the zipped package; getting & keeping it all properly in sync is one of the biggest challenges. XPath etc. otherwise work fine in the individual XML files.

Just curious: is there a reason to make it an internal Octave function right from the start, rather than completely stand-alone .cc and .oct files? I'd maybe (probably?) naively think that development could be easier if the whole of Octave doesn't have to be rebuilt completely for a next trial.

BTW, I just remembered that in old versions of the io package there used to be an almost completely undocumented binary xmlread by Laurent Mazet. Maybe there's some useful stuff inside.
At some moment it got replaced by the Java-based versions from Panxto.

Philip Nienhuis <philipnienhuis>
Group Member
Sat 07 Nov 2020 08:29:21 PM UTC, comment #13: 

To check if the XML written by RapidXML is compatible with what Excel expects in .xlsx files, I read one sheet of a random .xlsx file and wrote it back.
The resulting file looks the same when opened in Excel afaics. That's not a conclusive test. But it gives hope that it might be possible to get it to work with something that is close to what is in the patch already.

To check if the functions that are implemented until now are sufficient to traverse the DOM tree and get the most important information, I tried to write a wrapper for `xmlread` that is more or less compatible to Matlab's `readstruct`. It turned out that some features were still missing.

The attached patches implement a more or less Matlab compatible `readstruct`. This is still a work in progress.
But the features of `xmlread` and `xmlwrite` that are needed for `readstruct` and `writestruct` are probably more or less what is needed for `readcell` and `writecell`. (The latter are probably a lot more complex though.)

My plan is to get skeleton versions of `readstruct` and `writestruct`. I hope that, writing those, it will be possible to detect and smooth out the roughest edges of `xmlread` and `xmlwrite`.
Maybe later on, those skeleton versions can be fleshed out and make their way into Octave.

For the proper goal of getting `readcell` and `writecell` for reading Excel files: I'll definitely need help for implementing those. I hope we can draw from the knowledge accumulated in the io package.
Maybe, we could try to implement those functions in .m files using `xmlread` and `xmlwrite` when they are ready. If it turns out that the performance cost for those interfaces is too high, we might be able to speed things up by getting more specialized interfaces to RapidXML. (But I'm getting ahead of myself...)

(file #50222, file #50223)

Markus Mützel <mmuetzel>
Group administrator
Mon 12 Oct 2020 04:50:07 PM UTC, comment #12: 

Sorry if I'm stating the obvious but the point I was trying to make is that xmlread/xmlwrite can be used to implement readstruct/writestruct but not the other way round. OOXML files probably require the level of detail that xmlread/xmlwrite but I would guess these functions would be too slow to use when manipulating the output object.
I used yxml (https://dev.yorhel.nl/yxml) to parse XML and I liked it - it's not as widespread as RapidXML or libxml2 but it's easy to use and I can see it being used in Fuchsia to parse SVG files so there are credentials.

Guillaume <gyom>
Mon 12 Oct 2020 09:44:02 AM UTC, comment #11: 

About error messages:
Excel may show file names and line numbers [*], but that is of little use in a long XML string of several tens-of-thousands or more characters - it'll be a long search.
LibreOffice is so nice to also tell you at what character position it encountered a problem.

[*] Excel 2013 and 2016 only tell me something along the lines of "We found a problem in <filename>. Delete content"?
Furthermore, are line numbers really useful? OOXML files usually consist of just two lines: Line 1 is the XML version header, line 2 contains all the actual XML content.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 12 Oct 2020 09:33:32 AM UTC, comment #10: 

RapidXML's I/O? => that was mentioned in the decision tree you referred to in comment #5. Go to that site, scroll down to "Maximum XML Parsing Performance" and then scroll a little more down under the RapidXL header to read this;

> One other downside with RapidXML is that it is painful for writing XML. It requires you to do a lot of explicit memory allocation of string names in order to build its DOM. It does provide a kind of string buffer, but that still requires a lot of explicit work on your end. It's certainly functional, but it's a pain to use.


But if you've already sorted it out this is obviously moot.

Philip Nienhuis <philipnienhuis>
Group Member
Mon 12 Oct 2020 07:22:43 AM UTC, comment #9: 

@Philip: Thank you very much for your feedback.
Could you please elaborate on what you think is bad about how RapidXML implements IO?
It takes a pointer to an array for input and writes to any string iterator. Imho, it is a plus point that it doesn't try to hide file IO in its API because that is bound to cause issues on some platforms (Windows, I am looking at you!). We have several wrappers in Octave already that handle the platform specific peculiarities...

Excel's error messages (including file names and line numbers) proved to be quite useful to identify the issue...

AFAICT, there is no XML standard that dictates or suggests order of sibling nodes. That must be OOXML specific.

Markus Mützel <mmuetzel>
Group administrator
Sat 10 Oct 2020 09:38:45 PM UTC, comment #8: 

@Markus
I welcome your attempt very much.
All I wanted to convey in comment #3 is that IMO a project comprising implementing XML I/O with an underlying aim of providing spreadsheet I/O needs very careful thinking over.

RapidXML seems a pain for I/O; for me that would be enough to just skip it at first sight. But maybe some clever extension or wrapper can alleviate the writing complications.

A hint:
Excel may complain about invalid OOXML but only offers to wipe the offending content w/o further detail.
To get the exact error location you need LibreOffice. That tells you exactly at what character position in which XML file it lost track.

It is reassuring to read that Excel is picky about ordering of tags. Reassuring because such order is the major weakness of parsing XML with regular expressions.
Do XML standards dictate or suggest a default tag ordering, e.g., alphabetical or so? AFAICS OOXML has a fairly alphabetical tag ordering (but not consistently, as nothing from Microsoft is consistent).

Philip Nienhuis <philipnienhuis>
Group Member
Sat 10 Oct 2020 03:45:21 PM UTC, comment #7: 

I tried to read the xml of an .xlsx file with `readstruct` and write it again (without changing anything) with `writestruct`. There were issues writing the header (which are easy enough to work around).
I then zipped everything pack to an .xlsx file.
If I try to open the resulting file, Excel complains about invalid syntax.
The main issue seems to be that the order of tags seems to be important for OOXML. E.g. it looks like v-tags must not appear before f-tags. If I manually edit the xml to correct the order of these tags, Excel seems to be able to open the resulting file.

It looks like `readstruct` successively adds fields to the struct arrays while iterating through the DOM nodes. That means that the order of fields can differ from the one in the file if some tags don't exist in earlier nodes (see the attached screenshot).

This is probably what you were hinting at in comment #6.

I'm still debating how to proceed.
Maybe I shouldn't give up on xmlread/xmlwrite so quickly.
One possible strategy might be to rename them to oxmlread/oxmlwrite (to avoid having them shadowed by the functions in the io package which will always be more Matlab compatible). That way we could leave them deliberately incompatible where we don't need full compatibility.
We could also make them internal functions for the (not yet existing) readcell/writecell. But I guess they could come handy for end users, too.

Also, implementing readstruct/writestruct might still be worthwhile even if it won't be possible to use those functions for reading and writing OOXML...

I'll have to think about that. Input very welcome.



Markus Mützel <mmuetzel>
Group administrator
Sat 10 Oct 2020 01:49:32 PM UTC, comment #6: 

readstruct/writestruct are targetting very specific types of XML files, e.g. it starts to get ugly with this:


<a e="f"><b>c</b>d<Text>g</Text><b>3</b></a>



>> readstruct('test.xml')
ans =
  struct with fields:
    eAttribute: "f"
             b: ["c"    "3"]
          Text: "d"
        Text_1: "g"


xmlread/xmlwrite always felt more Java than Matlab but you can access appropriately any content of an XML file. In the example above, you can see that the data order is lost with "3" appearing before "g" in the struct, so you wouldn't be able to use readstruct/writestruct if you wanted to parse an XHTML file. I guess it would be a problem with OOXML too.
I can see why they added the new functions though as that's what many people want (it is equivalent to the convert() method of the xmltree class I wrote a long long time ago https://www.artefact.tk/software/matlab/xml/).

Guillaume <gyom>
Sat 10 Oct 2020 11:49:37 AM UTC, comment #5: 

@Guillaume: I wasn't aware of these functions. Their interface looks much cleaner and "more native" than xmlread/xmlwrite.
For a test, I unzipped a random .xlsx file I had lying around and tried to read one of the worksheet .xml files in Matlab R2020b. It was successfully read into a (nested) structure that resembles the DOM tree of that file.
I wonder if working with these structures would cost performance as opposed to accessing the XML DOM tree more directly. But afaict, it might be possible to use these functions as intermediate layers for reading/writing .xlsx files.
Given that, I'll drop trying to implement xmlread/xmlwrite in favor of readstruct/writestruct.
I just wish I knew about these functions earlier...

@Philip: There are also readcell/writecell which work with cell arrays instead of tables. We might be able to implement those already. That could be done either in Octave language using readstruct/writestruct (if the performance penalty isn't too bad) or by accessing the XML DOM tree directly in C++. (Or maybe first implement it in Octave language to get the hierarchy sorted out and then translate that to C++ if necessary.)

AFAICT, the only XML "library" that core Octave is using already is QtXml. But Qt is an optional dependency (for the GUI and graphics toolkit). Additionally, QtXml isn't even mentioned as a contender in comparisons I found...
There might be different reasons why other software choses to use different XML libraries. But RapidXML seems to outperform libxml in comparisons (for XML it is able to read at least):
https://pugixml.org/benchmark.html

A nice decision tree with different XML parsers is e.g. here:
https://stackoverflow.com/questions/9387610/what-xml-parser-should-i-use-in-c

Also RapidXML seems to be stable and equally available on many distros.

I'd prefer to implement these features in incremental steps. Imho, we don't need to support .xlsx, .ods and .xls right from the start. Just one of them (.xlsx) would be better than none. If users want to read/write other formats, they will still be able to use the functions in the io package...

Being able to read "plain" XML would be a good initial step imho.

Markus Mützel <mmuetzel>
Group administrator
Sat 10 Oct 2020 10:09:42 AM UTC, comment #4: 

@Philip: you are right, I was distracted by the title of this report. If the aim was to improve XML support in Octave, I thought it might be worth focussing on readstruct/writestruct but this will not really help if the aim is to build the tools in order to implement xlsread/xlswrite. Given the rather generic naming of readstruct/writestruct, I wonder if they will go beyond XML in future releases.

Guillaume <gyom>
Fri 09 Oct 2020 11:07:57 PM UTC, comment #3: 

Some remarks, in no particular order:

In the mxe-octave installers there's libxml2 (and libxml++). Can that also be used? I suppose it's also in many Linux distros so if yes it may be a better library to use.

xmlread and xmlwrite are Java based and as such highly Matlab compatible.  I'd suppose many use cases and Matlab user code Out There are also Java based; could that compatibility be partly or wholly lost when switching to non-Java based parsers and other XML functions?

I'd support getting some form of xlsread and xlswrite into core Octave. Years ago JWE wrote that he also wanted it.
If this is done IMO it should read .xlsx, .ods and if possible, .xls (BIFF8 and/or BIFF5) as many embedded systems spew out .xls. Gnumeric is a very easy format so could also be included.

But writing .xlsx is a daunting task. The ECMA specs for the OOXML specs comprise thick tomes and to further put you off I'd suggest to read e.g., this:
http://ooxmlisdefectivebydesign.blogspot.com/2007/08/microsoft-office-xml-formats-defective.html   (hmm yet I found many useful hints in there :-) )
That said, .ods is even more complicated to handle with many implicit rules and no address in spreadsheet cell nodes.
In the io package OOXML and ODS formats have already been sorted out (many corner cases excluded) so that could serve for a start.

Matlab these days has readtable and writetable that can read from and write to spreadsheets. Their interface / calling form looks much more practical than the xlsread/xlswrite requirements of fixed but inconsistent argument order.
To implement readtable and writetable Octave needs the Table class. Andrew Janke has made Tablicious and on the task tracker there's a task for Table I/O based on the io package (task #15419)

@gyom:
readstruct doesn't look like it can read a spreadsheet file, can it?

Philip Nienhuis <philipnienhuis>
Group Member
Fri 09 Oct 2020 08:50:10 PM UTC, comment #2: 

Have you noticed that there are two new functions in Matlab R2020b, readstruct and writestruct:
https://www.mathworks.com/help/matlab/ref/readstruct.html
https://www.mathworks.com/help/matlab/ref/writestruct.html
They are probably more convenient than xmlread/xmlread for most use cases.

Guillaume <gyom>
Fri 09 Oct 2020 07:25:28 PM UTC, comment #1: 

Attached the patch.

I only tested it with toy examples. A better test would be to try if they work for reading and writing .xlsx files.
Is that possible at all or are there missing features?
How do the functions perform?

(file #49955)

Markus Mützel <mmuetzel>
Group administrator
Fri 09 Oct 2020 07:22:10 PM UTC, original submission:  

The io package provides the functions `xmlread` and `xmlwrite` that are basically wrappers for the Xerces Java jar files.
I'm not sure if this (Java) is the reason why they aren't moved to core Octave. But Philip hinted at this in the past.

My "real" goal here is to get `xlsread` and `xlswrite` into core Octave.
Those function are also provided by the io package. They are implemented using several different interfaces. But those are either slow (the OCT interface), rely on proprietary software (the COM interface) or need Java (the remainder?).
IIUC, the most likely to ever be implemented in core Octave is the OCT interface. Unfortunately, it is quite slow for reasonably large .xlsx files. This is probably because it uses regular expressions that are getting more and more complex to match the XML of .xlsx files.
My hopes are that using an XML parser will be easier and faster than matching the XML with regular expressions.

There are several free C/C++ XML parsers. To be able to read and write the XML of .xlsx files, we won't need to have full XML compliance.
Others [1] are using RapidXML [2] for that task which is supposedly very fast and has a simple but adequate interface.

So, I gave it a shot and wrote a simple interface from Octave to RapidXML which is more or less compatible to Matlab.
I'll upload a work-in-progress patch as soon as I have a bug number.

CC'ing Philip as the maintainer of the io package.

[1]: https://readxl.tidyverse.org/
[2]: http://rapidxml.sourceforge.net/manual.html

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #49955:  bug59245_xmlread_xmlwrite.patch added by mmuetzel (55KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by gyom (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  • -email is unavailable- added by mmuetzel
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-11-07 mmuetzel Attached File- Added bug59245_xmlread_xmlwrite_v2.patch, #50222
        Attached File- Added bug59245_readstruct.patch, #50223
    2020-10-10 mmuetzel Attached File- Added readstruct_fieldorder.PNG, #49958
    2020-10-10 mmuetzel SummaryImplementation of xmlread/xmlwrite without Java dependency Implementation of an XML interface without Java dependency
    2020-10-09 mmuetzel Attached File- Added bug59245_xmlread_xmlwrite.patch, #49955
    2020-10-09 mmuetzel Carbon-Copy- Added philipnienhuis

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code