Go over row box parsing

[rocky]

@mmatera : this branch possibly parses boxing operators better. But I am not sure. The below seem to match better WMA output, but one thing that is tricky is that there is a level of box evaluation going on and our box evaluation is probably wrong.

And then there are the unit tests. Our unit test test/core/parser/test_parser.py seems to work with both existing master and these changes. And it has tests for SuperscriptBox that purport that the existing code should be doing the right thing, but inside mathics, I am not seeing that.

So it is also possible that the existing parser is okay, but our boxing routines are completely at fault, aside from the syntax error show below.

I think I can easily fix the parsing error, but I don't want to do this until I understand what the situation is in master, what is wrong with it, and that we determine that this is a parsing problem. Likewise, failures in FormBox tests raise the question: is it the parser that is at fault, our boxing routines, or our test expectations.

Please let me know your thoughts or investigation in this.

In[1]:= \( x \)
Out[1]= x

In[2]:= \(x \^ y \)
Out[2]= SuperscriptBox[x, y]

In[3]:= \(x \^ y ^ z \)
Syntax::sntxf: "\(x \^ y " cannot be followed by "^ z \)" (line 1 of "<stdin>").

The above should be SuperscriptBox[x, RowZBox[{y, ^ z}]]. Continuing...

In[3]:= \(x \^ y \^ z \)
Out[3]= SuperscriptBox[x, SuperscriptBox[y, z]]

In[4]:= \(x \_ y \)
Out[4]= SubscriptBox[x, y]

In[5]:= \(x \_ y \_ z \)
Out[5]= SubscriptBox[x, SubscriptBox[y, z]]

In[6]:= \(x \^ y \_ z \)
Out[6]= SuperscriptBox[x, SubscriptBox[y, z]]

In[7]:= \(x \_ y \^ z \)
Out[7]= SuperscriptBox[SubscriptBox[x, y], z]

[rocky]

I am now thinking that there is nothing wrong with master, and the whatever problems that appear lie elsewhere.

[mmatera]

@rocky, I need to look at this again, but one clue that could be useful is that Box expressions should not be "evaluated", in the sense that once something is parsed as a box expression, it should keep its form. So, if WMA parses
"""
(x ^y ^ z)
"""
as SuperscriptBox[x, SuperscriptBox[y,z]], it would stay like this, without changes by the evaluation.

Maybe the most useful thing to start with this would be to add more pytests, to check the compatibility.

[rocky]

@rocky, I need to look at this again, but one clue that could be useful is that Box expressions should not be "evaluated", in the sense that once something is parsed as a box expression, it should keep its form. So, if WMA parses """ (x ^y ^ z) """ as SuperscriptBox[x, SuperscriptBox[y,z]], it would stay like this, without changes by the evaluation.

Maybe the most useful thing to start with this would be to add more pytests, to check the compatibility.

There are already lots of tests in test/core/parser/parser.py

[mmatera]

I am trying to find out what requires to be fixed here. I remember we had some issues posted, and also some prs, but I am not finding them.

[mmatera]

@rocky, in this branch https://github.com/Mathics3/mathics-core/tree/improving_parse_FormBox_0 , I put something that seems to be related to the parsing step. Still, there is one pytest that still is not working. Maybe it deserves to have a look.

[rocky]

@rocky, in this branch https://github.com/Mathics3/mathics-core/tree/improving_parse_FormBox_0 , I put something that seems to be related to the parsing step. Still, there is one pytest that still is not working. Maybe it deserves to have a look.

I will look at possibly next weekend sometime.