One can always dream...
Released 7.0.0: small bug fixes to assist downstream packager of OS distributions.
For more details on the items, see the links in this section header. Here, I am going to just list the items with some small notes.
## Forms, Boxing, and Formatting
It feels like this should be doable, or at least we start on this path. I believe we should be able to prototype in Mathics3, but I am open to making small changes in the code. I think something like a change to the implementation of MakeBoxes was requested where we need to be looking at "upvalues" instead of "ourvalues" or maybe it is the other way around. I do however believe there is a way where we can prototype this before committing to a full implementation.
## Performance
This is a more nebulous overall and needs to be broken down more. There are some shorter-term and more measurable steps you will below and elsewhere.
### Initial Startup Time
With the work done in 2023 to allow mathics.builtin modules to be loaded deterministically through a "load" function, we can probably reduce startup time by as much as 2 seconds or so. And at the same time, addresses problems that would be caused by scaling the number of built-in functions up many times.
Python/Numpy literal expressions fall into this category and this is especially useful in data crunching and rendering graphics.
With the work done in 2023 towards providing access to instruction-like procedure calls, we probably are in a better position to handle smarter compilation, i.e. something other than turning it into Cython, and hope that Cython and figure out how to speed up what we don't have a handle on.
Core object like BaseElement and possibly Symbol, (and probably others) are _still_ too "fat": they have too many custom methods that are not applicable for most of the subclasses support. It is likely another pass will be made over this.
We have started moving "eval" code out of the "eval" methods and into its own module.
I hope Pattern Matching and rewrite rules will be started sometime in 2024. Otherwise, not much to say here.
Jupyter integration depends on getting Boxing under control.
## Migration of the documentation system to Sphinx
This is important but probably less important than performance, boxing & formatting, and pattern matching.
And in the interim, we will probably be making more smaller fixes. Sigh. I am optimistic that the smaller fixes can work in the direction of making moving out easier. Modularity here.
# New Work
## Sparse Array Implementation
Li-Xiang-Ideal mentioned this. Perhaps he can elaborate.
Rocky is writeing a debugger for Mathics3.
This is be done largely outside of Mathics-core although it heavily relies on Mathics-Core internals.
This pulls out of Mathics Core any knowledge of Operator Precedence and instead uses tables found in mathics-scanner. This would be based on Robert Jacobson's work.
- Revised to support Python 3.12 along with dropping Python 3.7 support
- Possible miscellaneous updates in packages and git commit hooks
- Be on the lookout for Open-Source WMA Packages like KnotTheory or Rubi that can be run in Mathics3
# Individual Leads or Responsibilities
- Debugger
- Compilation
- Literal Expressions for performance (e.g. data processing and graphics rendering)
- Mathics Autoloading (with mmatera)
- Operator Precedence (with mmatera)
- Releases (with mmatera)
- Forms, Boxing and Formatting
- Sparse Array Implementation
When the release settles, "Forms, Boxing, And "Formatting" is the next large refactor slated. Having this will allow us to supporting Jupyter or other front ends. And it is something that is most visibly wrong in Mathics3 output.
See PAST.rst for how the 2023 Roadmap compares to the 2022 Roadmap.
This remains the biggest holdover item from 2022, and seems easily doable. It hinders interaction with Jupyter or other front ends.
Right now "Form" (a high-level specification of how to format) and "format" (a low level specification of how output is encoded) are sometimes muddied.
For example, TeXForm may be a "Form", but output encoded for AMS-LaTeX is done by a formatter. So AMS-LaTeX rendering and other kinds of rendering should be split into its own rendering for formatter module. Currently we have asymptote, and svg "format" modules.
Back to high-level again, Boxing is something that can be written in Mathics3, and doing this at least initially ensures that we have design that fits more naturally into the Wolfram Language philosophy.
While this is probably more of an overall concern, for now, big refactoring needed here, such as going over pattern matching, will get done after Forms, Boxing and Formatting .
Forms, Boxing and Formatting will however contain one improvement that should speed up our performance: separating M-Expression evaluation from Box "evaluations).
We expect there will be other little opportunities here and there as we have seen in the past.
We scratched the surface here with ListExpression. Associations and Python/Sympy/numpy literals can be customized with an aim towards reducing conversions from and to M-expressions. A number of compound expressions, especially those which involve literals are more efficiently represented in some other way. For example, representing a Mathics3 Association as a Python ordered dictionary, a Mathics3 List as a Python list or tuple, or as a numpy array.
Core object like BaseElement and possibly Symbol, (and
probably others) are too "fat": they have too many custom methods that
are not applicable for most of the subclasses support. It is likely
another pass will be made over this.
We have started moving "eval" code out of the "eval" methods and into its own module.
While we have put in quite a bit of effort to get these to be 6.0.0 compliant. There is still more work to do, and there are numerous bugs there.
Refactoring code to generate Graphs in pymathics.graph might happen. Porting the pymathics.graph code to use NetworkX 3.0 would be nice;
pymathics.natlang could also use a look over in terms of the libraries we are using.
After Mathics3 Version 6.0.0, Python 3.6 will be dropped and possibly 3.7. Changes are needed to support 3.11 so we will be focusing on 3.8 to 3.11.
We have gradually been using a more modern Python programming style
and idioms: more type annotation, use of isort (order Python
imports), black (code formatting), and flake8 (Python lint
checking).
As mentioned before, pattern-matching revision is for later. This discussion is a placeholder for this discussion.
Overhauling the documentation to use something better supported and more mainstream like sphinx is deferred. This would really be nice to have, but it will require a bit of effort and detracts from all of the other work that is needed.
We will probably try this out in a limited basis in one of the Mathics3 modules.
Speaking of Mathics3 Modules, there are probably various scoping/context issues that Mathics3 modules make more apparent. This will is deferred for now.
Way down the line, is converting to a more sequence-based interpreter which is needed for JIT'ing and better Compilation support.
Likewise, speeding up startup time via saving and loading an image is something that is more of a long-term goal.
Things in this section can change, depending on the help we can get.
No doubt there will be numerous bug fixes, and builtin-function additions especially now that we have a better framework to support this kind of growth. Some of the smaller deferred issues refactorings may get addressed.
As always, where and how fast things grow here depends on help available.
This has been the biggest impediment to doing just about anything else.
We will isolate and make more scalable how boxing and top-level formatting is done. This will happen right after release 5.0.0
We have an API for graphics3d which is largely used for many Graphics 3D objects like spheres and regular polyhedra. However, this needs to get expanded for Plotting.
An API for JSON 2D plotting is needed too.
While we have made a start on this in 5.0, much more is needed.
We have only gone over the top-level evaluation for compound expressions. The following evaluation phases need to be gone over and revised:
- pattern-matching and rewrite rules
- apply steps
With respect to top-level evaluation, we have only scratched the surface of what can be done with evaluation specialization. We currently have a kind of specialization for Lists. Possibly the same is needed for Associations.
This work will continue after the 5.0.0 release. We expect plotting will be faster by the next release or major release.
Sadly, Mathics cannot run most of the open-source WMA packages.
In particular we would like to see the following run:
- Rubi
- KnotTheory
This is a longer-term goal.
The current home-grown documentation should be replaced with Sphynx and autodoc.
Compilation is a rather unsophisticated process by trying to speed up Python code using llvmlite. The gains here will always be small compared the kinds of gains a compiler can get. However in order to even be able to contemplate writing a compiler (let alone say a JIT compiler), the code base needs to be made to work more like a traditional interpreter. Some work will be needed just to be able or create a sequence of instructions to run.
Right now the interpreter is strictly a tree interpreter.
There have been a number of things that have been deferred:
- Using unicode symbols in output
- Making StandardOutput of polynomials match WMA
- Finish reorganizing Builtin Functions so that the structure matches is more logical
- Adding more Graphics Primitives
- Working on Jupyter integrations
In some cases like the first two items these are easy, and more important things have prevented doing this. In some cases like the last two, there are more foundational work that should be done first.
With 4.0.0, we have started defining a Graphics3D protocol. It is currently expressed in JSON. There is an independent threejs-based module to implement this. Tiago Cavalcante Trindade is responsible for this code and for modernizing our JavaScript, and it use in threejs.
We expect a lot more to come. For example UniformPolyhedra is too new to have been able to make this release.
We also need to define a protocol and implementation for 2D Graphics.
While we have started to segregate boxing (bounding-box layout) and formatting (translation to a conventional rendering format or language), a lot more work needs to be done.
Also, a lot more Forms should be defined. And those that exist, like TeXForm, and StandardForm, could use improvement.
This area is still a big mess.
Although we had planned to move forward on this previously, it now appears that we should nail down some of the above better, before undertaking. Jupyter uses a wire protocol, and we still have work to do in defining the interfaces mentioned above.
That said, this is still on the horizon.
Interest has also been expressed in WebGL, and Flask front ends. But these too will require use to have better protocols defined and in place.
Sometime around release 4.0.0, all of the code related to producing documentation in LaTeX and in Mathics Django, and running doctests will be split off and put into its own git repository.
I've spent a lot of time banging on this to try to get to to be be less fragile, more modular, more intelligible, but it still needs a lot more work and still is very fragile.
Also there is much to do on the editor side of things in terms of reorganizing sections (which also implies reorganizing the builtin module structure, since those are tightly bound together).
We still need to convert this into Sphinx-based, with its doctest. We also need to be able to extract information in sphinx/RsT format rather than its home-brew markup language which is sort of XML like.
This is one area where we know a lot about what kinds of things need to be done, but have barely scratched the surface here.
The current implementation is pretty bare bones.
We have problems with recursion, memory consumption, loading time, and overall speed in computation.
I would have liked to have seen this going earlier. However right now Mathics is still at too primitive a level for any serious package to be run on it. This will change at some point though.
This is something that I think would be extremely useful and is straightforward to do someone has used Mathematica over the years knows it well. I think most of this could be supported in Mathics code itself and loaded as packages. Any takers?