mathics3 / mathics-scanner Goto Github PK
View Code? Open in Web Editor NEWTokenizer, and character tables, and conversion routines for the Wolfram Language.
License: GNU General Public License v3.0
Tokenizer, and character tables, and conversion routines for the Wolfram Language.
License: GNU General Public License v3.0
See #3 (comment) for details.
mathics-scanner has been released to 1.1.0 and this produces an error in Mathics:
pkg_resources.DistributionNotFound: The 'Mathics_Scanner<1.1.0,>=1.0.0' distribution was not found and is required by Mathics3
Consider the input in the
In[1]:= "\\[Integral]" == "\\" <> "[Integral]"
In WMA, the result is
Out[1]= True
On the other hand, in Mathics the result is
Out[1]= False
The reason is mathics-scanner parses the LHS as "\u222b"
, which is different to the RHS "\[Integral]"
I found it when I tried to write tests for Mathics3/mathics-core#541
We should add some documentation on the tokenizer before releasing the package. Simple docstrings will do for now.
The file requirements-extra.txt is not included in the released source code, so building directly from that is not possible.
Since I'm packaging this for AUR, I would like to have a brief description of what each extra requirement brings. In this case, what would a user gain by installing ujson? I know it used to say #Optional Used in mathics_scanner.characters
but that doesn't mean too much in terms of why (not) installing it.
While working in #11 it became clear to me that the tokeniser is still very much tied up to internals of Mathics core. For instance, the whole messaging mechanism is completely useless to anyone other us (the developers of Mathics) and it could likely be entirely replaced by simply throwing errors. Also, there are multiple improvements that could be made to make the public interface cleaner and more intuitive.
I propose the following changes:
Tokeniser
and LineFeed
(this will require some refactoring in core)__next__
for Tokeniser
by simply calling the next
methodtag
parameter of Token(tag, text, pos)
and mark the type of the token by using subclasses of Token
(i.e. Token("Number", "3", 4)
becomes NumberToken("3", 4)
)incomplete
method: I'd like to remove it (since it's more of an implementation detail than anything else), but it's used in core so we'd had to deal with that too. Even if we don't remove it, we should rename it to something descriptiveIdeally, I'd like to take care of this before the release, since this are breaking changes and therefore would require a major version bump if we were to merge them after the first release. However, I understand that the refactoring required will take some time and therefore I'm OK with doing this after the first release.
I also take full responsibility over this. I can do most of this work on my own if the rest of the contributors aren't interested. @rocky @mmatera Thoughts?
We should decide what's going to be part of our public API before releasing. Things that aren't in the public API should be marked with a _
at the start of their names.
As a start, I propose to keep everything other than replace_*_with_*
, named_characters
, esc_aliases
, generate.rl_inputrc
and whatever is used by Mathics core outside of the public API. @rocky @mmatera Thoughts?
For tables this size, consistency tests are useful of the generated tables is useful
Most symbols round trip. Those that don't would be noted in a list in the test and all others would be checked for
round-tripping. The number of keys in each table agrees. The set of symbols that maps back to a WL long name is eactly the set of WL symbols.
All tests passed successfully with 1.2.4, but 5 AssertionError are emitted with 1.3.0:
FAILED test/test_tokeniser.py::test_apply - assert [Token(Symbol...Symbol, x, 5)] == [Token(Symbol...Symbol, x, 5)]
FAILED test/test_tokeniser.py::test_association - assert [Token(RawLef...ation, |>, 8)] == [Token(RawLef...ation, |>, 8)]
FAILED test/test_tokeniser.py::test_integeral - assert [Token(Integr...Symbol, y, 6)] == [Token(Integr...Symbol, y, 6)]
FAILED test/test_tokeniser.py::test_set - assert [Token(Symbol...Symbol, y, 4)] == [Token(Symbol...Symbol, y, 4)]
FAILED test/test_unicode_equivalent.py::test_has_unicode_equivalent - AssertionError: In Alternative - remove add unicode equivalent
Full log attached.
mathics-scanner_log.txt
In the current database in master, there is a mistake: a\[RawTilde]b\[RawTilde]c
should be parsed as b[a,c]
,while
'a[Tilde]b[Tilde]c' parses to Tilde[a,b,c]
. \[Tilde]
has associated the Unicode character \u223c
, while \[RawTilde]
corresponds to the ASCII ~
(\u007e
).
https://reference.wolfram.com/language/ref/character/Tilde.html
https://reference.wolfram.com/language/ref/character/RawTilde.html
In WMA, a not very well documented feature, allows to input unicode characters which does not have a name. For example, for a 32-bits character, we can write \|xxxxxx
where xxxxxx are the hexadecimal digits of the character code:
In[1] := "\|01D451"
produces in WMA
Out[1] = ๐
(notice that this character does not have a name in WL).
After the last release, the behavior of MathicsScanner changed, in a way that named characters in strings are mapped to unicode-equivalent
instead of wl-code
as before. After fighting with the formatter code in mathics-core, I think this behavior is wrong.
The reason is that the goal of having unicode-equivalent
is to provide a readable output, not to have an efficient way to store characters.
The example comes up with "\[DifferentialD]"
. In 1.2.4, this string was parsed as "\u7f4c"
, which was a WL specific character, with a specific meaning. If the string has a form like "\[Integral]F[x]\[DifferentialD] x"
, the string can be parsed afterward as the expression
Integrate[F[x], x]
. On the other hand, if we want to produce a printable version, \[DifferentialD]
could be converted into d
, or \u0001D451
or \, d
, according to the place we need it.
With the current behavior in master, the test/format/test_format.py
tests in mathics-core fails.
The release tar ball as well as the pypi release is missing the mathics_scanner/data/named-characters.yml
file, so running the testsuite will fail
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.