Support for currently active python versions (e.g. 3.7) about tatsu HOT 20 CLOSED

@youtux, TatSu does not have any third-party library requirements.

Stating compatibility with >=Py310 has the purpose free core developers to use the language features introduced up to Py311.

@dnicolodi made sure that the latest TatSu works fine with Py38, and has committed to try keeping it so on the 5.7.x branches.

If TatSu 5.7.x works for you with Py37, then just stick to it. TatSu is complete in that it does everything it is intended to do, and there are no important bugs posted against it. Any changes made to TatSu in the future will be optimizations and enhancements (for example, the new strategy for code generation inspired by the new Python PEG parser, which will not break the old strategy).

The worse that can happen is that a future TatSu 5.8.x or 5.9.x will use a language feature not available in Py37, but you may have moved to Py311 by then.

My estimation is that other libraries you use will become incompatible with Py37 before TatSu does.

from tatsu.

Furthermore, the technical reason why Python 3.7 is not supported has been explained. You haven't proposed a way to overcome the issue. Reiterating that TatSu must support 3.7 does not help to get the issue solved.

from tatsu.

I've also been lobbing for wider Python versions support. With @apalala we came to the compromise that support for Python 3.8 and later is guaranteed only on the 5.7 release series. Changes to master (to become the next 5.8.x releases) will be backported and released as 5.7.x as required. See recent (closed) issue and (merged PRs).

Supporting Python 3.7 requires some changes that would make the API between the two releases subtly different, thus I decided not to keep support for Python 3.7.

What is the reason for wanting support for Python 3.7? Several widespread projects already dropped support for it and all major Python distributions channels moved to more recent versions a long while ago. Do you actually have users requesting support for Python 3.7?

from tatsu.

Because it is quite common for companies to update their python version only in the moment they stop being supported.
This is especially true for companies with large codebase and a lot of tech debt, where upgrading python version requires some extra work.

Or it can also be that somebody is stuck on an old (but still supported) python version because of some other dependency that is not compatible with the more recent python version.

In the end, we want to make things easy for our userbase, that's why we make software.
I usually add compatibility logic in a separate module, so that I can abstract away differences between pythons (and even pytest versions in my case) and still support a wide variety of versions.

from tatsu.

Also, most popular projects still actively support python 3.7:

• boto3
• requests
• python-dateutil (it even supports 2.7, that's even too much)
• attrs (>=3.5)
• jinja2

from tatsu.

Why someone stuck for whatever reason on Python 3.7 does need to use the latest TatSu release or the latest release of a project potentially depending on TatSu? Python 3.8 has been released almost three years ago, I don't see anyone stuck on Python 3.7 sprinting to update to the latest release of anything.

There are language syntax changes between Python 3.7 and 3.8, there is no way to overcome that with compatibility modules https://peps.python.org/pep-0570/

from tatsu.

I think that it's common for parsing libraries like tatsu to not be used by end users directly, but rather by other libraries that need to parse things in order to provide a feature, just like pytest-bdd.

In this scenario, users don't even care what parsing library a dependency uses. But it would be inconvenient if they can't upgrade version of pytest-bdd because a transitive dependency (tatsu) doesn't support py3.7.

I think it's good not to make too restrictive version requirements, since the larger your dependency set, the larger the chance of having unsatisfiable requirements, especially if libraries drop support for older python versions that are still maintained.

from tatsu.

And even if you are "stuck" on 3.7 you should still be able to use all the latest version of libraries, or at least always update to security patches for all the libs.

Some companies even update their python version every 4 years, so that they are always on a supported version, but they have the "burden" of upgrading only once every 4 years.

from tatsu.

Tatsu 5.7 does not work with python 3.7. Probably the last version of Tatsu that works with it is in the 4.x range

from tatsu.

There's lack of features and too many bugs in the TatSu 4.x versions.

from tatsu.

@youtux, you could evaluate what it would take for TatSu 5.7.x to be compatible with Py37. I don't think that @dnicolodi would object to a pull request that brings in the compatibility.

from tatsu.

And even if you are "stuck" on 3.7 you should still be able to use all the latest version of libraries, or at least always update to security patches for all the libs.

Can you name a good reason why someone would need to keep using Python 3.7?

As I see it, there are essentially two ways in which you can get a Python environment: provided by your OS vendor or by installing a third party package. AFAIK, not considering very niche platforms, only Linux distribution provide a Python interpreter. On these platforms it is somehow cumbersome to upgrade to Python version more recent than the one provided by the OS vendor. For users of these platforms, it makes sense to provide Python packages compatible with the Python version included in all currently supported OS versions. AFAIK, this sets the baseline to at least Python 3.8. For all other OSes you need to install a third party package and, unless there are self inflicted restrictions, there is no reason not to upgrade the Python version as required. This was my reasoning when I offered @apalala to curate releases of TatSu targeting Python 3.8 and later.

Some companies even update their python version every 4 years, so that they are always on a supported version, but they have the "burden" of upgrading only once every 4 years.

Doing this transfer the burden of upgrading Python versions to the open source maintainer that have to maintain their packages compatible with unreasonably old Python versions, and I am not sympathetic to this way of exploiting the open source communities.

I would not oppose supporting Python 3.7 if it would come with no consequences. But, as already stated, TatSu uses the Python syntax introduced in Python 3.8 for positional only arguments. There is simply no way to go back and fix this in a way that will not subtly change the TatSU API. I don't want to introduce this kind of subtle incompatibilities between TatSu 5.7 and 5.8. Even more so if the need for supporting Python 3.7 is only theoretical.

Do you know of anyone with a good reason for not upgrading to at least Python 3.8? I am not aware of any Python package that supports Python 3.7 but not later versions. Even then, I would argue that it would be best to spend efforts trying to make these packages compatible with newer Python releases that keeping all other packages, including TatSu, compatible with Python 3.7.

from tatsu.

Any changes made to TatSu in the future will be optimizations and enhancements (for example, the new strategy for code generation inspired by the new Python PEG parser, which will not break the old strategy).

@apalala This is interesting. Is there any published work toward this?

I have a couple of ideas for enhancements. I'll try to find time to put together some concrete proposals or patches.

from tatsu.

Looking at the download stats for my package, 11% of the downloads is still for python 3.7. Probably other packages have the same distribution.

Since it's an officially supported version, I want to maintain compatibility with it.

This of course is not only about 3.7 in particular, but it's about the policy for python support for Tatsu.

The question in the end is, what's the policy about python version support?

If Tatsu maintainers don't want to commit to any, or if they want to only commit to the last 3 python versions, that's fine of course, I will just have to use a different library that fits my needs.
And I think that maintaining compatibility with all supported versions of python is a must for any open source project.

from tatsu.

How many of these downloads are from CI jobs that verify some other package is compatible with Python 3.7? Accordingly to these download statistics you need to care as mush for Python 2.7 users as for Python 3.7 users (or for a mysterious Null Python version, for what matters). There is no way TatSu will support Python 2.7. Download statistics alone are not a good metric, especially if you cannot distinguish between CI jobs and real users.

from tatsu.

My decision was that TatSu would follow Python releases after PEP602 became official: https://peps.python.org/pep-0602/

The anual release cycle was adopted by python.org because of the pain caused by abandoning 2.7, and some mishaps while moving to to 3.5.

The result is that programs written for 3.5 will run on even 3.11, probably without any changes, and that should be good enough for the software engineering planning of most projects.

Asking open-source libraries keep compatibility with security-only releases of Python, or that they not use the new language or stdlib features does not make sense, because in most cases projects have just to upgrade the interpreter to 3.10, and not have to change a line of code.

Add to that that libraries still compatible with 3.7 are missing years of bug and security fixes, standards upgrades, and optimizations.

Furthermore, TatSu 5.8 got rid of the huge parameter lists and of the numerous bugs they generated using Py38 language features. @dnicolodi had minimal effort (basically testing and documenting) to make sure that TatSu 5.8 worked over Py38.

If @youtux wants to provide a pull request for a TatSu version that works with Py37, I'll be happy to publish it, but I think it's no longer possible to achieve that without loosing years of bug fixes and enhancements.

I, @apalala , will not work even on 3.9 compatibility because:

1. I lack the resources
2. All my code base is already running on 3.10 and tested against 3.11 (which is due out in October 2022)
3. Old versions of third-party libraries have deprecation warnings in 3.10 that will become errors in 3.11, or in pep8, or in pylint

What I think @youtux could do is to start adding Py310 to the testing pipelines. The likely outcome is that everything will work with recent versions of all libraries, and Python.

To close:

• Any version of TatSu published after October 2020 will state that Python>=3.11 is required
• @dnicolodi will decide for how long he will support older versions of Python on the 5.7.x branch by backporting

Remember that it is not necessary to embed a parser into a software project. It is perfectly possible to use the latest TatSu over the latest Python through a REST-alike service.

from tatsu.

What I think @youtux could do is to start adding Py310 to the testing pipelines.

Yeah I have always have the latest pythons in the pipelines since the alpha version becomes available.

Any version of TatSu published after October 2020 will state that Python>=3.11 is required

All right, thanks for the clarification.

Unfortunately that doesn't fit my needs, so I'll find another way to implement the parser.

Remember that it is not necessary to embed a parser into a software project. It is perfectly possible to use the latest TatSu over the latest Python through a REST-alike service.

That would require internet access and keeping up a server just to parse files, it's a little bit extreme.

I think we can close this issue now that the version support policy is clear.

from tatsu.

Just to be clear, I will keep maintaining TatSu 5.7.x releases compatible with Python 3.8 as long as Python 3.8 is relevant. Only the 5.8.x releases will require more recent Python versions. TatSu 5.7.4 and 5.8.3 are virtually identical. Bug fixes (and features, when possible) will be backported from the 5.8.x releases to the 5.7.4 releases. Thus, if TatSu 5.7.4 does all what you need, the only Python version not covered by the release policy is Python 3.7.

from tatsu.

Any changes made to TatSu in the future will be optimizations and enhancements (for example, the new strategy for code generation inspired by the new Python PEG parser, which will not break the old strategy).

@apalala This is interesting. Is there any published work toward this?

I have a couple of ideas for enhancements. I'll try to find time to put together some concrete proposals or patches.

That would be good!

from tatsu.

This is decided, so I'm closing the issue.

The current version of TatSu was teste with TOX back to 3.8. Now the tests will be from 3.11 onwards, as 3.10 is already deprecated.

from tatsu.