jazzband / docopt-ng Goto Github PK
View Code? Open in Web Editor NEWHumane command line arguments parser. Now with maintenance, typehints, and complete test coverage.
License: MIT License
Humane command line arguments parser. Now with maintenance, typehints, and complete test coverage.
License: MIT License
As mentioned in #54 by @NickCrews and @Sylvain303 , I'm splitting of the issue to ensure it doesn't get lost.
Having used the original docopt - so excuse me if you already support this.
I would really love for a special argument that just dumps a structured (json / yaml) representation of the argument parser, that easily allows me to check what is parsed and wether changes I did might have changed the way args will be parsed.
This could also allow diffing between the way stuff is parsed in versions (as a kind of acceptance tests).
That is something I always missed when using the original docopt, that I always just used very rarely and was having a lot of trouble remembering the intricacies of the docopt configuration language.
@dwt that feature makes sense, and should be theoretically be possible.
If you want to discuss this further, please make a new feature request issue, so this thread doesn't go off topic. But in short:
I am not going to implement that (don't have the time time or interest), but I would consider merging a PR if
I don't know of any common json schema that would work for this. I would love to not reinvent the wheel here, so would be worth it to explore what other. It also seems to me like "checking for a change in parsing behavior" would not be that useful on its own. If the behavior started off incorrect, you aren't actually catching that. It seems more useful to actually test assert my_parse("myprogram new --name foo") == , even if it might be a bit more verbose. Also would serve as documentation.
Hello, I'm not very active yet on docopts coding (golang version for bash) watch-out the extra 'S'. But I
try to implement such thing: dumping what was actually parsed by docopt parser.
Could you create an new issue with some sample of what you would love to see outputted? That would be very useful. 🤩
To be honest I do not have a precise idea what the correct output format would be. My initial idea was, that all implementations should be able to dump that exact same parse tree, to then be able to compare them to each other and have them act as a regression test suite for each other.
However that use case would be mostly about machines being able to compare this, not necessarily humans. I guess some json/yaml, that just documents all the options would be mostly fine here? The format should be as simple as possible to ease interoperability.
The other use case I had for myself is that it can be hard for a casual user to actually write the correct docopt syntax, as it has it's intricacies that can be hard to get right.
To help with that, I'm not quite sure what would help best? To me generating the source for an equivalent argparse based parser would be quite helpful, but having something declarative gives me a better belly feeling.
To be honest, the first thing I would do if this was my project is probably to just dump the tree of the internal data structure that docopt-ng uses to then parse command lines, then take a look at that and see what can be achieved from there. This would already give you internal comparability between different versions of the parser and would enable regression testing with easy regeneration of the fixtures. Then tweak from there to make it useful for users without big knowledge of the internals and propose that to other implementations to make them comparable.
Does that make sense?
Some thoughts about what this dump format should contain
docopt_format: "Naval Fate …" # what was parsed
program_name: naval_fate
alternatives:
- marker: ship new
positional:
- name: name
repeats: infinite
- marker: ship
positional:
- name: name
rest_alternatives:
- marker: move
positional:
- name: x
- name: y
named:
- flag_long: speed
name: kn
Obviously this format idea is very incomplete and probably unusable for several reasons, but that's what is floating in my mind.
After a few days and maybe dropping in docopt-ng for some popular projects that use docopt, I'm hoping to do a 1.0 release - slightly breaking semver, but with 100% coverage I think the project deserves to be pulled out of "beta".
This issue tracks the implementation of the Jazzband guidelines for the project docopt-ng
It was initiated by @itdaniher who was automatically assigned in addition to the Jazzband roadies.
See the TODO list below for the generally required tasks, but feel free to update it in case the project requires it.
Feel free to ping a Jazzband roadie if you have any question.
README
fileCONTRIBUTING.md
or CONTRIBUTING.rst
filejazzband
account to PyPI project as maintainer role (e.g. URL: https://pypi.org/manage/project/docopt-ng/collaboration/)jazzband-bot
as maintainer to the Read the Docs project (e.g. URL: https://readthedocs.org/dashboard/docopt-ng/users/)Description | Humane command line arguments parser. Now with maintenance, typehints, and complete test coverage. |
Homepage | |
Stargazers | 36 |
Open issues | 9 |
Forks | 6 |
Default branch | master |
Is a fork | False |
Has Wiki | False |
Has Pages | False |
regarding the changelog, there should be smarter error messages and an autocorrect function, but still when iam using more_magic=True
I will get only the usage message returned and a meaningless message already reported in #5
For the following option definition
-c --count=<int> Option for a count of some kind [default: 42]
Is there a way to retrieve the default value ( 42
) and the value of the value designation (<int>
) programatically? I.e. is the a way to get from the resulting args
after docopt-ng is done parsing? I have an oustanding PR in original docopt for that sort of change but, as well all know, that's not going to be merged.
I have started using docopt-dispatch and I find it very intriguing. It makes docopt CLI code super-concise and readable.
Unfortunately, that package both has docopt as a hard dependency and is unmaintained for 7+ years.
If we integrated the dispatching project in docopt-ng some CLI code could look like this, preserving the compatibility with the original docopt spec and staying aligned with the syntax of the original dispatch project:
"""Run something in development or production mode.
Usage: run.py --development <host> <port>
run.py --production <host> <port>
run.py remote add <item>
run.py remote delete <item>
"""
from docopt import dispatch
@dispatch.on('--development')
def development(host, port, **kwargs):
print('in *development* mode')
@dispatch.on('--production')
def development(host, port, **kwargs):
print('in *production* mode')
if __name__ == 'main':
dispatch(__doc__)
Would you accept a PR that adds the dispatching feature to docopt-ng?
This is a broad topic that I've been thinking about recently, that has come up in several places, such as #4 and #45
What does the community want for the future of docopt-ng?
I have so far been pursuing 1. This was because
a function that takes a usage message and sys.argv, and returns a dict
. It's well defined, has a narrow interface but a complex implementation, and composes well. In release 0.9.0 I even removed the magic
stuff in pursuit of this (see the changelog for rationale)docopt-dispatch
could be implemented as an extensionI would love to hear people's opinions here. I don't have a lot of open source maintainership experience, so I apologize for any frustration that is caused by that.
Cheers,
Nick
Not seeing any references to an actual docopt formal grammar yet, I am writing a PEG grammar for use in Python with Tatsu (https://tatsu.readthedocs.io). I am hopeful that it will be usable with other PEG parser generators but am not well enough versed in the area to know by how much they differ in syntax.
I would like to work with others interested in developing a formal grammar for docopt. I'm not fixed on Tatsu or even PEG though I suspect PEG grammars will provide more accurate results for the corner cases.
I appended a sketch of the steps I believe necessary to develop the grammar. I've written a grammar for the usage examples section. Now I need to take a step back and write the frame work to snag each major section. Divide and conquer seemed like the best approach. In a day or so, I could post the grammar and test cases to github.
So, any information in this area or would anyone like to collaborate ?
Philip
Or, a formal grammar for the docopy language.
First identify which elements must be parsed and which (if any) may be discarded (or perhaps remain free form text).
Characterize the delineations of each section sufficiently to write regular expression matchers for their start and end, or span.
Write a grammar to parse to just the sections out as blocks of free form text. This forms the framework within which the section parsers will operate.
For each section independently write a grammar to parse the it.
One by one, incorporate the section parsers into the framework.
Hello!
Similar to this StackOverflow message I've got the same error while trying to (indirectly) use docopt-ng with Python 3.11.
I've added print(instr.opname)
right after the line 941 and the resulting list of instructions indeed doesn't have anything stareted with CALL_ (see file atached). On the contrary here is a move shorter list of instructions when I run the same with Python 3.10:
LOAD_GLOBAL
LOAD_GLOBAL
LOAD_CONST
LOAD_CONST
CALL_FUNCTION_KW
If I remove the following block completely along with output_value_assigned
variable then everything seems to work just fine with Python 3.11:
Lines 937 to 950 in 130a542
Picture:
Probably something needs to be followed from https://packaging.python.org/guides/making-a-pypi-friendly-readme/
maybe long_description_content_type = "text/markdown"
in setup()
I removed the public interface to the spellchecking in https://github.com/jazzband/docopt-ng/releases/tag/0.9.0, but that spellchecking code still exists.
I think that code is still useful, but it should be pivoted. Instead of auto-correcting mispelled args (which might be dangerous, much better to be strict), the parse should still fail. BUT, we can still extract useful help messages from this, and say "perhaps you meant XXXX".
This issue tracks that progess
Hello
This docstring was working before release docopt-ng==0.9.0
Note only the Usage:
+ up to the empty line should be parsed by docopt language. Unless you added new section name handling?
By the way the error message doesn't help to figure out what is wrong.
#!/usr/bin/env python
"""
prepare an upgrade archive for upgrade_instance.py
Usage:
./prepare_upgrade.py create UPGRADE_FOLDER [--new]
./prepare_upgrade.py generate_sql DATE --sql-repos=INPUT_DIR_SQL [UPGRADE_DIR]
./prepare_upgrade.py fetch_releases RELEASE_PIPELINE [UPGRADE_DIR]
./prepare_upgrade.py show [UPGRADE_DIR]
./prepare_upgrade.py build [--upgrade-folder=UPGRADE_DIR] [--dest-zip=UPGRADE_ZIP]
./prepare_upgrade.py add (--sql|--repair) SQL_FILE [UPGRADE_DIR]
./prepare_upgrade.py convert_to_dep_txt [--old] [UPGRADE_DIR]
./prepare_upgrade.py tree [UPGRADE_DIR]
./prepare_upgrade.py install_helpers [--clean] [UPGRADE_DIR] [--ext_dir=INFRASTRUCTURE_DIR]
Env var:
UPGRADE_DIR if set, replace the UPGRADE_DIR parameter, so it can be ommited. If
both are present, parameter will be chosen.
Actions:
create Create an empty folder in UPGRADE_FOLDER (parent) that will contains
the upgrade tree Output the foldername suitable for UPGRADE_DIR env var
assignment.
--new will instruct to create a new_one if the folder exists with
the current date: upgrade_YYYY-MM-DD.2 (the last digit will
increase etc.)
generate_sql Take the SQL file in INPUT_DIR_SQL and filter from DATE included
in order to generate an sql folder tree into UPGRADE_DIR
fetch_releases Connect to VSTS to fetch the releases in prod based on RELEASE_PIPELINE.
This is a slow operation that take ~40s to complete. The outout yaml file
will be saved in UPGRADE_DIR.
show Explain what is in UPGRADE_DIR and validate.
build Create the zip archive UPGRADE_ZIP based on the content of the
UPGRADE_DIR. If UPGRADE_ZIP is not given the zip will be generated
in to the parent forled (aka: UPGRADE_FOLDER)
add Add a new sql file manually to the UPGRADE_DIR. In sql/ dir if --sql
at the end with continuous order. In repair/ if --repair, with no
specific order.
convert_to_dep_txt For internal transitional conversion dep_version.txt format from releases.yaml.
Outputs to stdout.
tree Display UPGRADE_DIR as a tree view.
install_helpers Copy helpers into UPGRADE_DIR. If --clean is present remove all helper first.
"""
from docopt import docopt
args = docopt(docstring=__doc__)
print(args)
python bug_docopt-ng.py create prepare_upgrade 2>&1 | sed -e 's/cloudiway/mycode/'
Traceback (most recent call last):
File "/home/sylvain/code/mycode/upgrade_instance/bug_docopt-ng.py", line 47, in <module>
args = docopt(docstring=__doc__)
File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 913, in docopt
pattern = parse_pattern(formal_usage(sections.usage_body), options)
File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 608, in parse_pattern
result = parse_expr(tokens, options)
File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 617, in parse_expr
seq_0: list[Pattern] = parse_seq(tokens, options)
File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 638, in parse_seq
atom = parse_atom(tokens, options)
File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 657, in parse_atom
matched_pattern = pattern(*parse_expr(tokens, options))
File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 617, in parse_expr
seq_0: list[Pattern] = parse_seq(tokens, options)
File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 638, in parse_seq
atom = parse_atom(tokens, options)
File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 659, in parse_atom
raise tokens.error("unmatched '%s'" % token)
docopt.DocoptLanguageError: unmatched '['
I would like to set a default value containing spaces like in -
. It does work:
--name-sep=<sep> name field separator [default: - ]
But it is not so pleasing to the eye and prone to bad interpretation. I don't know what is the standard here, but I would imagine accepting a quoted default value, as in:
--name-sep=<sep> name field separator [default: " - "]
Could be one way to go. Or am I missing something ?
After migrating from docopt 0.6.2 to docopt-ng 0.7.2, this help caused an assertion at https://github.com/bazaar-projects/docopt-ng/blob/bbed40a2335686d2e14ac0e6c3188374dc4784da/docopt.py#L616
"""
Evaluate tile generation performance against a PostgreSQL database.
Usage:
test-perf <tileset> [-t <set>] [-h <host>]
<tileset> Tileset definition yaml file
Options:
-t --test=<set> Which predefined test to run. [default: us-across]
PostgreSQL Connection Options:
-h --pghost=<host> Postgres hostname.
"""
Wenn I run my script without a necessary subcommand, docopt-ng
issues a warning - shouldn't it be an error? - which in my opinion is not suitable for the enduser (found unmatched (duplicate?) arguments [Argument(None, '...')]
.
Even I as the script writer, don't know what that exactly means. It looks to me like docopt-ng
exposes internals from the parsing process which should not be given to the script user.
> python interpreter.py '.\10. PowerShell.ps1'
Warning: found unmatched (duplicate?) arguments [Argument(None, '.\\10. PowerShell.ps1')]
Usage:
interpreter.py run <script> [-- <script_option> ...]
interpreter.py doc [-s <text>] <script>
This little program here:
#!/usr/bin/env python
"""
Usage:
test.py [--to=SITE]... [--] FILE...
test.py [--to=SITE]... --config CONFIG [[--] FILE...]
Options:
--config CONFIG Configuration file.
--to=SITE Target site
"""
import docopt
options = docopt.docopt(__doc__)
print(options)
If I run test.py --to a --to b c
, I get this output:
{'--': False,
'--config': None,
'--to': ['a', 'b', 'b'],
'FILE': ['c']}
Notice the double occurrence of 'b'
If this project is a more maintained version of the original docopt (now unmaintained), I think that this project should claim the pypi docopt name.
I just began using docopt-ng. I was creating my interface from scratch (I currently use a functional but spaghetti-like script), and my docstring read something like:
"""Performous macOS Bundler
Usage:
macos_bundler.py
macos_bundler.py [options]
macos_bundler.py (-h | --help)
Options:
...
"""
But I kept getting the following error and couldn't figure out why.
raise tokens.error("unmatched '%s'" % token)
docopt.DocoptLanguageError: unmatched '('
I double checked brackets and there were definitely no unmatched ones. I then double-checked the documentation to make sure I hadn't made a mistake with the syntax, and I couldn't find one.
I had read sometimes docopt (I think the original one?) had issues with pipes, so even though that exact syntax appeared in the documentation, I tried to remove the macos_bundler.py (-h | --help)
line and, the error went away; but I couldn't use either -h nor --help, because it kept telling me those optons needed an argument.
Upon closer inspection, I noticed I had missed a space between -h --help
and the option description.
So, it seems missing writing something such as
"""Command
Usage:
cli_tool.py (-s | --long)
Options:
-s --long Some description for this option.
"""
Will produce such an error.
Of course, there is an error there, and one the documentation specifically warns against. However, in this particular scenario, the error message was very obscure.
This question was previously asked about docopt, but now docopt-ng seems to have ended up in the same situation - unmerged/unanswered PRs and issues.
It would be really great to have a 'live' docopt for Python.
the idea of forking docopt because it's unmaintained it's cool but the idea of docopt is to be available in tons of languages, while docopt-ng only forked the python implementation.
Do you plan to fork all others https://github.com/docopt?
@NickCrews would you be OK with auto sorting/formatting imports (probably with isort)?
When I was rebasing the error message PR just now, and a few other branches I've got, probably 3/4 of my merge conflicts were just imports. I'm thinking it would help to use isort with the single line option, so that instead of:
from typing import Any, Callable, Tuple, Type, Union, cast
we get:
from typing import Any
from typing import Callable
from typing import Tuple
from typing import Type
from typing import Union
from typing import cast
Which doesn't look as nice, but means import changes shouldn't cause merge conflicts.
https://github.com/jazzband/docopt-ng/blob/master/docopt/__init__.py#L920-L922
Because the message produced if you get to this point is extremely unhelpful, I wanted to inspect what had/hadn't been parsed and produce a more meaningful exception.
But because left
and collected
aren't passed, I've had to do a very nasty bit of code inspecting the stack to find out what they should have been.
Could they be passed in that particular instance so that the client has a chance of producing a more reasonable error (and it might be worth subclassing DocoptError so that it's clear to clients whether docopt has given up as opposed to finding a specific error).
@itdaniher not a big deal at all, but I would love it if I could test installs using the test mirror of PyPI. Could you add @NickCrews as a maintainer to https://test.pypi.org/project/docopt-ng/? Thanks!
Many tools with a lot of options separate them in subsections in the help message; is it possible to do that in doctopt-ng?
Deprecation warning reported by pytest during testing: direct construction is deprecated as of pytest 5.4 and will be removed in the future. Use of the [Node].from_parent constructor is recommended."
example:
$ py setup test
conftest.py:14
.../docopt/conftest.py:14: PytestDeprecationWarning: direct construction of DocoptTestFile has been deprecated, please use DocoptTestFile.from_parent
return DocoptTestFile(path, parent)
conftest.py:44: 175 tests with warnings
.../docopt/conftest.py:44: PytestDeprecationWarning: direct construction of DocoptTestItem has been deprecated, please use
DocoptTestItem.from_parent
yield DocoptTestItem("%s(%d)" % (name, index), self, doc, case)
-- Docs: https://docs.pytest.org/en/latest/warnings.html
Using docopt
for building utilities with relatively wide range of options is pretty limited because of a huge performance penalty.
Namely, a combinatorial explosion may happen in the transform
function: the pattern expansion (like ((-a | -b) (-c | -d)) => (-a -c | -a -d | -b -c | -b -d)
) has unacceptable computational complexity.
A good example would be the GNU ls
utility. See the sample below.
The script below takes almost 3 seconds to run which is terribly slow for just to parse CLI arguments.
"""
ls with a subset of GNU options (that's not even all of them!)
Usage:
ls [-a|-A] [--hide=PATTERN] [-I=PATTERN] [-dLR]
[--color] [-h|--si] [--indicator-style=WORD|-p|-F|--file-type]
[--format=WORD|-x|-m|-x|-l|-1|-C|-g|-n|-o] [-Giks]
[--sort=WORD|-f|-U|-S|-t|-v|-X] [--group-directories-first] [-r]
[--time=WORD|-u|-c] [--time-style=TIME_STYLE]
[FILES ...]
ls --help
ls --version
Arguments:
FILES
list of files
"""
from docopt import docopt
args = docopt()
My project https://github.com/staticjinja/staticjinja uses mypy, but it can't actually use the type hints
defined here because we aren't following https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages
PR to follow
I've got an old CLI program which uses OG docopt. I'm giving it a bit of minor TLC to refresh the tooling, and I tried switching to docopt-ng, but -ng fails to parse my usage string for some reason:
vscode@46fada5e45f4 /w/rnginline ((bef5200c…)) [127]> poetry run ipython
Python 3.10.5 (main, Jun 6 2022, 12:05:50) [GCC 9.5.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import docopt
In [2]: docopt.__version__
Out[2]: '0.8.1'
In [3]: from rnginline import cmdline
In [4]: docopt.docopt(cmdline.__doc__, argv=['rnginline', '--no-libxml2-compat', '/some/file'])
An exception has occurred, use %tb to see the full traceback.
DocoptExit: Warning: found unmatched (duplicate?) arguments [Option(None, '--no-libxml2-compat', 0, True)]
usage: rnginline [options] <rng-src> [<rng-output>]
rnginline [options] --stdin [<rng-output>]
/home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3406: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)
In [5]:
Do you really want to exit ([y]/n)? y
vscode@46fada5e45f4 /w/rnginline ((bef5200c…))> poetry run pip uninstall docopt-ng
Found existing installation: docopt-ng 0.8.1
Uninstalling docopt-ng-0.8.1:
Would remove:
/home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/docopt/*
/home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/docopt_ng-0.8.1.dist-info/*
Proceed (Y/n)? y
Successfully uninstalled docopt-ng-0.8.1
vscode@46fada5e45f4 /w/rnginline ((bef5200c…))> poetry run pip install docopt
Collecting docopt
Using cached docopt-0.6.2-py2.py3-none-any.whl
Installing collected packages: docopt
Successfully installed docopt-0.6.2
vscode@46fada5e45f4 /w/rnginline ((bef5200c…))> poetry run ipython
Python 3.10.5 (main, Jun 6 2022, 12:05:50) [GCC 9.5.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import docopt
In [2]: docopt.__version__
Out[2]: '0.6.2'
In [3]: from rnginline import cmdline
In [4]: docopt.docopt(cmdline.__doc__, argv=['rnginline', '--no-libxml2-compat', '/some/file'])
Out[4]:
{'-': None,
'--base-uri': None,
'--default-base-uri': None,
'--help': False,
'--no-libxml2-compat': True,
'--stdin': False,
'--traceback': False,
'--version': False,
'<rng-output>': '/some/file',
'<rng-src>': 'rnginline'}
This is the usage string: https://github.com/h4l/rnginline/blob/b1d1c8cda2a17d46627309950f2442021749c07e/rnginline/cmdline.py#L14
I really appreciate your efforts in keeping docopt going, It's a great library.
If you install
pipenv install pipreqs pip-upgrader docopt-ng
Which sometimes happens, you can't put everything into system or pipx.
Then apps use the wrong docopt.
$ python
>>> import docopt
>>> docopt.__file__
'C:\\Users\\matth\\.virtualenvs\\demo_doc-j-5bRj6B\\lib\\site-packages\\docopt.py'
>>> exit()
$ pip install docopt-ng
Requirement already satisfied: docopt-ng in c:\users\matth\.virtualenvs\demo_doc-j-5brj6b\lib\site-packages (0.7.2)
In C# there was a way to handle this and ensure that two similarly named dlls weren't confused with each other, I don't know what the python way is to handle this scenario.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.