Synthtool is a library of common templates and code that makes generating source code easier.
See the inner README for more details.
License: Apache License 2.0
Synthtool is a library of common templates and code that makes generating source code easier.
See the inner README for more details.
Trying to use private repos is prompting for usernames. This fails for 2FA. Perhaps it should be using ssh keys.
synthtool
synthtool > Executing /Users/crwilcox/workspace/private-java/google-cloud-clients/google-cloud-spanner/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:2f6b261ee7fe1aedf238991c93a20b3820de37a343d0cacf3e3e9555c2aaf2ea
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning private-repo.
Username for 'https://github.com': ^C
As these tools get new fun features, I'd love to get a notification when there's an update. Something similar to update-notifier would be rad.
Today, you can set SYNTHTOOL_GOOGLEAPIS
and it'll use a local version of googleapis:
27: LOCAL_GOOGLEAPIS: Optional[str] = os.environ.get("SYNTHTOOL_GOOGLEAPIS")
It adds a --volume
to the docker invocation of the googleapis/artman Docker image.
The way you specify a local googleapis or GAPIC generator with the Docker artman image is by providing -v
or --volume
s which map these local directories on your PC to directories inside of the container.
If you provide your own path for /googleapis
, then it will mount your local path to googleapis, else it will pull down the latest https://github.com/googleapis/googleapis repo and use that.
If you provide your own path for /toolkit
, then it will mount your local path to gapic-generator, else it will pull down the latest https://github.com/googleapis/gapic-generator repo and use that.
Add the following to support a SYNTHTOOL_GAPIC_GENERATOR
environment variable :)
28: LOCAL_GAPIC_GENERATOR: Optional[str] = os.environ.get("SYNTHTOOL_GAPIC_GENERATOR")
And pass it along as -v [THIS PATH]:/toolkit
(see how we pass /googleapis
as an example)
This is important so that we can run synthool to generate libraries using a LOCAL copy of the GAPIC generator :)
Sometimes we need to test using a version of GAPIC generator which has not yet been deployed in a new googleapis/artman DockerHub image.
artman.py
we don't actually map to /googleapis
, we use -w
to set our googleapis directory to be the working directory.-v f"{root_dir}:{root_dir}",
"-w", root_dir
-v f"{local_gapic_generator}:/toolkit"
As an example, here is the 'Usage' for how to invoke googleapis/artman with options:
docker pull googleapis/artman
googleapis/gapic-generator
(output to current directory) docker run --rm \
--workdir /googleapis \
--volume `pwd`:/googleapis/artman-genfiles \
--env RUNNING_IN_ARTMAN_DOCKER=True \
googleapis/artman /bin/bash -c \
"artman --local --config /googleapis/google/cloud/kms/artman_cloudkms.yaml generate python_gapic"
--volume LOCAL_PATH_TO_GOOGLEAPIS:/googleapis
--volume LOCAL_PATH_TO_GAPIC_GENERATOR:/toolkit
--volume OUTPUT_DIR:/googleapis/artman-genfiles
(override where the output is written)It presently checks every time gapic is run.
Right now we need to go edit a file for autosynth to pick up new repos. This is error prone, and will lead to us forgetting things. Lets update autosynth to just run against any repo in in the org that has a synth.py
in it.
Linters (Flake8 and Black) don't run by default on synth.py
scripts.
Node.js templates have kokoro configs that need to be executable, but we currently can't keep the +x permission when copying over the templates to the destination.
We have a few weird npm modules that don't have links to things like reference API docs or product docs. An example of that is here:
https://github.com/googleapis/nodejs-proto-files/pull/171/files
If those fields aren't defined in .repo-metadata.json
, the bullet points are still rendered into the README, just with a dead link. Instead, synthtool should either:
To apply a git-style patch from a file on the filesystem. E.g.:
s.patch(patch_file='filename.patch')
Maybe it could also take the patch as a triple-quoted string:
patch = '''\
--- a/spanner/google/cloud/spanner_v1/gapic/spanner_client_config.py
+++ b/spanner/google/cloud/spanner_v1/gapic/spanner_client_config.py
@@ -83,7 +83,7 @@ config = {
"retry_params_name": "default"
},
"PartitionQuery": {
- "timeout_millis": 3600000,
+ "timeout_millis": 30000,
"retry_codes_name": "idempotent",
"retry_params_name": "default"
},
'''
s.patch(patch=patch)
Noticed in this pull request: https://github.com/googleapis/nodejs-bigquery/pull/411/files
That we handle acronyms poorly during README generation -- perhaps if we see several capital letter consecutively we could skip title-casing and treat it as an acronym.
We've been making a lot of changes to the templating in the nodejs dir for synth. Needing to publishing a new version of synthtool to pypi every time is just another step we tend to forget. Lets move autosynth over to install directly from GitHub so we always get the latest.
Adding a tracking issue for setting up tests with CI on the repo
When writing templates, files are left without newline character at the end of the file, even though the template file had it.
When autosynth runs and opens a PR, it's a beautiful thing. But what happens when it fails? Some of us are great at watching the logs and checking in to make sure it ran. Others (ok, me) are awful at that and will just forget. In the event that autosynth fails for a given repository, it would be very nice to open a bug on that repo with a copy of the log, and make it super clear it failed. That way I can sleep at night knowing that if it failed, at least I know about it.
Java apiary and Elixir synth scripts expect to have access to argv
. This doesn't work well with running synthtool instead of the synth script (#130).
We need some means for access the remainder arguments, like Nox's session.args
, perhaps synthtool.args
.
https://github.com/googleapis/google-api-nodejs-client has a fairly bespoke README.md file, which we don't want to be repeatedly rewritten when synthtool
is run on the repo. We should provide a mechanism for turning off README generation.
We should run a generation for each language synthtool supports as part of CI to make sure synthesis succeeds.
Currently , the flow is to checkout googleapis (or googleapis-private) and from there generate. There exist cases where pointing this to a locally modified version would be useful.
Occasionally we need to exclude patterns. being able to glob the exclude array would be useful.
Likely this just needs to use expand_paths
https://github.com/GoogleCloudPlatform/synthtool/blob/master/synthtool/transforms.py#L121
After running releasetool tag
, I am getting the following error:
Which one do you want to tag and release?: 1
> Determining what the release tag should be.
Release tag is synthtool-0.3.1.
> Determining the package name and version.
Package name: synthtool, package version: 0.3.1.
> Grabbing the release notes.
> Creating the release.
Release is at https://github.com/GoogleCloudPlatform/synthtool/releases/tag/synthtool-0.3.1
> Publishing to PyPI.
warning: check: missing required meta-data: url
Traceback (most recent call last):
File "/Users/beckwith/Library/Python/3.6/bin/releasetool", line 11, in <module>
load_entry_point('gcp-releasetool==0.2.1', 'console_scripts', 'releasetool')()
File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/releasetool/main.py", line 56, in tag
return releasetool.commands.tag.python_tool.tag()
File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/releasetool/commands/tag/python_tool.py", line 88, in tag
publish_to_pypi(ctx)
File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/releasetool/commands/tag/python_tool.py", line 69, in publish_to_pypi
subprocess.check_call(['twine', 'upload'] + dists)
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 286, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 267, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'twine': 'twine'
Could we upload the docs somewhere and link to from the README and repo configuration?
I find myself referencing the code to figure out what I can do for operations (move, replace, etc.).
(let's take this offline. hush hush)
In google-cloud-python, PRs generated by robots are labeled codegen
. It would be nice to have the option for additional labels to be added on a per-language basis.
This was part of a request made for PR labeling in #138.
Currently it's possible to run both python3 synth.py
and python3 -m synthtool
/synthtool
. We should eliminate one of these, and I vote that we keep the latter.
I am having trouble using synthtool, and I wanted to check the version. It made me realize I'd really like to have this installable as an executable instead of needing to invoke it with python3 synth.py
.
I'd rather just run synthtool
and have it pick up the synth.py
automatically. I would also like a synthtool --version
command :)
The 'bot submitted nineteen PRs to 'google-cloud-python' this morning, of which eighteen consisted of nothing more than changes to 'synth.metadata'. src/apply-change.ts
should exclude synth.metada
from the list of candidate changes triggering PR creation.
In order to know that replacement patterns in a synth script aren't active anymore a warning on the PRs sent my autosynth would be very helpful.
Allo! The repos we oversee will frequently receive multiple-file updates where docs and method names change subtly. The code changes aren't in the part of the API we normally interact with, and sometimes, we're not familiar with the API in general, if it's completely autogenerated. I can't understand those changes as well as the author(s), and I'm concerned I could easily mistake something as just a straightforward change, and miss the part in the code where, e.g. "templateNamePath" is now "templateNamesPath", and break someone's code in the next release.
In the best case scenario, our repos would receive PRs from synthtool that use a title that describes exactly what has changed, and indicates if it's a breaking change, feature, or bug fix. I would guess that's tricky due to multiple commits being combined into a single synchronization commit. However, as a starting point, could the description and commit include the messages of the combined commits to the source repo? Other ideas welcome as well.
Running on Mac. asdf is managing python version (3.6.6). Tried rebooting and deleting cache.
Any insight would be greatly appreciated.
synthtool > Failed executing /Users/paye/.cache/synthtool/artman_venv/bin/pip install --upgrade googleapis-artman:
Collecting googleapis-artman
Using cached https://files.pythonhosted.org/packages/8c/18/d58e5ceb61f4f24ed883d94ae88d1e8e151ca2998790c594bacbf68e4df0/googleapis-artman-0.15.7.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/v4/l0qwz2tx6pb3tj97c8pjmtph00hyhq/T/pip-install-589ui9sp/googleapis-artman/setup.py", line 29, in <module>
with io.open(os.path.join(cur_dir, 'Dockerfile')) as dockerfile:
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/v4/l0qwz2tx6pb3tj97c8pjmtph00hyhq/T/pip-install-589ui9sp/googleapis-artman/Dockerfile'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/v4/l0qwz2tx6pb3tj97c8pjmtph00hyhq/T/pip-install-589ui9sp/googleapis-artman/
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Traceback (most recent call last):
File "synth.py", line 26, in <module>
gapic = gcp.GAPICGenerator()
File "/Users/paye/.local/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 90, in __init__
self._install_artman()
File "/Users/paye/.local/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 220, in _install_artman
f"googleapis-artman{version_specifier}",
File "/Users/paye/.local/lib/python3.6/site-packages/synthtool/shell.py", line 34, in run
raise exc
File "/Users/paye/.local/lib/python3.6/site-packages/synthtool/shell.py", line 28, in run
encoding="utf-8",
File "/Users/paye/.asdf/installs/python/3.6.6/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '[PosixPath('/Users/paye/.cache/synthtool/artman_venv/bin/pip'), 'install', '--upgrade', 'googleapis-artman']' returned non-zero exit status 1.
synthtool > Cleaned up 0 temporary directories.```
If a replacement doesn't return a match there is a good chance the defect we were patching has been fixed.
synth.metadata
.This morning I woke up to ~60 PRs that all follow the same pattern:
https://github.com/googleapis/nodejs-iot/pull/89/files
The PR added the following files (which shouldn't be added):
__pycache__/synth.cpython-36.pyc
And checked in a change to the synth metadata. Guessing these PRs shouldn't have happened :)
Downloading the googleapis/artman:
Docker image can take a long time. Beyond the initial "Pulling artman image", synthtool gives feedback on what is occurring, which has led me to Ctrl-C it, assuming that the hub might be down, etc.
Using stdout=PIPE
swallows output from subprocesses, which is not ideal for something like docker pull
.
At a minimum, it should apply the correct api: *
label to the PR. Even better would be to add the label which indicates the issue is autogen (in google-cloud-python
, we use codegen
).
For debugging and repro purposes, it's very helpful to know exactly which version of googleapis/discovery-artifacts-manager/artman were used by a run of synthtool.
Context: I'm just trying to recreate googleapis/google-cloud-java#4527 locally, but I often have to try several different commit hashes before I can find the right commit hash for the discovery-artifacts-manager version used.
I'd like to be able to generate Python code samples via synthtool, but as of googleapis/gapic-generator#2625, this requires an additional --dev_samples
argument to artman.
I see that Artman.run
accepts an arbitrary number of additional flags to artman via
synthtool/synthtool/gcp/artman.py
Line 70 in c258635
but, there is no way to pass extra parameters from gapic.py_library
.
See: googleapis/google-cloud-python#6880
Running synthtool
locally in logging
against the current master
of google-cloud-python
produces no output except synth.metadata
.
See: googleapis/google-cloud-python#8174
Asserting a coverage level for unit tests is not useful: we actually care only that the overall coverage reaches the goal (which should always be 100%, but that is a different issue).
Having the synth.py
files pass through a "majyk number" for the threshold creates a landmine for future changes.
Very often I have to manually edit src/index.js
when new version of an API is being pulled in. I'd like that to be templated, however the index.js
file for each API is different. Here is an example from Cloud Vision:
/**
* @type {object}
* @property {constructor} ImageAnnotatorClient
* Reference to {@link v1.ImageAnnotatorClient}
*/
module.exports.v1 = gapic.v1;
/**
* @type {object}
* @property {constructor} ImageAnnotatorClient
* Reference to {@link v1p1beta1.ImageAnnotatorClient}
*/
module.exports.v1p1beta1 = gapic.v1p1beta1;
/**
* @type {object}
* @property {constructor} ImageAnnotatorClient
* Reference to {@link v1p2beta1.ImageAnnotatorClient}
*/
module.exports.v1p2beta1 = gapic.v1p2beta1;
Here we have a reference to ImageAnnotatorClient in the jsdoc section of the code, and it is generated based on gapic configs (I think?).
The templating strategy could be:
src/index.js
using artman for a single versionsrc/index.js
In node.js land, we have this problem where we have mostly the same stuff across ~40 repositories (and growing). I would like to be able to splat a set of templated / generated files into every repo.
I think synthtool might? be the right way to do that. Specifically, here are a few things we need:
These files need to support templates with mixins. For example, I should be able to create a .gitignore.tpl
that contains additional ignores I want to add over the default.
In some cases, we may want to support mixins. For example - today we generate a README.md
file using repo-tools that uses basic metadata. You can see an example of the data file here:
https://github.com/googleapis/nodejs-spanner/blob/master/.cloud-repo-tools.json
And the README it generates here:
https://github.com/googleapis/nodejs-spanner/blob/master/README.md
It's important that we can define mixins for individual portions in the event that we want to add additional information. For example here's an issue from cloud-debug-nodejs:
GoogleCloudPlatform/nodejs-repo-tools#123
Ideally, we could define:
The command that re-generates these files should be run as part of autosynth, or some other periodic tool that performs all of the updates via pull requests that occur automagically after PRs to synthtool with template changes are accepted.
when you add the new samples_body partial, ensure that it shows up in the TOC
CC: @JustinBeckwith, @bcoe
Related to:
#246
googleapis/nodejs-translate#268
Synthtool is now creating a synth.metadata
file next to the synth.py script. For apiary clients, we reuse the same synth.py
for each of the n clients because new clients come in and generated automatically.
We should either skip metadata for these clients, or be able to provide a path to the metadata file.
This usage pattern for fixups:
s.replace(
'**/*.py',
<source string>,
<target string>,
)
blows up for unrelated files (e.g., under .nox/
) which are not writeable (e.g., symlinks into non-local Python installs).
We could:
replace
so that it skips over files which aren't writeable.root
to be used for expanding the globs.root
isn't needed?currently synthtool have envvars that can be used to do advanced things. One such feature is export SYNTHTOOL_GOOGLEAPIS=~/.cache/synthtool/googleapis-private
which supports using a local git checkout. There are others such as SYNTHTOOL_ARTMAN_VERSION
or AUTOSYNTH_USE_SSH
.
Improving the documentation for these could assist users.
Encountered this issue generating the nodejs-talent library today (caught indirectly by Node.js's docs-test).
In the latest upstream commit to googleapis (googleapis/googleapis@5fe8a4f), ResumeService
was removed entirely from the proto definitions. I know at least in Node.js and Python, the files associated with ResumeService
still lingers in the repo.
I had to remove the gapic and protos directories and re-run synthtool locally to me to see that the files are being deleted.
This should not be a breaking behaviour, because the removed service client is removed from the exports in {version}/index.js
, so users would not be able to require
the client anyways, unless as @busunkim96 points out if they do weird, unsupported things like requiring the client using the direct file system path.
We're currently adding a bunch of files in addition to those generated by synthtool, so a rm -rf * && synthtool
is out of the question.
Proposals:
Record files generated in synthtool.metadata
and if a new generation doesn't yield files previously generated, then remove them.
Specifically target generated gapic directories, knowing that we do not add additional files to them and generate into the directory from a clean state.
See: googleapis/google-cloud-python#7565
The bug was introduced in PR #206, in _generic_library
:
def _generic_library(self, directory: str, **kwargs) -> Path:
self._load_generic_metadata(kwargs["metadata"])0
That keyword is only present (added explicitly) in node_library
. The fix is to use kwargs.get("metadata", some_appropriate_default)
.
README.md still shows internal sso:// link, let's change that to this repo.
see:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.