Trying to use private repos is prompting for usernames. This fails for 2FA. Perhaps it should be using ssh keys.

synthtool   
                                                                                                                                                 
synthtool > Executing /Users/crwilcox/workspace/private-java/google-cloud-clients/google-cloud-spanner/synth.py.
synthtool > Ensuring dependencies.
synthtool > Pulling artman image.
latest: Pulling from googleapis/artman
Digest: sha256:2f6b261ee7fe1aedf238991c93a20b3820de37a343d0cacf3e3e9555c2aaf2ea
Status: Image is up to date for googleapis/artman:latest
synthtool > Cloning private-repo.
Username for 'https://github.com': ^C

As these tools get new fun features, I'd love to get a notification when there's an update. Something similar to update-notifier would be rad.

Today, you can set SYNTHTOOL_GOOGLEAPIS and it'll use a local version of googleapis:

gapic_generator.py

27: LOCAL_GOOGLEAPIS: Optional[str] = os.environ.get("SYNTHTOOL_GOOGLEAPIS")

How does this work?

It adds a --volume to the docker invocation of the googleapis/artman Docker image.

The way you specify a local googleapis or GAPIC generator with the Docker artman image is by providing -v or --volumes which map these local directories on your PC to directories inside of the container.

If you provide your own path for /googleapis, then it will mount your local path to googleapis, else it will pull down the latest https://github.com/googleapis/googleapis repo and use that.

If you provide your own path for /toolkit, then it will mount your local path to gapic-generator, else it will pull down the latest https://github.com/googleapis/gapic-generator repo and use that.

Feature request

Add the following to support a SYNTHTOOL_GAPIC_GENERATOR environment variable :)

gapic_generator.py

28: LOCAL_GAPIC_GENERATOR: Optional[str] = os.environ.get("SYNTHTOOL_GAPIC_GENERATOR")

And pass it along as -v [THIS PATH]:/toolkit (see how we pass /googleapis as an example)

This is important so that we can run synthool to generate libraries using a LOCAL copy of the GAPIC generator :)

Sometimes we need to test using a version of GAPIC generator which has not yet been deployed in a new googleapis/artman DockerHub image.

Follow-up / Implementation Notes

• Reading artman.py we don't actually map to /googleapis, we use -w to set our googleapis directory to be the working directory.
• -v f"{root_dir}:{root_dir}",
• "-w", root_dir
• For a custom GAPIC generator, you would want Artman to have a property for the path to gapic_generator, if locally present, and (if locally present) pass:
• -v f"{local_gapic_generator}:/toolkit"

As an example, here is the 'Usage' for how to invoke googleapis/artman with options:

Usage [googleapis/artman]

1. Pull down latest artman Docker image:

• docker pull googleapis/artman

2. Build a library using its artman config yaml and the latest HEAD googleapis/gapic-generator (output to current directory)

•  docker run --rm                                     \
       --workdir /googleapis                           \
       --volume `pwd`:/googleapis/artman-genfiles      \
       --env RUNNING_IN_ARTMAN_DOCKER=True             \
       googleapis/artman /bin/bash -c                  \
       "artman --local --config /googleapis/google/cloud/kms/artman_cloudkms.yaml generate python_gapic"
  

3. If you’d like to use your own local version of googleapis or gapic-generator, add the following flags to the Docker command:

• --volume LOCAL_PATH_TO_GOOGLEAPIS:/googleapis
• --volume LOCAL_PATH_TO_GAPIC_GENERATOR:/toolkit
• --volume OUTPUT_DIR:/googleapis/artman-genfiles (override where the output is written)

It presently checks every time gapic is run.

Right now we need to go edit a file for autosynth to pick up new repos. This is error prone, and will lead to us forgetting things. Lets update autosynth to just run against any repo in in the org that has a synth.py in it.

Linters (Flake8 and Black) don't run by default on synth.py scripts.

Node.js templates have kokoro configs that need to be executable, but we currently can't keep the +x permission when copying over the templates to the destination.

We have a few weird npm modules that don't have links to things like reference API docs or product docs. An example of that is here:
https://github.com/googleapis/nodejs-proto-files/pull/171/files

If those fields aren't defined in .repo-metadata.json, the bullet points are still rendered into the README, just with a dead link. Instead, synthtool should either:

• Make the fields required and throw if they're not defined or...
• Hide the markup they would be used to generate

To apply a git-style patch from a file on the filesystem. E.g.:

s.patch(patch_file='filename.patch')

Maybe it could also take the patch as a triple-quoted string:

patch = '''\
--- a/spanner/google/cloud/spanner_v1/gapic/spanner_client_config.py
+++ b/spanner/google/cloud/spanner_v1/gapic/spanner_client_config.py
@@ -83,7 +83,7 @@ config = {
                     "retry_params_name": "default"
                 },
                 "PartitionQuery": {
-                    "timeout_millis": 3600000,
+                    "timeout_millis": 30000,
                     "retry_codes_name": "idempotent",
                     "retry_params_name": "default"
                 },
'''
s.patch(patch=patch)

Noticed in this pull request: https://github.com/googleapis/nodejs-bigquery/pull/411/files

That we handle acronyms poorly during README generation -- perhaps if we see several capital letter consecutively we could skip title-casing and treat it as an acronym.

We've been making a lot of changes to the templating in the nodejs dir for synth. Needing to publishing a new version of synthtool to pypi every time is just another step we tend to forget. Lets move autosynth over to install directly from GitHub so we always get the latest.

Adding a tracking issue for setting up tests with CI on the repo

When writing templates, files are left without newline character at the end of the file, even though the template file had it.

When autosynth runs and opens a PR, it's a beautiful thing. But what happens when it fails? Some of us are great at watching the logs and checking in to make sure it ran. Others (ok, me) are awful at that and will just forget. In the event that autosynth fails for a given repository, it would be very nice to open a bug on that repo with a copy of the log, and make it super clear it failed. That way I can sleep at night knowing that if it failed, at least I know about it.

Java apiary and Elixir synth scripts expect to have access to argv. This doesn't work well with running synthtool instead of the synth script (#130).

We need some means for access the remainder arguments, like Nox's session.args, perhaps synthtool.args.

https://github.com/googleapis/google-api-nodejs-client has a fairly bespoke README.md file, which we don't want to be repeatedly rewritten when synthtool is run on the repo. We should provide a mechanism for turning off README generation.

We should run a generation for each language synthtool supports as part of CI to make sure synthesis succeeds.

Currently , the flow is to checkout googleapis (or googleapis-private) and from there generate. There exist cases where pointing this to a locally modified version would be useful.

Occasionally we need to exclude patterns. being able to glob the exclude array would be useful.

Likely this just needs to use expand_paths
https://github.com/GoogleCloudPlatform/synthtool/blob/master/synthtool/transforms.py#L121

After running releasetool tag, I am getting the following error:

Which one do you want to tag and release?: 1
> Determining what the release tag should be.
Release tag is synthtool-0.3.1.
> Determining the package name and version.
Package name: synthtool, package version: 0.3.1.
> Grabbing the release notes.
> Creating the release.
Release is at https://github.com/GoogleCloudPlatform/synthtool/releases/tag/synthtool-0.3.1
> Publishing to PyPI.
warning: check: missing required meta-data: url

Traceback (most recent call last):
  File "/Users/beckwith/Library/Python/3.6/bin/releasetool", line 11, in <module>
    load_entry_point('gcp-releasetool==0.2.1', 'console_scripts', 'releasetool')()
  File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/releasetool/main.py", line 56, in tag
    return releasetool.commands.tag.python_tool.tag()
  File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/releasetool/commands/tag/python_tool.py", line 88, in tag
    publish_to_pypi(ctx)
  File "/Users/beckwith/Library/Python/3.6/lib/python/site-packages/releasetool/commands/tag/python_tool.py", line 69, in publish_to_pypi
    subprocess.check_call(['twine', 'upload'] + dists)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 286, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 267, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'twine': 'twine'

Could we upload the docs somewhere and link to from the README and repo configuration?

I find myself referencing the code to figure out what I can do for operations (move, replace, etc.).

(let's take this offline. hush hush)

In google-cloud-python, PRs generated by robots are labeled codegen. It would be nice to have the option for additional labels to be added on a per-language basis.

This was part of a request made for PR labeling in #138.

Currently it's possible to run both python3 synth.py and python3 -m synthtool/synthtool. We should eliminate one of these, and I vote that we keep the latter.

I am having trouble using synthtool, and I wanted to check the version. It made me realize I'd really like to have this installable as an executable instead of needing to invoke it with python3 synth.py.

I'd rather just run synthtool and have it pick up the synth.py automatically. I would also like a synthtool --version command :)

The 'bot submitted nineteen PRs to 'google-cloud-python' this morning, of which eighteen consisted of nothing more than changes to 'synth.metadata'. src/apply-change.ts should exclude synth.metada from the list of candidate changes triggering PR creation.

In order to know that replacement patterns in a synth script aren't active anymore a warning on the PRs sent my autosynth would be very helpful.

Allo! The repos we oversee will frequently receive multiple-file updates where docs and method names change subtly. The code changes aren't in the part of the API we normally interact with, and sometimes, we're not familiar with the API in general, if it's completely autogenerated. I can't understand those changes as well as the author(s), and I'm concerned I could easily mistake something as just a straightforward change, and miss the part in the code where, e.g. "templateNamePath" is now "templateNamesPath", and break someone's code in the next release.

In the best case scenario, our repos would receive PRs from synthtool that use a title that describes exactly what has changed, and indicates if it's a breaking change, feature, or bug fix. I would guess that's tricky due to multiple commits being combined into a single synchronization commit. However, as a starting point, could the description and commit include the messages of the combined commits to the source repo? Other ideas welcome as well.

Running on Mac. asdf is managing python version (3.6.6). Tried rebooting and deleting cache.
Any insight would be greatly appreciated.

synthtool > Failed executing /Users/paye/.cache/synthtool/artman_venv/bin/pip install --upgrade googleapis-artman:

Collecting googleapis-artman
  Using cached https://files.pythonhosted.org/packages/8c/18/d58e5ceb61f4f24ed883d94ae88d1e8e151ca2998790c594bacbf68e4df0/googleapis-artman-0.15.7.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/v4/l0qwz2tx6pb3tj97c8pjmtph00hyhq/T/pip-install-589ui9sp/googleapis-artman/setup.py", line 29, in <module>
        with io.open(os.path.join(cur_dir, 'Dockerfile')) as dockerfile:
    FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/v4/l0qwz2tx6pb3tj97c8pjmtph00hyhq/T/pip-install-589ui9sp/googleapis-artman/Dockerfile'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/v4/l0qwz2tx6pb3tj97c8pjmtph00hyhq/T/pip-install-589ui9sp/googleapis-artman/
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Traceback (most recent call last):
  File "synth.py", line 26, in <module>
    gapic = gcp.GAPICGenerator()
  File "/Users/paye/.local/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 90, in __init__
    self._install_artman()
  File "/Users/paye/.local/lib/python3.6/site-packages/synthtool/gcp/gapic_generator.py", line 220, in _install_artman
    f"googleapis-artman{version_specifier}",
  File "/Users/paye/.local/lib/python3.6/site-packages/synthtool/shell.py", line 34, in run
    raise exc
  File "/Users/paye/.local/lib/python3.6/site-packages/synthtool/shell.py", line 28, in run
    encoding="utf-8",
  File "/Users/paye/.asdf/installs/python/3.6.6/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '[PosixPath('/Users/paye/.cache/synthtool/artman_venv/bin/pip'), 'install', '--upgrade', 'googleapis-artman']' returned non-zero exit status 1.
synthtool > Cleaned up 0 temporary directories.```

If a replacement doesn't return a match there is a good chance the defect we were patching has been fixed.

Internal spec.

• Create protos (or JSON-RPC or something) for the Metadata format. (#127)
• Create a means for sources to amend the metadata during generation. (#132 for git sources, #136 for artman, #149 for templates)
• Create a means for generators to amend the metadata during generation (#155).
• Figure out how to deal with FileSets and such (spec updated).
• Output the metadata to synth.metadata.

This morning I woke up to ~60 PRs that all follow the same pattern:
https://github.com/googleapis/nodejs-iot/pull/89/files

The PR added the following files (which shouldn't be added):

• __pycache__/synth.cpython-36.pyc

And checked in a change to the synth metadata. Guessing these PRs shouldn't have happened :)

Downloading the googleapis/artman: Docker image can take a long time. Beyond the initial "Pulling artman image", synthtool gives feedback on what is occurring, which has led me to Ctrl-C it, assuming that the hub might be down, etc.

Using stdout=PIPE swallows output from subprocesses, which is not ideal for something like docker pull.

At a minimum, it should apply the correct api: * label to the PR. Even better would be to add the label which indicates the issue is autogen (in google-cloud-python, we use codegen).

For debugging and repro purposes, it's very helpful to know exactly which version of googleapis/discovery-artifacts-manager/artman were used by a run of synthtool.

Context: I'm just trying to recreate googleapis/google-cloud-java#4527 locally, but I often have to try several different commit hashes before I can find the right commit hash for the discovery-artifacts-manager version used.

I'd like to be able to generate Python code samples via synthtool, but as of googleapis/gapic-generator#2625, this requires an additional --dev_samples argument to artman.

I see that Artman.run accepts an arbitrary number of additional flags to artman via

but, there is no way to pass extra parameters from gapic.py_library.

See: googleapis/google-cloud-python#6880

Running synthtool locally in logging against the current master of google-cloud-python produces no output except synth.metadata.

See: googleapis/google-cloud-python#8174

Asserting a coverage level for unit tests is not useful: we actually care only that the overall coverage reaches the goal (which should always be 100%, but that is a different issue).

Having the synth.py files pass through a "majyk number" for the threshold creates a landmine for future changes.

Very often I have to manually edit src/index.js when new version of an API is being pulled in. I'd like that to be templated, however the index.js file for each API is different. Here is an example from Cloud Vision:

/**
 * @type {object}
 * @property {constructor} ImageAnnotatorClient
 *   Reference to {@link v1.ImageAnnotatorClient}
 */
module.exports.v1 = gapic.v1;

/**
 * @type {object}
 * @property {constructor} ImageAnnotatorClient
 *   Reference to {@link v1p1beta1.ImageAnnotatorClient}
 */
module.exports.v1p1beta1 = gapic.v1p1beta1;

/**
 * @type {object}
 * @property {constructor} ImageAnnotatorClient
 *   Reference to {@link v1p2beta1.ImageAnnotatorClient}
 */
module.exports.v1p2beta1 = gapic.v1p2beta1;

Here we have a reference to ImageAnnotatorClient in the jsdoc section of the code, and it is generated based on gapic configs (I think?).

The templating strategy could be:

• generate src/index.js using artman for a single version
• for each version, take the module.exports... line that correspond to the version (regex?), along with its jsdoc section
• bring it all into the generated src/index.js

In node.js land, we have this problem where we have mostly the same stuff across ~40 repositories (and growing). I would like to be able to splat a set of templated / generated files into every repo.

I think synthtool might? be the right way to do that. Specifically, here are a few things we need:

• .appveyor.yml
• .circleci/config.yml
• .eslintignore
• .eslintrc.yml
• .github/CONTRIBUTING.md
• .github/ISSUE_TEMPLATE.md
• .github/PULL_REQUEST_TEMPLATE.md
• .gitignore
• CODE_OF_CONDUCT.md
• LICENSE
• index.js
• ... probably more :)

These files need to support templates with mixins. For example, I should be able to create a .gitignore.tpl that contains additional ignores I want to add over the default.

In some cases, we may want to support mixins. For example - today we generate a README.md file using repo-tools that uses basic metadata. You can see an example of the data file here:
https://github.com/googleapis/nodejs-spanner/blob/master/.cloud-repo-tools.json

And the README it generates here:
https://github.com/googleapis/nodejs-spanner/blob/master/README.md

It's important that we can define mixins for individual portions in the event that we want to add additional information. For example here's an issue from cloud-debug-nodejs:
GoogleCloudPlatform/nodejs-repo-tools#123

Ideally, we could define:

• A set of common files that get generated for all repos
• A set of language specific files that get generated

The command that re-generates these files should be run as part of autosynth, or some other periodic tool that performs all of the updates via pull requests that occur automagically after PRs to synthtool with template changes are accepted.

when you add the new samples_body partial, ensure that it shows up in the TOC

CC: @JustinBeckwith, @bcoe
Related to:
#246
googleapis/nodejs-translate#268

Synthtool is now creating a synth.metadata file next to the synth.py script. For apiary clients, we reuse the same synth.py for each of the n clients because new clients come in and generated automatically.

We should either skip metadata for these clients, or be able to provide a path to the metadata file.

This usage pattern for fixups:

s.replace(
    '**/*.py',
    <source string>,
    <target string>,
)

blows up for unrelated files (e.g., under .nox/) which are not writeable (e.g., symlinks into non-local Python installs).

We could:

• Harden replace so that it skips over files which aren't writeable.
• Allow the script to pass in a specific root to be used for expanding the globs.
• Or maybe document the syntax for globs such that root isn't needed?

currently synthtool have envvars that can be used to do advanced things. One such feature is export SYNTHTOOL_GOOGLEAPIS=~/.cache/synthtool/googleapis-private which supports using a local git checkout. There are others such as SYNTHTOOL_ARTMAN_VERSION or AUTOSYNTH_USE_SSH.

Improving the documentation for these could assist users.

Encountered this issue generating the nodejs-talent library today (caught indirectly by Node.js's docs-test).

In the latest upstream commit to googleapis (googleapis/googleapis@5fe8a4f), ResumeService was removed entirely from the proto definitions. I know at least in Node.js and Python, the files associated with ResumeService still lingers in the repo.

I had to remove the gapic and protos directories and re-run synthtool locally to me to see that the files are being deleted.

This should not be a breaking behaviour, because the removed service client is removed from the exports in {version}/index.js, so users would not be able to require the client anyways, unless as @busunkim96 points out if they do weird, unsupported things like requiring the client using the direct file system path.

We're currently adding a bunch of files in addition to those generated by synthtool, so a rm -rf * && synthtool is out of the question.

Proposals:

1. Record files generated in synthtool.metadata and if a new generation doesn't yield files previously generated, then remove them.

2. Specifically target generated gapic directories, knowing that we do not add additional files to them and generate into the directory from a clean state.

See: googleapis/google-cloud-python#7565

The bug was introduced in PR #206, in _generic_library:

    def _generic_library(self, directory: str, **kwargs) -> Path:
        self._load_generic_metadata(kwargs["metadata"])0

That keyword is only present (added explicitly) in node_library. The fix is to use kwargs.get("metadata", some_appropriate_default).

README.md still shows internal sso:// link, let's change that to this repo.

see:

• Node Templates: https://github.com/googleapis/synthtool/tree/master/synthtool/gcp/templates/node_library
• Py Library Method to Implement: https://github.com/googleapis/synthtool/blob/master/synthtool/gcp/common.py#L28
• Example node synth using templates: https://github.com/googleapis/nodejs-spanner/blob/master/synth.py#L28
• Example node synth using templates (with replacement vars): https://github.com/googleapis/nodejs-resource/blob/3c753c280a0f3601c60e38ad30f0f4efed78e363/synth.py#L8