litex-hub / conda-build-prepare Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 1.0 56 KB

License: Apache License 2.0

Python 99.26% Batchfile 0.33% Shell 0.41%

conda-build-prepare's People

Contributors

Stargazers

Watchers

Forkers

proppy

conda-build-prepare's Issues

subpackages should be supported

It seems that the current implementation strips the outputs section when rendering meta.yaml thus preventing the tool to "prepare" recipe with subpackages.

Example:

# Rendered by conda-build-prepare
# Original meta.yaml can be found at the end of this file

package:
  name: open_pdks.sky130a
  version: 1.0.329_0_gf245a7a
source:
- git_rev: master
  git_url: /tmp/conda-gf180/git-repos/open_pdks
  patches:
  - disable-link-targets.patch
  - use-sources-from-conda.patch
- folder: skywater-pdk
  url: https://github.com/google/skywater-pdk/archive/refs/heads/main.zip
- folder: skywater-pdk/libraries/sky130_fd_sc_hd/latest
  url: https://github.com/google/skywater-pdk-libs-sky130_fd_sc_hd/archive/refs/heads/branch-0.0.2.zip
- folder: skywater-pdk/libraries/sky130_fd_sc_hvl/latest
  url: https://github.com/google/skywater-pdk-libs-sky130_fd_sc_hvl/archive/refs/heads/branch-0.0.3.zip
- folder: skywater-pdk/libraries/sky130_fd_io/latest
  url: https://github.com/google/skywater-pdk-libs-sky130_fd_io/archive/refs/heads/branch-0.2.1.zip
- folder: skywater-pdk/libraries/sky130_fd_pr/latest
  url: https://github.com/google/skywater-pdk-libs-sky130_fd_pr/archive/refs/heads/branch-0.20.1.zip
- sha256: 1d7f3a284a49d350ddbabde0e7d0c5ffa34a144aaf1bcb5b9f2c87673ff0c76e
  url: https://files.pythonhosted.org/packages/07/1f/3d9ae865addc9ef6cb7b102d7d93e227c46b6e5e94db345cae2a30944efa/dataclasses_json-0.5.6-py3-none-any.whl
- folder: gf180mcu-pdk/gf180mcu_fd_pr
  git_depth: '1'
  git_rev: main
  git_url: /tmp/conda-gf180/git-repos/globalfoundries-pdk-libs-gf180mcu_fd_pr
- folder: gf180mcu-pdk/gf180mcu_fd_io
  git_depth: '1'
  git_rev: main
  git_url: /tmp/conda-gf180/git-repos/globalfoundries-pdk-libs-gf180mcu_fd_io
- folder: gf180mcu-pdk/gf180mcu_fd_sc_mcu7t5v0
  git_depth: '1'
  git_rev: main
  git_url: /tmp/conda-gf180/git-repos/globalfoundries-pdk-libs-gf180mcu_fd_sc_mcu7t5v0
- folder: gf180mcu-pdk/gf180mcu_fd_sc_mcu9t5v0
  git_depth: '1'
  git_rev: main
  git_url: /tmp/conda-gf180/git-repos/globalfoundries-pdk-libs-gf180mcu_fd_sc_mcu9t5v0
- folder: gf180mcu-pdk/gf180mcu_fd_ip_sram
  git_depth: '1'
  git_rev: main
  git_url: /tmp/conda-gf180/git-repos/globalfoundries-pdk-ip-gf180mcu_fd_ip_sram
build:
  noarch: false
  noarch_python: false
  number: '20220908014309'
  script: build_sky130.sh
  script_env: []
  string: '20220908_014309'
requirements:
  build:
  - autoconf
  - automake
  - make
  host:
  - python 3.7.*
  - pip
  - typing_inspect
  - marshmallow
  - marshmallow-enum
  - magic
  run_constrained:
  - magic
test:
  commands:
  - test -d $PREFIX/share/pdk/sky130A/libs.ref/sky130_fd_io
  - test -d $PREFIX/share/pdk/sky130A/libs.ref/sky130_fd_pr
  - test -d $PREFIX/share/pdk/sky130A/libs.ref/sky130_fd_sc_hd
  - test -d $PREFIX/share/pdk/sky130A/libs.ref/sky130_fd_sc_hvl
about:
  home: http://opencircuitdesign.com/open_pdks/
  license: Apache-2.0
  summary: PDK installer for open-source EDA tools and toolchains. Distributed with
    a setup for the Google/SkyWater 130nm process.
extra:
  build_type: local
  final: true
  parent_recipe:
    name: open_pdks
    path: /tmp/conda-gf180/recipe
    version: 1.0.329_0_gf245a7a
  recipe-maintainers:
  - proppy
  recipe_source:
    branch: gf180
    commit: 004d21b47706e4d198fef4d977bf4d54cc5966c0
    date: '20220908_090957'
    describe: v0.0-1329-g004d21b
    repo: https://github.com/hdl/conda-eda

# Original meta.yaml:
#
# # Use `conda-build-prepare` before building for a better version string.
# {% set version = '%s_%04i_%s'| format(GIT_DESCRIBE_TAG | default('0.X'), GIT_DESCRIBE_NUMBER | default(0) | int, GIT_DESCRIBE_HASH | default('gUNKNOWN')) %}
# 
# package:
#   name: open_pdks
#   version: {{ version }}
# 
# source:
#   - git_url: https://github.com/RTimothyEdwards/open_pdks.git
#     git_rev: master
#     patches:
#       - disable-link-targets.patch
#       - use-sources-from-conda.patch
#   - url: https://github.com/google/skywater-pdk/archive/refs/heads/main.zip
#     folder: skywater-pdk
#   - url: https://github.com/google/skywater-pdk-libs-sky130_fd_sc_hd/archive/refs/heads/branch-0.0.2.zip
#     folder: skywater-pdk/libraries/sky130_fd_sc_hd/latest
#   - url: https://github.com/google/skywater-pdk-libs-sky130_fd_sc_hvl/archive/refs/heads/branch-0.0.3.zip
#     folder: skywater-pdk/libraries/sky130_fd_sc_hvl/latest
#   - url: https://github.com/google/skywater-pdk-libs-sky130_fd_io/archive/refs/heads/branch-0.2.1.zip
#     folder: skywater-pdk/libraries/sky130_fd_io/latest
#   - url: https://github.com/google/skywater-pdk-libs-sky130_fd_pr/archive/refs/heads/branch-0.20.1.zip
#     folder: skywater-pdk/libraries/sky130_fd_pr/latest
#   - url: https://files.pythonhosted.org/packages/07/1f/3d9ae865addc9ef6cb7b102d7d93e227c46b6e5e94db345cae2a30944efa/dataclasses_json-0.5.6-py3-none-any.whl
#     sha256: 1d7f3a284a49d350ddbabde0e7d0c5ffa34a144aaf1bcb5b9f2c87673ff0c76e
#   - git_url: https://github.com/google/globalfoundries-pdk-libs-gf180mcu_fd_pr.git
#     git_rev: main
#     folder: gf180mcu-pdk/gf180mcu_fd_pr
#     git_depth: 1
#   - git_url: https://github.com/google/globalfoundries-pdk-libs-gf180mcu_fd_io.git
#     git_rev: main
#     git_depth: 1
#     folder: gf180mcu-pdk/gf180mcu_fd_io
#   - git_url: https://github.com/google/globalfoundries-pdk-libs-gf180mcu_fd_sc_mcu7t5v0.git
#     git_rev: main
#     git_depth: 1
#     folder: gf180mcu-pdk/gf180mcu_fd_sc_mcu7t5v0
#   - git_url: https://github.com/google/globalfoundries-pdk-libs-gf180mcu_fd_sc_mcu9t5v0.git
#     git_rev: main
#     git_depth: 1
#     folder: gf180mcu-pdk/gf180mcu_fd_sc_mcu9t5v0
#   - git_url: https://github.com/google/globalfoundries-pdk-ip-gf180mcu_fd_ip_sram.git
#     git_rev: main
#     folder: gf180mcu-pdk/gf180mcu_fd_ip_sram
#     git_depth: 1
# 
# build:
#   noarch: generic
#   # number: 202202031935
#   number: {{ environ.get('DATE_NUM') }}
#   # string: 20220203_1935
#   string: {{ environ.get('DATE_STR') }}
#   script_env:
#     - CI
# 
# outputs:
#   - name: open_pdks.sky130a
#     requirements:
#       build:
#         - autoconf
#         - automake
#         - make
#       host:
#         - python
#         - pip
#         - typing_inspect
#         - marshmallow
#         - marshmallow-enum
#         - magic
#       run_constrained:
#         - {{ pin_compatible('magic', min_pin='x.x.x', max_pin='x.x.x') }}
#     script: build_sky130.sh
#     files:
#       - $PREFIX/share/pdk/sky130A
#     test:
#       commands:
#         - test -d $PREFIX/share/pdk/sky130A/libs.ref/sky130_fd_sc_hd
#         - test -d $PREFIX/share/pdk/sky130A/libs.ref/sky130_fd_sc_hvl
#         - test -d $PREFIX/share/pdk/sky130A/libs.ref/sky130_fd_io
#         - test -d $PREFIX/share/pdk/sky130A/libs.ref/sky130_fd_pr
#   - name: open_pdks.gf180mcuc
#     requirements:
#       build:
#         - autoconf
#         - automake
#         - make
#       host:
#         - python
#         - pip
#         - magic
#       run_constrained:
#         - {{ pin_compatible('magic', min_pin='x.x.x', max_pin='x.x.x') }}
#     script: build_gf180.sh
#     files:
#       - $PREFIX/share/pdk/gf180mcu
#     test:
#       commands:
#         - test -d $PREFIX/share/pdk/gf180mcuC/libs.ref/gf180mcu_fd_io
#         - test -d $PREFIX/share/pdk/gf180mcuC/libs.ref/gf180mcu_fd_sc_mcu7t5v0
#         - test -d $PREFIX/share/pdk/gf180mcuC/libs.ref/gf180mcu_fd_sc_mcu9t5v0
#         - test -d $PREFIX/share/pdk/gf180mcuC/libs.ref/gf180mcu_fd_ip_sram
#         # from gf180mcu_fd_pr
#         - test -d $PREFIX/share/pdk/gf180mcuC/libs.tech/ngspice
#         - test -d $PREFIX/share/pdk/gf180mcuC/libs.tech/xyce
#         - test -d $PREFIX/share/pdk/gf180mcuC/libs.tech/klayout
#         - test -d $PREFIX/share/pdk/gf180mcuC/libs.tech/xschem
# 
# about:
#   home: http://opencircuitdesign.com/open_pdks/
#   license: Apache-2.0
#   summary: 'PDK installer for open-source EDA tools and toolchains. Distributed with a setup for the Google/SkyWater 130nm process.'
# 
# extra:
#   recipe-maintainers:
#     - proppy

For some reason the name of the package also get "promoted" to the one of the first of the outputs.

`GIT_DESCRIBE_TAG` on macos

I'm unclear if this is a macos issue or a problem with conda-build-prepare, but I'm trying to build some EDA files using the recommended method and it's failing on GIT_DESCRIBE_TAG.

As an example, the yosys meta.yaml contains a line that looks like:

{% set version = '%s_%04i_%s'|format(GIT_DESCRIBE_TAG|replace('yosys-', '') or '0.X', GIT_DESCRIBE_NUMBER|int, GIT_DESCRIBE_HASH or 'gUNKNOWN') %}

If I export GIT_DESCRIBE_TAG to the output of $(git describe --tag), this does not work:

Cloning git sources...

Cloning into '/Users/sean/Code/EDA/conda-eda/workdir/git-repos/yosys'...
remote: Enumerating objects: 71310, done.
remote: Counting objects: 100% (939/939), done.
remote: Compressing objects: 100% (454/454), done.
remote: Total 71310 (delta 612), reused 748 (delta 485), pack-reused 70371
Receiving objects: 100% (71310/71310), 23.37 MiB | 13.75 MiB/s, done.
Resolving deltas: 100% (52431/52431), done.
Already on 'master'
Modifying git tags to set proper package version...

Rewriting tags in "/Users/sean/Code/EDA/conda-eda/workdir/git-repos/yosys"...

Successfully tagged '7e588664e7efa36ff473f0497feacaad57f5e90c' object as '0.26'
Rendering package metadata, please wait...

/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/conda_package_streaming/package_streaming.py:19: UserWarning: zstandard could not be imported. Running without .conda support.
  warnings.warn("zstandard could not be imported. Running without .conda support.")
/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/conda_package_handling/api.py:29: UserWarning: Install zstandard Python bindings for .conda support
  _warnings.warn("Install zstandard Python bindings for .conda support")
Error: Failed to render jinja template in /Users/sean/Code/EDA/conda-eda/workdir/recipe/meta.yaml:
'GIT_DESCRIBE_TAG' is undefined

ERROR conda.cli.main_run:execute(47): `conda run conda render -f /Users/sean/Code/EDA/conda-eda/workdir/recipe/rendered_metadata.yaml /Users/sean/Code/EDA/conda-eda/workdir/recipe` failed. (See above for error)
!! =======================================================
!! ERROR: conda-build-prepare subprocess failed!
!!
!! Command called: conda run -p /Users/sean/Code/EDA/conda-eda/workdir/conda-env conda render -f /Users/sean/Code/EDA/conda-eda/workdir/recipe/rendered_metadata.yaml /Users/sean/Code/EDA/conda-eda/workdir/recipe
!!
!! STDERR not captured; look above for errors too! STDOUT:
!!
Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was:
Command '['/Users/sean/Code/EDA/conda-eda/workdir/conda-env/bin/patch', '--no-backup-if-mismatch', '--batch', '-Np1', '-i', '/var/folders/w2/hfp7dn0d30590f7npsqlj1hh0000gn/T/tmpby1c76n5/makefile-conda-config.patch.native', '--binary', '--dry-run']' returned non-zero exit status 1.
!!
!! =======================================================

Error during package metadata rendering!
/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/conda_package_streaming/package_streaming.py:19: UserWarning: zstandard could not be imported. Running without .conda support.
  warnings.warn("zstandard could not be imported. Running without .conda support.")
/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/conda_package_handling/api.py:29: UserWarning: Install zstandard Python bindings for .conda support
  _warnings.warn("Install zstandard Python bindings for .conda support")

Traceback (most recent call last):
  File "/tmp/really-long-path/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/tmp/really-long-path/conda/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/really-long-path/conda/lib/python3.10/site-packages/conda_build_prepare/__main__.py", line 54, in <module>
    prepare_recipe(recipe_dir, git_dir, env_dir)
  File "/tmp/really-long-path/conda/lib/python3.10/site-packages/conda_build_prepare/conda_cmds.py", line 375, in prepare_recipe
    meta = render_metadata(package_dir, env_dir)
  File "/tmp/really-long-path/conda/lib/python3.10/site-packages/conda_build_prepare/conda_cmds.py", line 59, in render_metadata
    _call_conda_cmd_in_env(f"conda render -f {rendered_path} {package_dir}", env_dir)
  File "/tmp/really-long-path/conda/lib/python3.10/site-packages/conda_build_prepare/conda_cmds.py", line 177, in _call_conda_cmd_in_env
    return _check_output(f'conda run -p {env_path} {cmd_string}')
  File "/tmp/really-long-path/conda/lib/python3.10/site-packages/conda_build_prepare/conda_cmds.py", line 157, in _check_output
    return subprocess.check_output(command.split(), encoding='utf-8',
  File "/tmp/really-long-path/conda/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/tmp/really-long-path/conda/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['conda', 'run', '-p', '/Users/sean/Code/EDA/conda-eda/workdir/conda-env', 'conda', 'render', '-f', '/Users/sean/Code/EDA/conda-eda/workdir/recipe/rendered_metadata.yaml', '/Users/sean/Code/EDA/conda-eda/workdir/recipe']' returned non-zero exit status 1.

sean in 🌐 Garlemald in conda-eda on  master [!?] took 33s
❯

However, if I set it to the following, I can get it to build:

{% set version = '0.X_0000_gUNKNOWN' %}

Note that I'm following the instructions at: https://github.com/hdl/conda-eda/blob/master/docs/DEVELOPMENT.md#preparing-the-recipe-with-conda-build-prepare

Rework 'check_output' to deal with large amount of potential output.

The current check_output wrapper (shown below) will die in horrible ways if the amount of output gets really large;

conda-build-prepare/conda_build_prepare/conda_cmds.py

Lines 153 to 171 in 87015e0

    
           # A wrapper for subprocess.check_output printing stdout in case of process 
        
           # error. By default it isn't printed and can make debugging much harder. 
        
           def _check_output(command, **kwargs): 
        
               try: 
        
                   return subprocess.check_output(command.split(), encoding='utf-8', 
        
                           **kwargs).strip() 
        
               except subprocess.CalledProcessError as e: 
        
                   print('!! =======================================================') 
        
                   print('!! ERROR: conda-build-prepare subprocess failed!') 
        
                   print('!!') 
        
                   print('!! Command called: ' + command) 
        
                   print('!!') 
        
                   print('!! STDERR not captured; look above for errors too! STDOUT:') 
        
                   print('!!') 
        
                   print(e.output.strip()) 
        
                   print('!!') 
        
                   print('!! =======================================================') 
        
                   print() 
        
                   raise

What needs to be done instead;

Create a temporary file for stdout.
Create a temporary file for stdin.
Use subprocess.Popen(stdout=<file>, stderr=<file>). You will probably need close_fds=True too.
Close the temporary files in the parent process (make sure they are not deleted from the file system however).
Use subprocess.wait(timeout=XXXX) in a while loop. Catch the timeout exception and print some type of "waiting for XYZ command" type information. If you want to be super fancy you could even open the temporary file and print the last couple full lines found in the file. Something like;
```
f = open(fname)
f.seek(0, io.SEEK_END)
filesize = f.tell()
f.seek(1000, io.SEEK_END) # seek to 1000 characters before the end
data = f.read(min(1000, filesize))
lines = data.splitlines()[:-1] # Last line might not be complete
```
If the return code is non-zero, print the command failed, dump the contents of the stdout and stderr files to the output.
finally: remove the temporary files.

You will probably need to use tempfile.NamedTemporaryFile for doing this. Do not used the SpooledTemporaryFile no matter how tempting it sounds!

litex-hub / conda-build-prepare Goto Github PK

conda-build-prepare's People

Contributors

Stargazers

Watchers

Forkers

conda-build-prepare's Issues

subpackages should be supported

`GIT_DESCRIBE_TAG` on macos

Rework 'check_output' to deal with large amount of potential output.

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	# A wrapper for subprocess.check_output printing stdout in case of process
	# error. By default it isn't printed and can make debugging much harder.
	def _check_output(command, **kwargs):
	try:
	return subprocess.check_output(command.split(), encoding='utf-8',
	**kwargs).strip()
	except subprocess.CalledProcessError as e:
	print('!! =======================================================')
	print('!! ERROR: conda-build-prepare subprocess failed!')
	print('!!')
	print('!! Command called: ' + command)
	print('!!')
	print('!! STDERR not captured; look above for errors too! STDOUT:')
	print('!!')
	print(e.output.strip())
	print('!!')
	print('!! =======================================================')
	print()
	raise