pcdshub / ads-deploy Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 5.0 176 KB

[Conda/Docker] TwinCAT ADS IOC + pytmc + related tools for deploying projects on Windows

License: Other

Dockerfile 0.74% Makefile 0.20% Batchfile 7.22% Python 90.52% Shell 1.33%

ads-ioc docker epics linux python visual-studio

ads-deploy's People

Contributors

Stargazers

Watchers

Forkers

klauer ghalym zllentz tacaswell jvamosi-slac

ads-deploy's Issues

plcprog-console builds may not find database correctly for load_plc_databases.cmd

Current Behavior

The database is built correctly but the additional .cmd file is not.

Traceback (most recent call last):
  File "c:/Repos/ads-ioc/R0.5.0//iocBoot/templates/dbloads.py", line 63, in <module>
    main()
  File "c:/Repos/ads-ioc/R0.5.0//iocBoot/templates/dbloads.py", line 46, in main
    total_records = sum(count_number_of_records(fn) for fn in db_filenames)
  File "c:/Repos/ads-ioc/R0.5.0//iocBoot/templates/dbloads.py", line 46, in <genexpr>
    total_records = sum(count_number_of_records(fn) for fn in db_filenames)
  File "c:/Repos/ads-ioc/R0.5.0//iocBoot/templates/dbloads.py", line 23, in count_number_of_records
    with open(db_filename, 'rt') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/c/Users/klauer/Documents/Repos/lcls-plc-las-bts/iocBoot/ioc-las-bts/.pytmc_build/las_bts.db'
* Installing: las_bts.db

Need to dig into details here.

Can always rebuild on linux, of course, so it's not a show-stopper.

autosave file location should be located in /reg/d/iocData/ioc-plc-lfe-vac/autosave/ info_positions.sv

Starting iocInit
############################################################################
## EPICS R7.0.2.2-2.0.1
## EPICS Base built Jun  4 2019
############################################################################
reboot_restore: entry for file 'info_positions.sav'
reboot_restore: Found filename 'info_positions.sav' in restoreFileList.
*** restoring from '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav' at initHookState 6 (before record/device init) ***
save_restore: Can't open file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav'.
save_restore: Trying backup file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.savB'
save_restore: Can't open file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.savB'.
save_restore: Can't figure out which seq file is most recent,
save_restore: so I'm just going to start with '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav0'.
save_restore: Trying backup file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav0'
save_restore: Can't open file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav0'.
save_restore: Trying backup file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav1'
save_restore: Can't open file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav1'.
save_restore: Trying backup file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav2'
save_restore: Can't open file '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav2'.
save_restore: Can't find a file to restore from...
save_restore: ...last tried '/reg/neh/home/adpai/iocData/ioc-plc-lfe-vac/autosave/info_positions.sav2'. I give up.
save_restore: **********************************

Add helper for installing pre-commit

Goal: single-click install pre-commit hooks for the given twincat (git) repository

https://github.com/pcdshub/pre-commit-hooks/

Why

.tmc diffs and such are vastly improved when using pre-commit hooks
Style check fixing

Use NC defaults in the generated documentation

Can be used in documentation and elsewhere

https://github.com/pcdshub/ads-deploy/blob/master/ads_deploy/default_nc_params.xml

The format is different than the XML/XTI from an axis. It's still parseable by pytmc.parser, but may need some custom stuff written around it.

TODO

Ensure it's in the MANIFEST
Ensure it's in setup.py

Jinja2 version 3 documentation build failure

(ads-deploy) PC98125:docs klauer$ ads-deploy docs ../*.sln
INFO:ads_deploy.docs:Rendering general template: index.rst -> index.rst
INFO:ads_deploy.docs:Rendering per-PLC template: {{tsproj.name}}_{{plc.name}}_summary.rst -> lcls-plc-tmo-optics_tmo_optics_summary.rst
INFO:ads_deploy.docs:Rendering per-PLC template: {{tsproj.name}}_{{plc.name}}_epics.rst -> lcls-plc-tmo-optics_tmo_optics_epics.rst
INFO:ads_deploy.docs:Rendering per-tsproj template: {{tsproj.name}}_nc.rst -> lcls-plc-tmo-optics_nc.rst
INFO:ads_deploy.docs:Rendering per-tsproj template: {{tsproj.name}}_boxes.rst -> lcls-plc-tmo-optics_boxes.rst
INFO:ads_deploy.docs:Rendering per-tsproj template: {{tsproj.name}}_pragmas.rst -> lcls-plc-tmo-optics_pragmas.rst
INFO:ads_deploy.docs:Rendering per-PLC template: {{tsproj.name}}_{{plc.name}}_source.rst -> lcls-plc-tmo-optics_tmo_optics_source.rst
INFO:ads_deploy.docs:Rendering per-tsproj template: {{tsproj.name}}_links.rst -> lcls-plc-tmo-optics_links.rst
INFO:ads_deploy.docs:Done


(ads-deploy) PC98125:docs klauer$ pip install --upgrade jinja2
Collecting jinja2
  Downloading Jinja2-3.0.1-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 7.7 MB/s
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1-cp37-cp37m-macosx_10_9_x86_64.whl (13 kB)
Installing collected packages: MarkupSafe, jinja2
  Attempting uninstall: MarkupSafe
    Found existing installation: MarkupSafe 1.1.1
    Uninstalling MarkupSafe-1.1.1:
      Successfully uninstalled MarkupSafe-1.1.1
  Attempting uninstall: jinja2
    Found existing installation: Jinja2 2.11.2
    Uninstalling Jinja2-2.11.2:
      Successfully uninstalled Jinja2-2.11.2
Successfully installed MarkupSafe-2.0.1 jinja2-3.0.1

(ads-deploy) PC98125:docs klauer$ ads-deploy docs ../*.sln
INFO:ads_deploy.docs:Rendering general template: index.rst -> index.rst
INFO:ads_deploy.docs:Rendering per-PLC template: {{tsproj.name}}_{{plc.name}}_summary.rst -> lcls-plc-tmo-optics_tmo_optics_summary.rst
INFO:ads_deploy.docs:Rendering per-PLC template: {{tsproj.name}}_{{plc.name}}_epics.rst -> lcls-plc-tmo-optics_tmo_optics_epics.rst
Traceback (most recent call last):
  File "/Users/klauer/mc/envs/ads-deploy/bin/ads-deploy", line 11, in <module>
    load_entry_point('ads-deploy', 'console_scripts', 'ads-deploy')()
  File "/Users/klauer/docs/Repos/ads-deploy/ads_deploy/__main__.py", line 99, in main
    func(**kwargs)
  File "/Users/klauer/docs/Repos/ads-deploy/ads_deploy/docs.py", line 404, in main
    write_file(template_path, render_args, 'per-PLC')
  File "/Users/klauer/docs/Repos/ads-deploy/ads_deploy/docs.py", line 386, in write_file
    template = jinja_env.get_template(template_path.name)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/environment.py", line 997, in get_template
    return self._load_template(name, globals)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/environment.py", line 958, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/loaders.py", line 553, in load
    return loader.load(environment, name, globals)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/loaders.py", line 137, in load
    code = environment.compile(source, name, filename)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/environment.py", line 750, in compile
    source = self._generate(source, name, filename, defer_init=defer_init)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/environment.py", line 684, in _generate
    optimized=self.optimized,
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 117, in generate
    generator.visit(node)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/visitor.py", line 40, in visit
    return f(node, *args, **kwargs)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 888, in visit_Template
    self.blockvisit(node.body, frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 449, in blockvisit
    self.visit(node, frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/visitor.py", line 40, in visit
    return f(node, *args, **kwargs)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 1262, in visit_For
    self.blockvisit(node.body, loop_frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 449, in blockvisit
    self.visit(node, frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/visitor.py", line 40, in visit
    return f(node, *args, **kwargs)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 1299, in visit_If
    self.blockvisit(node.body, if_frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 449, in blockvisit
    self.visit(node, frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/visitor.py", line 40, in visit
    return f(node, *args, **kwargs)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 1262, in visit_For
    self.blockvisit(node.body, loop_frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 449, in blockvisit
    self.visit(node, frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/visitor.py", line 40, in visit
    return f(node, *args, **kwargs)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 1582, in visit_AssignBlock
    self.pop_assign_tracking(frame)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/compiler.py", line 813, in pop_assign_tracking
    ref = frame.symbols.ref(name)
  File "/Users/klauer/mc/envs/ads-deploy/lib/python3.7/site-packages/jinja2/idtracking.py", line 81, in ref
    "Tried to resolve a name to a reference that was"
AssertionError: Tried to resolve a name to a reference that was unknown to the frame ('key')

Add a seperate vscode settings file for plcprog

Expected Behavior

The idea for this issue is to create a separate vscode settings file that is tailored to the plcprog-console machine.

Current Behavior

Currently, updating the ads-deploy installation requires each user to add the settings file then manually deselect two docker-dependent items, "ads-deploy: use Docker" and "ads-deploy: use Conda". My concern is that each user has to do this for each update of ads-deploy and missing these steps could break everyone's installation of ads-deploy.

Possible Solution

Add a secondary vscode settings file that does not load the two problematic external tools. As a result users will not need to take the two additional steps to keep from breaking the global installation.

Context

This is a point of concern as it creates many opportunities for failure in a commonly used tool. The likelihood of this issue arising as well of the severity of the issue if it arises both scale with the number of users.

Your Environment

plcprog-console

ADS deploy builds/updates st.cmd file conditionally

The chain window must be closed in order for the st.cmd file to appear/update.

Clean up the tools menu

Explanation

This is the current tools menu:

We only really use:

Lint pragmas
Configure and build
Record debugging

Project summary and qt designer aren't part of the workflow now, and the IOC/screens are not useful on the programming node at all.

I'd say it'd be worthwhile to remove the unused ones to clean things up.
For now, users can remove the ones they don't use on a case-by-case basis. I think I'll do that for myself.

Database file not present for record count while building on Windows

Such that load_plc_databases.cmd doesn't get generated there:

* Found database files to install; creating load_plc_databases.cmd...
Traceback (most recent call last):
  File "c:/Repos/ads-ioc/R0.4.1//iocBoot/templates/dbloads.py", line 63, in <module>
    main()
  File "c:/Repos/ads-ioc/R0.4.1//iocBoot/templates/dbloads.py", line 46, in main
    total_records = sum(count_number_of_records(fn) for fn in db_filenames)
  File "c:/Repos/ads-ioc/R0.4.1//iocBoot/templates/dbloads.py", line 46, in <genexpr>
    total_records = sum(count_number_of_records(fn) for fn in db_filenames)
  File "c:/Repos/ads-ioc/R0.4.1//iocBoot/templates/dbloads.py", line 23, in count_number_of_records
    with open(db_filename, 'rt') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/c/Users/klauer/Documents/linking-types/iocBoot/ioc-Untitled1/.pytmc_build/Untitled1.db'

Remove `typhos` requirement

Current Behavior

typhos is a hard requirement of ads-deploy but is only used in ads-deploy typhos (i.e., to spawn a typhos screen for the entire PLC project)

Why?

This hasn't been used since the project first came about 4 years ago. Just @klauer poking around with testing what we could do (along with the mock caproto IOC that never was really used - a subject for another issue)
It's now a problem for ads-deploy to be installed on TwinCAT BSD (TcBSD) - see here
- This isn't strictly because of typhos, but because of its dependencies (numpy)

Possible Solution

Remove typhos from the requirements
Make it optional
Or, consider removing the functionality entirely

Refactor "ads-deploy docs" around "pytmc template" tool

Expected Behavior

Functionality provided here was moved into pytmc where it rightfully belongs.

See:
https://github.com/slaclab/pytmc/blob/master/pytmc/bin/template.py

Possible Solution

Rely on that code and remove it from ads-deploy.

New Windows installation requires some local fixes to work

Issue

Installing ads-deploy should be as simple as using the provided conda install script
... but at least for Ricardo, it wasn't. It required these 3 fixes:

lxml installation failed; it required a pip uninstall lxml; pip install lxml. This caused pytmc and ads-deploy imports to just bail without any error message. python -vc 'import pytmc' (verbose import) indicated that lxml was at fault. This may have been a one-off, but it's worth noting and remembering for the next install
The ads-ioc version it looks for by default is R0.4.1 which isn't the latest. Set it to master since that'll always exist. When new versions come up, we should git pull in Repos/ads-ioc/master and then clone a new versioned directory
There is a pytmc template bug for Windows that we hotfixed on plcprog-console that never made it upstream. That issue is: pcdshub/pytmc#299 and we should resolve that sooner rather than later

Windows build.sh should find the latest ads-ioc automatically

Currently, build.sh is hard-coded to R0.4.y. Other scripts auto-detect the latest version of ads-ioc, but this one does not.
Patched in-place currently on plcprog-console, but to be fixed at some point

Remove unused/old scripts (display.py)

ads-deploy/tools/python/display.py

Line 1 in e545880

from qtpy.QtWidgets import QApplication

Deploy new version using DUT_MotionStage (IOC + pytmc)

Library version enumeration is insufficient

Several forms found - PlaceholderReference:

    <PlaceholderReference Include="Tc3_Module">
      <DefaultResolution>Tc3_Module, * (Beckhoff Automation GmbH)</DefaultResolution>
      <Namespace>Tc3_Module</Namespace>
    </PlaceholderReference>

    <PlaceholderResolution Include="lcls-twincat-motion">
      <Resolution>lcls-twincat-motion, 1.4.0 (SLAC)</Resolution>
    </PlaceholderResolution>

LibraryReference

    <LibraryReference Include="LCLS General,2.4.2,SLAC">
      <Namespace>LCLS_General</Namespace>
    </LibraryReference>

For some reason, I searched for DefaultResolution and Resolution, which skips the LibraryReference.
This should be fixed up here and in pytmc template which aggregates these libraries for easier html rendering.

Add a `sleep` after running the IOC to ensure stdout/stderr have been flushed

ads-deploy/tools/python/make_scripts.py

Line 31 in e545880

    
                   "make -C ${{ADS_IOC_PATH}}/iocBoot/templates && cd '{ioc_path}' && make && sed -i '/^adsIoc_registerRecord.*$/a adsSetLocalAddress({local_net_id})' st.cmd && ./st.cmd"

When https://github.com/pcdshub/ads-ioc/blob/4548df191547a240ceb11d078761c14136daf8e5/iocBoot/templates/st.cmd.template#L72 (EthercatMCCreateAxis("$(MOTOR_PORT)", "$(AXIS_NO)", "$(AMPLIFIER_FLAGS)", "$(AXISCONFIG)")) failed due to creating an axis number greater than the number of symbols, the IOC segfaulted but docker truncated the IOC shell output - leading to confusion over where it crashed.

fbFormat(sFormat := 'Fast Faults: %d', arg1:=F_ULINT(PMPS_GVL.AccumulatedFF), sOut=>DisplayStats.sLine2, bError => bError, nErrID => nErrID ); 

DisplayStats(eMode:=E_CX2100_DisplayModesWr.eCX2100_WriteLines);


Traceback (most recent call last):
  File "/usr/local/bin/pytmc", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/pytmc/bin/pytmc.py", line 83, in main
    func(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/pytmc/bin/summary.py", line 236, in main
    log_level=log_level, debug=debug
  File "/usr/local/lib/python3.7/site-packages/pytmc/bin/summary.py", line 143, in summary
    if not decl_or_impl.text.strip():
AttributeError: 'NoneType' object has no attribute 'strip'```

Use `pytmc db --dbd adsIoc.dbd` where possible

Such that when using ads-ioc + generating a database file, the generated database is linted against the actual adsIoc database definition.

Better support for variant IOCs?

Expected Behavior

Determine a better way of specifying how to build multiple IOCs from the same PLC project.

Current Behavior

The default ads-deploy iocboot will create ioc-{plcname} and build it.

Currently this requires:

Copy ioc-{plcname} to ioc-{myname}
Adjust ioc-{myname}/Makefile with new macros or PLC/AMS Net ID overrides
Rebuild each directory when the PLC project changes

Changes?

At least:

iocBoot/Makefile probably should be generated to rebuild everything

Maybe:

ads-deploy iocboot --copy original new?
A configuration file specifying the details (👎 )

Remove windows *.cmd scripts

They ended up being proof-of-concept things and not so useful.
Let's remove these:

ads-deploy/ads_deploy/windows/make_scripts.cmd

Line 4 in fcdb56e

@echo Creating windows_run-ioc-in-docker.cmd and windows_run-typhos-gui.cmd...

pcdshub / ads-deploy Goto Github PK

ads-deploy's People

Contributors

Stargazers

Watchers

Forkers

ads-deploy's Issues

Current Behavior

Expected Behavior

Current Behavior

Possible Solution

Context

Your Environment

Explanation

Current Behavior

Possible Solution

Expected Behavior

Possible Solution

Issue

Expected Behavior

Current Behavior

Changes?

Recommend Projects

Recommend Topics

Recommend Org