The rock-createiso from rocknsm

UEFI installs fail

During the installer setup, anaconda fails to write the bootloader config. Going to the terminal on tmux window 5 (program-log), I see this.

Seemingly the fedora version of something got installed and is looking for a directory that doesn't exist. Manually changing that command to use centos allows for the install to complete as expected.

Unexplained Exception - jinja2

Once I made the ISO, which appeared to work fine, I got the following once I tried using the deploy_script.sh.

I haven't done any troubleshooting yet, dropping here and I'll work on this.

ERROR! Unexpected Exception, this is probably a bug: jinja2
the full traceback was:

Traceback (most recent call last):
  File "/bin/ansible-playbook", line 85, in <module>
    mycli = getattr(__import__("ansible.cli.%s" % sub, fromlist=[myclass]), myclass)
  File "/usr/lib/python2.7/site-packages/ansible/cli/__init__.py", line 38, in <module>
    from ansible.inventory.manager import InventoryManager
  File "/usr/lib/python2.7/site-packages/ansible/inventory/manager.py", line 29, in <module>
    from ansible.inventory.data import InventoryData
  File "/usr/lib/python2.7/site-packages/ansible/inventory/data.py", line 29, in <module>
    from ansible.plugins.cache import FactCache
  File "/usr/lib/python2.7/site-packages/ansible/plugins/cache/__init__.py", line 30, in <module>
    from ansible.plugins.loader import cache_loader
  File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 22, in <module>
    from ansible.parsing.plugin_docs import read_docstring
  File "/usr/lib/python2.7/site-packages/ansible/parsing/plugin_docs.py", line 12, in <module>
    from ansible.parsing.yaml.loader import AnsibleLoader
  File "/usr/lib/python2.7/site-packages/ansible/parsing/yaml/loader.py", line 30, in <module>
    from ansible.parsing.yaml.constructor import AnsibleConstructor
  File "/usr/lib/python2.7/site-packages/ansible/parsing/yaml/constructor.py", line 29, in <module>
    from ansible.parsing.vault import VaultLib
  File "/usr/lib/python2.7/site-packages/ansible/parsing/vault/__init__.py", line 45, in <module>
    from cryptography.hazmat.backends import default_backend
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 7, in <module>
    import pkg_resources
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3007, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 728, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
    raise DistributionNotFound(req)
DistributionNotFound: jinja2

RPM's that can't be download and are required for the ISO to function offline

rock-createiso/ansible/roles/common/defaults/main.yml

Line 28 in f1f6bec

download_packages:

The following packages need to be assessed. They either have dependency issues or are not able to be download currently.

rock
rock-release
pam_pkcs11
policycoreutils-python
zeek
zeek-aux
zeek-plugin-kafka
zeek-plugin-af_packet
zeek-plugin-gquic
zeek-plugin-communityid
GeoIP
GeoIP-update
stenographer
docket
zookeeper
kafka
python2-xkcdpass
python2-pytest
python36-PyYAML

Rsync overrides time on public mirror

rock-createiso/concourse/upload.sh

Lines 20 to 21 in 12d9f88

	# Sync to public mirror
	ssh -i /root/.ssh/id_ed25519 -o StrictHostKeyChecking=no [email protected] 'rsync -rlvP -e "ssh -i ~/.ssh/mirror_sync" /var/www/mirror/ [email protected]:/var/www/mirror/ --delete'

Needs to be addressed eventually. Not a critical error but this could potentially cause issues with the hourly cron job to clean up old testing ISO's.

Add nano to base install.

For users that don't want to be trapped in vim for eternity, it might be helpful to add nano or something similar to the base install.

Ansible doesn't run on second play

When running the master-iso.sh when it gets to the ansible portion it will fail if you run it twice. This is only an issue when you sign packages and they either partially download or all download and get signed. Once these conditions are met and you want to run the script again for any reason it will crash out in download packages task.

rock-createiso/ansible/roles/common/tasks/main.yml

Lines 11 to 18 in 9144b22

    
           - name: download packages 
        
             repobuilder: 
        
               packages: "{{ download_packages }}" 
        
               groups: "{{ package_groups }}" 
        
               config: "/tmp/rock-offline.conf" 
        
               download_path: "{{ rock_cache_dir }}/Packages" 
        
               exclude: "i686" 
        
               tempcache: true

The out put from ansible is below.

TASK [common : download packages] **********************************************************************************************************************************************************************************************************************************************
task path: /home/admin/rock-createiso/ansible/roles/common/tasks/main.yml:11
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216 `" && echo ansible-tmp-1554392843.21-207182491353216="` echo /root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216 `" ) && sleep 0'
Using module file /home/admin/rock-createiso/ansible/roles/common/library/repobuilder.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-36132nxq0YE/tmpNU27M_ TO /root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/ /root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2 /root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_repobuilder_payload_idX4fO/__main__.py", line 417, in <module>
  File "/tmp/ansible_repobuilder_payload_idX4fO/__main__.py", line 405, in main
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1075, in getPackage
    **kwargs
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1042, in _getFile
    raise e
yum.Errors.NoMoreMirrorsRepoError: failure: Packages/GConf2-3.2.6-8.el7.x86_64.rpm from base: [Errno 256] No more mirrors to try.
http://mirror.centos.org/centos/7/os/x86_64/Packages/GConf2-3.2.6-8.el7.x86_64.rpm: [Errno 14] HTTP Error 416 - Requested Range Not Satisfiable

fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1554392843.21-207182491353216/AnsiballZ_repobuilder.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/tmp/ansible_repobuilder_payload_idX4fO/__main__.py\", line 417, in <module>\n  File \"/tmp/ansible_repobuilder_payload_idX4fO/__main__.py\", line 405, in main\n  File \"/usr/lib/python2.7/site-packages/yum/yumRepo.py\", line 1075, in getPackage\n    **kwargs\n  File \"/usr/lib/python2.7/site-packages/yum/yumRepo.py\", line 1042, in _getFile\n    raise e\nyum.Errors.NoMoreMirrorsRepoError: failure: Packages/GConf2-3.2.6-8.el7.x86_64.rpm from base: [Errno 256] No more mirrors to try.\nhttp://mirror.centos.org/centos/7/os/x86_64/Packages/GConf2-3.2.6-8.el7.x86_64.rpm: [Errno 14] HTTP Error 416 - Requested Range Not Satisfiable\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}
	to retry, use: --limit @/home/admin/rock-createiso/ansible/offline-snapshot.retry

PLAY RECAP *********************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1

It is possible to work around this locally by simply deleting the packages previously signed and downloaded. This works but is not desired as it requires redownloading all the things..

Steps to reproduce.

Run the master-iso script with a gpg key

./master-iso.sh -s ${INPUT_ISO} -o output.iso -g '${KEY_NAME}' -p '${KEY_PASSWORD}' -i ${KEY_PATH}

Repeat step 1 after completion

Missing RPM's for Rocky Linux

rock-createiso/bootstrap.sh

Line 21 in f1f6bec

    
           yum -y install python-pip python-jinja2 python-simplejson genisoimage pykickstart createrepo rsync isomd5sum syslinux pigz mock fuseiso libguestfs-tools-c initial-setup-gui firstboot tree grub2-efi-modules ansible rpm-sign

fuseiso and firstboot are not available for Rocky Linux in default repo's or Epel.

Default filesystem is ext4 for some reason

This is likely related to #33. The default filesystem on Fedora is ext4, while on CentOS it is XFS. We need to get back to the XFS filesystem, which allows us to use quotas for things like pcap retention.

py command not present on Rocky Linux

The following fails on Rocky because py is not a recognized command.

rock-createiso/master-iso.sh

Line 243 in f1f6bec

    
           py 'jinja2.Template(open("templates/isolinux.cfg.j2").read()).render(json.loads(sys.stdin.read()))' | \

rock-createiso/master-iso.sh

Line 247 in f1f6bec

    
           py 'jinja2.Template(open("templates/grub.cfg.j2").read()).render(json.loads(sys.stdin.read()))' | \

rock-createiso/master-iso.sh

Line 251 in f1f6bec

    
           py 'jinja2.Template(open("templates/os-release.j2").read()).render(json.loads(sys.stdin.read()))' | \

rock-createiso/master-iso.sh

Line 255 in f1f6bec

    
           py 'jinja2.Template(open("templates/buildstamp.j2").read()).render(json.loads(sys.stdin.read()))' | \

I tried locally updating the command to python3 but the script errors out with a file not found.

Nightly job needs error checking

The nightly ISO creation will be marked as succeeded if the upload script completes. However, if the upload itself fails the return code is lost, but Concourse believes it succeeded since the script proceeds.

Need to probably retry a number of times, and ultimately fail with a non-zero status so that Concourse can accurately track

UEFI fails to boot correctly on older hardware

Older UEFI systems do not load the RockNSM Grub boot splash and it tries to load the installer for "CentOS 7"

Need to migrate build system to non-root required

The best lead here for downloading all dependent packages is Mock. The problem is that Mock by default excludes system-installed packages, which are required for an install ISO.

Emerging threats should match pinned version

Need to update this to match pinned version of Suricata

rock-createiso/ansible/roles/common/tasks/main.yml

Line 53 in 44f2b18

url: https://rules.emergingthreats.net/open/suricata-4.0/emerging.rules.tar.gz

master-iso.sh creates new repodata after existing data was already signed

The offline-snapshot.sh script creates signed repodata, but in the master-iso.sh script we create new metadata, thus invalidating the signed data.

Packages installed to `/srv/rocknsm/Packages` aren't signed correctly

When running deploy_rock.sh, packages fail to install because the GPG signing key is incorrect for the key associated on disk for the rocknsm-local repo.

Need to check that we're signing ALL rpm's once they are copied offline.

Rock doesn't generate defaults during kickstart

We moved the script, but didn't update the kickstart

Update ISO branding with new RockNSM logos.

Need to merge deployer/bootstrap into Rock ISOs

We are preparing to support containers so we need to add support for that in the ISO.

Remove countdown timer on ISO boot

Under very few circumstances do you want a booted ISO, left unattended, to waylay your system.

I recommend removing the "choice countdown" and requiring a positive acknowledgement of which install option to choose.

Needs Python 3 update

rock-createiso/ansible/roles/common/library/repobuilder.py

Line 108 in f1f6bec

from urlparse import urljoin

Along with many other lines are not python3 compliant and requires updates to work on Rocky Linux.

Will also likely require an update to import dnf instead of yum

	- name: download packages
	repobuilder:
	packages: "{{ download_packages }}"
	groups: "{{ package_groups }}"
	config: "/tmp/rock-offline.conf"
	download_path: "{{ rock_cache_dir }}/Packages"
	exclude: "i686"
	tempcache: true

rocknsm / rock-createiso Goto Github PK

rock-createiso's People

Contributors

Stargazers

Watchers

Forkers

rock-createiso's Issues

Recommend Projects

Recommend Topics

Recommend Org