fireant-bot / fireant Goto Github PK

The Nutch development workflow demands that we create a JIRA issue before submitting a pull request.
The Python jira package provides a simple mechanism for this.

Dockerhub account name is fireantci, github account name is fireant-ci, and github organization hosting the repo is fireant-bot. There should be more consistency in naming here.

Building the Dockerfile on the Apache server now causes the following error:

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 openjdk-11-jdk : Depends: openjdk-11-jre (= 11.0.9.1+1-1~deb10u2) but 11.0.11+9-1~deb10u1 is to be installed
                  Depends: openjdk-11-jdk-headless (= 11.0.9.1+1-1~deb10u2) but 11.0.11+9-1~deb10u1 is to be installed
�[91mE: Unable to correct problems, you have held broken packages.
�[0mThe command '/bin/sh -c apt-get update  && apt-get -y --no-install-recommends install ant="1.10.5-2" openjdk-11-jdk="11.0.9.1+1-1~deb10u2" git="1:2.20.1-2+deb10u3"  && apt-get clean  && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

The Dockerfile code has not been edited since 13 days ago, Jenkins pipeline runs started to fail 9 days ago. Cannot recreate this issue on local Jenkins pipeline.

Another warning that showed up that was not present during the last successful run, might not be related:

WARNING: You are using pip version 21.0.1; however, version 21.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

Run "docker run ... " command in Jenkins. Include environment variable in a way that does not expose the private information.

We need to merge the (very updated) docker branch into the main branch and start working directly off main for consistency and maintainability.

Setup.py should be created and modeled after: https://github.com/capstone-coal/pycoal/blob/master/setup.py

I'm currently trying to work out documentation for portions of Fireant with readthedocs.io and sphinx. More specifically, I aim to document what Fireant does and how it works, as well as instructions on how to properly install / run it. If there's a need or want for more information (Jenkins / Docker / Unit Test documentation) as well, I can look into that. I'll most likely need to connect with all team members to get a better understanding on how I should document different aspects.

The only documentation I currently posses is the Updating Dependencies Tutorial which I based off of Nutch Confluence pages. It's currently formatted as a google doc in our team drive. Since Fireant is an automated service, I'm not sure whether this documentation is necessary. Let me know if I should still make a PR for this / update the documentation with readthedocs

Description

Avoid installing additional packages that you did not explicitly want. Problematic code: dockerfile FROM busybox RUN apt-get install -y python=2.7 Correct code: dockerfile FROM busybox RUN apt-get install -y --no-install-recommends python=2.7

Occurrences

There is 1 occurrence of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/fireant-bot/fireant/issue/DOK-DL3015/occurrences/

Deepsource allows us to automate code reviews with static analysis. I want someone to take the lead on activating Deepsource so we can Identy and fix bug risks, anti-patterns, performance issues, and security flaws on every commit and pull request.

While trying to move to a more secure xml library, some code/library changes make xml comments disappear. We need unit tests to ensure they are preserved.

See apache/nutch#612 and Nutch fireant build #33:

Should update okhttp from version 4.3.1 to version 3.12.13

Should recommend 4.9.1 instead (cf. https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp).
A downgrade by a major version is not a good idea and will likely fail due to API incompatibilities.

Edit the settings of this github repository to set up a webhook that will notify Jenkins whenever a commit is pushed to this repository. Also create an explanation of how to configure the jenkins instance to work with the webhook.

Add or explain how to add a username/password credential for dockerhub on a jenkins instance so that the jenkins pipeline will execute successfully.

Currently, if a dependency that needs to be updated exists twice (or more times) in the same file, two (or more) separate pull requests are created. The desired behavior is to delete the duplicate dependencies and submit a single pull request for the dependency upgrade, noting in the pull request that this deletion was made.

Description

Opening a file using with statement is preferred as function open implements the context manager protocol that releases the resource when it is outside of the with block. Not doing so requires you to manually release the resource.

Occurrences

There are 2 occurrences of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/fireant-bot/fireant/issue/PTC-W0010/occurrences/

Fix incorrect syntax in the docker run command that caused the pipeline to fail.

Add an explanation of the Jenkins pipeline to the Github wiki so others have a better idea of how it works and what it does.

There is a lot of excess whitespace in functions in python code and other files. Remove the whitespace.

Create the directory structure needed for the package to work and for maintainability/consistency purposes. Model after: https://github.com/capstone-coal/pycoal

Description

Cleaning up the apt cache and removing /var/lib/apt/lists helps keep the image size down. Since the RUN statement starts with apt-get update, the package cache will always be refreshed prior to apt-get install. You can read more about it here. Note: Clean up must be performed in the same RUN …

Occurrences

There is 1 occurrence of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/fireant-bot/fireant/issue/DOK-DL3009/occurrences/

@leetheod @rgw17 remember when we manually upgraded the dependencies on the Nutch side???
We had to follow the howto_upgrade_tika.txt for example.
Can you guys have a look at what it would take to update main.py to run the necessary logic to do that?
This requires running ant on the plugin directory and then some unix tools i.e. sed and ls.

All unit tests should be properly documented similar to this: https://github.com/capstone-coal/pycoal/blob/master/pycoal/conversion.py

Currently the pull request format is
fireant upgrade dependency phantomjsdriver in src/plugin/lib-htmlunit/ivy.xml to 1.4.4
I would like to see what the old version was i.e.
fireant upgrade dependency phantomjsdriver in src/plugin/lib-htmlunit/ivy.xml from X.X.X --> 1.4.4
Looks like this can be done in config.py#L30

requires.io ensures that all dependencies we use stay secure and up-to-date. Please activate it for Fireant.

All functions should be properly documented similar to this: https://github.com/capstone-coal/pycoal/blob/master/pycoal/conversion.py

Messages are non-descriptive and only execute on a successful build. Add more description and a failed message aswell.

All functions should be properly documented similar to this: https://github.com/capstone-coal/pycoal/blob/master/pycoal/conversion.py

Find way to keep github and requires.io credentials safe when publishing the package to PyPi and implement it.

main.py should do the following for pull requests:
Sync/update fireant-ci/nutch fork to latest
Create branch/code changes in the fireant-ci/nutch fork
Create the pull request in apache/nutch

Merge xml-unit-tests branch into main for consistency and maintainability of branch names/issues/pull requests. Branch name format should be ISSUE-# where # is the issue number.

Docker branch has to be up-to-date first.

Fireant logo and documentation should be added to the dockerhub fireant account

Description

Version pinning forces the build to retrieve a particular version regardless of what’s in the cache. This technique can also reduce failures due to unanticipated changes in required packages. You can read more about it here Problematic code: dockerfile FROM busybox RUN apt-get install python Correct code: ```dockerfile FROM …

Occurrences

There is 1 occurrence of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/fireant-bot/fireant/issue/DOK-DL3008/occurrences/

Jenkins should set certain environment variables: GITHUB_USERNAME, GITHUB_PASSWORD, GITHUB_EMAIL, and REQUIRES_IO_TOKEN.

Code should also be changed to reflect pull these values from environment variables instead of hard-coding them into the application.

Add rule to deepsource to ignore BAN-B314 so that we are able to keep comments in the xml files. Using the "safe" defusedxml package, we do not retain comments. Also edit lines to make fireant correct again.

Only the main ivy.xml is being checked by the code due to a bug introduced in a previous commit -- ISSUE-9

Description

subprocess.run uses a default of check=False, which means that a nonzero exit code will be ignored by default, instead of raising an exception. You can ignore this issue if this behaviour is intended. <!--more--> ## Examples: ### Bad practice: ```python # Nonzero exit code will be ignored here subprocess.run(['notify-send', '-u', …

Occurrences

There are 5 occurrences of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/fireant-bot/fireant/issue/PYL-W1510/occurrences/

Add Slack and email notifications to the Jenkins pipeline covering failures.