A framework to generate UI-based Splunk Add-ons.
Home Page: https://splunk.github.io/addonfactory-ucc-generator/
License: Apache License 2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
If you are trying to run ucc-gen in the add-on and metadata folder already exists there, ucc-gen will fail.
Traceback (most recent call last):
File "<path>/.venv/bin/ucc-gen", line 8, in <module>
sys.exit(main())
File "<path>/.venv/lib/python3.9/site-packages/splunk_add_on_ucc_framework/__init__.py", line 884, in main
_generate(args.source, args.config, args.ta_version)
File "<path>/.venv/lib/python3.9/site-packages/splunk_add_on_ucc_framework/__init__.py", line 814, in _generate
os.makedirs(os.path.join(outputdir, ta_name, "metadata"))
File "/usr/local/Cellar/[email protected]/3.9.8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 225, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '<path>/output/<addon_name>/metadata'
When defining field validators in globalConfig.json, we should be able to be more specific about numbers that are acceptable.
That are integers in most cases, but JavaScript number (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number) accepts floats as well.
Proposed options would be to reflect explicitly the type in validator type - eg. "number.integer" or implicitly in range values - eg. "range": [0, 1.0] if float number is expected; "range": [0, 1] would mean that only value 0 or 1 is expected.
If globalConfig.json has an alert which has a field like
{
"type": "text",
"label": "Endpoint",
"field": "endpoint",
"help":"REST endpoint to create incident. Format: /api/<API namespace>/<API ID>/<Relative path>.",
"required": false
}
ucc-gen fails to create a html for this alert to use in Splunk.
The exact error is:
...
File "/Users/arys/src/addonfactory-repository-template/splunk-add-on-for-servicenow/.venv/lib/python3.7/site-packages/defusedxml/lxml.py", line 143, in fromstring
rootelement = _etree.fromstring(text, parser, base_url=base_url)
File "src/lxml/etree.pyx", line 3237, in lxml.etree.fromstring
File "src/lxml/parser.pxi", line 1896, in lxml.etree._parseMemoryDocument
File "src/lxml/parser.pxi", line 1777, in lxml.etree._parseDoc
File "src/lxml/parser.pxi", line 1082, in lxml.etree._BaseParser._parseUnicodeDoc
File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError
File "<string>", line 31
lxml.etree.XMLSyntaxError: Specification mandates value for attribute namespace, line 31, column 70
cookiecutter library was used to develop ucc-gen init feature quite fast, but it can be replaced by jinja2 library and we will reduce the number of dependencies we have in the project.
Publish docs that we have in docs folder in splunk.github.io. Example: https://splunk.github.io/splunk-connect-for-syslog/main/
The app built with an Add-on builder has an issue/warning on the App-inspect for the jQuery version compatibility issue. (I'm posting here because I know UCC is what powers the Add-on builder.)
Reference - https://dev.splunk.com/enterprise/docs/developapps/visualizedata/updatejquery/
The issue is in common.js. I tried using the latest Add-on builder and has still the same warnings.
I think the issue could be specific to the Add-on builder and UCC has already resolved this issue. Let me know if this is the case. I can try using the latest version of UCC and see the common.js.
Problem
At the moment to run addonfactory-ucc-generator from a python file we need to create a subprocess
subprocess.run(["ucc-gen", "arguments go here"], env=os.environ)
Which makes it harder to debug and catch errors.
Possible solution
What about creating another function called main_with_args which we will be able to normally use in python code, e.g.
from splunk_add_on_ucc_framework import main_with_args
main_with_args(source="..", ....)
On Saturday 15th of July 2023 was released pip version 23.2 on pypi and it became the default.
When running ucc-gen, the new release of pip is automatically used and ucc-gen systematically fails to generate any package with:
Building wheels for collected packages: splunk-sdk
Building wheel for splunk-sdk (setup.py) ... done
Created wheel for splunk-sdk: filename=splunk_sdk-1.7.3-py3-none-any.whl size=125267 sha256=1968a464bc5315ec9a66e5a2aa84305d81207325a6e47c9f981623d23044098c
Stored in directory: /home/gmarchand/.cache/pip/wheels/85/7d/26/d4c4d6e96b70a9809ac667d226b9d2fe4cce761a9c435e03e3
Successfully built splunk-sdk
ERROR: Error while checking for conflicts. Please file an issue on pip's issue tracker: https://github.com/pypa/pip/issues/new
Traceback (most recent call last):
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 575, in _determine_conflicts
return check_install_conflicts(to_install)
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/operations/check.py", line 108, in check_install_conflicts
would_be_installed = _simulate_installation_of(to_install, package_set)
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/operations/check.py", line 131, in _simulate_installation_of
dist = abstract_dist.get_metadata_distribution()
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/distributions/wheel.py", line 23, in get_metadata_distribution
assert self.req.local_file_path, "Set as part of preparation during download"
AssertionError: Set as part of preparation during download
Installing collected packages: PySocks, charset-normalizer, idna, urllib3, certifi, requests, defusedxml, sortedcontainers, splunk-sdk, solnlib, splunktalib, splunktaucclib, pip
ERROR: Exception:
Traceback (most recent call last):
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
return func(self, options, args)
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 452, in run
installed = install_given_reqs(
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 72, in install_given_reqs
requirement.install(
File "/home/gmarchand/.local/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 805, in install
assert self.local_file_path
AssertionError
I have tried many various things, all unsuccessful for now, to prevent this pip version from being used by ucc-gen.
The same bevhiour can be observed on Mac & Linux, clearing pip cache, uninstall or other trick does help as ucc-gen systematically tries to acquire the latest version available on-line.
Super annoying and blocking issue, is there any way to force ucc-gen behaviour regarding the pip version it uses?(requirements.txt seems to be ignored in that case?)
splunk-add-on-ucc-framework==5.28.2
Tested on different Mac & Linux OS, Python 3.10.6
Warning: Each child in a list should have a unique "key" prop.
Check the render method of BaseFormView. See https://reactjs.org/link/warning-keys for more information.
in Fragment (created by BaseFormView)
in BaseFormView (created by EntityPage)
...
It is reproducible in browser runtime and tests, for example: link
Currently, dependencies are installed during ucc-gen for the platform the script is executed. It reads the lib/requirements.txt folder and performs installation via pip.
The issue is, when running ucc-gen on a Mac and testing in a Linux-based Docker container, the dependencies will not work if there are native dependencies. Also, as a TA author, I want to provide compatibility across platforms.
My wish would be to have means to get platform-specific dependencies installed via UCC.
First question - is Windows supported for this toolset?
When running ucc-gen on a Windows system, I received the following output.
(venv) D:\user\project>ucc-gen
2022-07-21 20:29:03,559 [UCC] INFO: ucc-gen version 5.12.0 is used
2022-07-21 20:29:03,710 [UCC] INFO: Cleaning out directoryD:\user\project\output
2022-07-21 20:29:03,718 [UCC] INFO: Cleaned out directory D:\user\project\output
2022-07-21 20:29:03,735 [UCC] INFO: Config is valid
2022-07-21 20:29:03,741 [UCC] INFO: Addon Version : 0.0.0R7b419b5
2022-07-21 20:29:03,742 [UCC] INFO: Package ID is project
2022-07-21 20:29:03,742 [UCC] INFO: Copy UCC template directory
2022-07-21 20:29:03,762 [UCC] INFO: Copy globalConfig to output
2022-07-21 20:29:03,762 [UCC] INFO: Install add-on requirements into D:\user\project\output\project\lib from package
2022-07-21 20:29:03,762 [UCC] INFO: Checking for requirements in package
2022-07-21 20:29:03,762 [UCC] INFO: Uses common requirements
'python3' is not recognized as an internal or external command,
operable program or batch file.
'python3' is not recognized as an internal or external command,
operable program or batch file.
2022-07-21 20:29:03,780 [UCC] INFO: Replace tokens in views
2022-07-21 20:29:03,805 [UCC] INFO: Copy package directory
(venv) D:\user\project>
Since I'm running this in a virtual environment, I checked the venv\Scripts directory. Sure enough, Windows doesn't include a python3 binary (only python.exe). I'm wondering if the ucc-gen binary requires a python3 binary? If so, does anyone know how to add a python3 binary or the equivalent of a symlink to python.exe in Windows? If not, can some conditional checks be added in the next version?
The version detail is not getting updated in the init.py file
If running on GitHub Actions, and after installing ucc-gen files have wrong permissions. It affects only UI-based add-ons.
Steps to reproduce:
.github/workflows foldermain.yaml file in itname: build
on:
push:
pull_request:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/[email protected]
- uses: actions/[email protected]
with:
python-version: 3.7
- run: pip install splunk-add-on-ucc-framework
- run: |
ls -al /opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/splunk_add_on_ucc_framework/package/appserver/static/js/build
-rw-rw-rw-+ permissionsBecause ucc-gen copies all those files, they appear in the output/<your-addon> with the incorrect permissions.
It affects Splunk Appinspect run and it reports that:
A posix world-writable file was found. File: default/data/ui/views/configuration.xml
A posix world-writable file was found. File: appserver/templates/base.html
A posix world-writable file was found. File: appserver/static/alerticon.png
A posix world-writable file was found. File: appserver/static/js/build/4.js
A posix world-writable file was found. File: appserver/static/js/build/entry_page.licenses.txt
A posix world-writable file was found. File: appserver/static/js/build/0.js
A posix world-writable file was found. File: appserver/static/js/build/1.licenses.txt
A posix world-writable file was found. File: appserver/static/js/build/5.js
A posix world-writable file was found. File: appserver/static/js/build/3.licenses.txt
A posix world-writable file was found. File: appserver/static/js/build/3.js
A posix world-writable file was found. File: appserver/static/js/build/entry_page.js
A posix world-writable file was found. File: appserver/static/js/build/1.js
A posix world-writable file was found. File: appserver/static/js/build/0.licenses.txt
When adding OAuth functionality via. globalConfig, the "placeholder" property of basic/oauth fields is now allowed.
Currently the documentation states:
Fields allowed in basic and oauth fields as of now:
oauth_field: This should be kept as it is and without any change.
label: This can be changed if the user wants to change the label of the field in UI.
field: For now this user must keep it as it is for mandatory fields as mentioned above.
help : This can be changed if user wants to change the help text displayed below field.
encrypted : This should be true if user wants that particular field encrypted else no need to have this parameter.placeholder: The placeholder for the field.
required: To specify whether the field is required or not. The default value is true.
However, when building an app with the ucc-gen build --ta-version=1.0.0 command, I am met with the following error message:
~/appname/ $ ucc-gen build --ta-version=1.0.0
INFO: ucc-gen version 5.28.4 is used
INFO: Python binary name to use: python3
INFO: Output folder is /Users/username/appname/output
INFO: Add-on will be built with version '1.0.0'
INFO: Cleaned out directory /Users/username/appname/output
INFO: Using globalConfig file located @ /Users/username/appname/globalConfig.json
ERROR: globalConfig file is not valid. Error: Additional properties are not allowed ('placeholder' was unexpected)
5.28.4
Python==3.11.2, OS=Mac OS Ventura
Do we need to swap try and for?
It does not work if you have lib/requirements.txt with the following content:
setuptools
pip
wheel
In the resulting package we will still have pkg_resources/, _distutils_hack/ folders and distutils-precedence.pth file.
Originally posted by @arys-splunk in #224 (comment)
UCC uses Splunk UI to render the UI part of the add-on. Splunk UI decided to deprecate placeholder prop for some of the components. In UCC we are doing the same. It is recommended to use help instead.
placeholder is still usable.globalConfig file, but do not render it in the UI.globalConfig file and users are going to see the error during the schema validation.When creating an Add-on with ucc-gen, a default.meta should be automatically created for the Add-on component to be made available out of its own app name space.
The purpose of an Add-on (except the pure data collection) to provide system wide available items such as alert actions or even index and search time parsing, these should by default be shared at the system level with the automatic generation of a similar content:
metadata/default.meta
# Application-level permissions
[]
owner = admin
access = read : [ * ], write : [ admin ]
export = system
If some of the Add-ons objects need to remain at the app level, this is the responsability from the dev / Splunk admins to perform the relevant configuration, but from the ucc-gen point of view we need to share all objects.
According to the schema here value can be number/string/boolean.
When running ucc-gen command, it throws a validation error that AttributeError: 'int' object has no attribute 'lower' for this particular part of the configuration.
{
"name": "advanced_settings",
"entity": [
{
"type": "singleSelect",
"label": "Some label",
"options": {
"disableSearch": true,
"autoCompleteFields": [
{
"value": 1,
"label": "1"
},
{
"value": 2,
"label": "2"
},
{
"value": 4,
"label": "4"
}
]
},
"defaultValue": 4,
"field": "some_field"
}
],
"title": "Advanced Settings"
}
5.27.1
macOS
When creating and Add-on with ucc-gen, we should generate automatically the relevant configuration for the Add-on to work properly in Search Head Cluster and synchronise the configuration file amongs the members.
This is achieved via the generation of a server.conf:
# server.conf
[shclustering]
conf_replication_include.<ta_name>_settings = true
conf_replication_include.<ta_name>_account = true
oauth if in use would added too.
This request depends on solnlib enhancement request
Idea is to have log prefix set automatically to TA name (globalConfig meta.name ; lowercased if needed) or hardcoded and to be able to get it for usecases like troubleshooting/monitoring
May be wrong and fail in some circumstances. For some reason, this did not fail during testing but has been observed to fail if pip 20.0.2 installed.
This probably should be
pip_update_command = f"{installer} -m pip install --upgrade pip=={pip_version}"
notice there was an extra pip in the cmdline which may have caused it to fail
latest
No response
Is there a trigger or parameter that will direct ucc-gen to use the input_with_helper template in the script stubs that are created?
Similar to #331.
The goal of this enhancement is to give people an option to define a customisation for the UCC-generated REST handlers, it can be useful in case of additional validation of some fields before saving them.
There is already a solution where you need to "overwrite" the REST handler created by ucc-gen and put your own validation there, documented here: https://splunk.github.io/addonfactory-ucc-generator/custom_rest_handler/.
Be able to run ucc-gen --version and get the version of ucc-gen installed.
Version should be taken from
main branchUseful links:
Hi,
From the UI point of view, it would be a great addition to be able change the UI based theme via a configuration parameter in globalConfig.json.
Currently, ucc-gen will use an hard encoded light mode, if the Add-on provides frontend components such as other dashboards and if these are using dark theme, then from a conception point of view the configuration UI looks a defect in the app.
And well, Dark theme is espicially fancy in React!
I am having issues setting files to be excluded using .uccignore
At
should the call to remove_listed_files not come after the copying of the files?Hi,
A recently introduced change mades the UCC gen way too much, and incorrectly, restrictive regarding the Addon naming convention.
Example of failure:
Some functions will not be available outside of a splunk hosted process
2021-04-24 14:54:24,899 [UCC] INFO: Cleaning out directory /Users/guilhem/DocsOffsync/git/splunk/addonfactory-ucc-generator/TA-jira-service-desk-simple-addon/data/output
2021-04-24 14:54:24,899 [UCC] INFO: Cleaned out directory /Users/guilhem/DocsOffsync/git/splunk/addonfactory-ucc-generator/TA-jira-service-desk-simple-addon/data/output
2021-04-24 14:54:24,920 [UCC] ERROR: Config is not valid. Error: 'TA-jira-service-desk-simple-addon' does not match '^\\w+$'
Failed validating 'pattern' in schema['properties']['meta']['properties']['restRoot']:
{'pattern': '^\\w+$', 'type': 'string'}
On instance['meta']['restRoot']:
'TA-jira-service-desk-simple-addon'
This change was I believe introduced in the following commit:
The regular expression should not forbid the usage of hypens, while this is not good practice anymore in Appinspect, this is only a warning not a failure, and the UCC gen must not decide on its own to prohibit its usage.
So I have my_custom_modular_input.py, my custom code for a modular input, which implements valiate_input and stream_events
ucc-gen creates my_custom.py, a rest handler skeleton which has stream_events, valiate_input, all the arguments defined in globalConfig.json, and the filename is already linked to the modular input definition.
Since I already have my_custom_modular_input.py, it would be great if ucc-gen could automatically import this modular when writing my_custom.py and use it in the stream_events and validate_input method calls.
I'm currently using some regex and file manipulation with additional_packaging.py which works fine but feels brittle if the format or output of ucc-gen would change.
I feel this would be a very useful feature for folks working with modular input scripts, but if I've missed a better workflow please let me know. Thanks for considering.
After using ucc-gen in a clean environment and the output directory is created, subsequent ucc-gen invocation fails.
2021-11-09 21:10:14,084 [UCC] INFO: Cleaning out directory /home/ubuntu/splunk-conf21/output
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/ucc-gen", line 8, in <module>
sys.exit(main())
File "/home/ubuntu/.local/lib/python3.8/site-packages/splunk_add_on_ucc_framework/__init__.py", line 884, in main
_generate(args.source, args.config, args.ta_version)
File "/home/ubuntu/.local/lib/python3.8/site-packages/splunk_add_on_ucc_framework/__init__.py", line 711, in _generate
clean_before_build(outputdir)
File "/home/ubuntu/.local/lib/python3.8/site-packages/splunk_add_on_ucc_framework/__init__.py", line 123, in clean_before_build
os.makedirs(os.path.join(outputdir))
File "/usr/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/ubuntu/splunk-conf21/output'
The output structure following the ucc-gen invocation is
ubuntu@ip-172-31-40-160:~/splunk-conf21/output/TA-conf21$ ls -Fal
total 16
drwxrwxr-x 4 ubuntu ubuntu 4096 Nov 9 22:37 ./
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 9 21:10 ../
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 9 22:37 bin/
drwxrwxr-x 7 ubuntu ubuntu 4096 Nov 9 22:37 lib/
Removing the output directory manually prior to ucc-gen results in proper output creation.
Is there a way for UCC to build inputs.conf.spec with python.version already set? Right now I am using regex via additional_packaging to modify the output but I was wondering if there was a cleaner solution I might have missed.
It gets flagged as a failure by appinspect otherwise.
$ splunk-appinspect inspect ${APP_DIR_NAME}${APP_VERSION}.spl --output-file ${APP_DIR_NAME}${APP_VERSION}_CORE_APPINSPECT.xml --data-format junitxml --mode precert --max-messages=all --excluded-tags manual
Check that all the modular inputs defined in inputs.conf.spec are
explicitly set the python.version to python3.
FAILURE: Modular input "soc_bulletin" is defined in
README/inputs.conf.spec, python.version should be explicitly set to
python3 under each stanza. File: README/inputs.conf.spec Line
Number: 26
When building an app using ucc-gen, I get the following AppInspect errors. Most of the *nix write permissions are referring to file generated by ucc-gen, as I suppose the hidden python files (which I could not locate).
Run splunk-appinspect inspect my-app.tgz --output-file appinspect.json
Enable Python analyzer.
Validating: my_app Version: 0.0.1
..........................................................................F...................................................................E...................................................................
.............................................................................
...
A default value of 25 for max-messages will be used.
Python file standards
Check that there are no hidden python files included in the app.
ERROR: 'charmap' codec can't decode byte 0x9d in position
3214: character maps to <undefined>
Source code and binaries standards
Check that no files have *nix write permissions for all users (xx2,
xx6, xx7). Splunk recommends 644 for all app files outside of the bin/
directory, 644 for scripts within the bin/ directory that are invoked using
an interpreter (e.g. python my_script.py or sh my_script.sh), and 755 for
scripts within the bin/ directory that are invoked directly (e.g.
./my_script.sh or ./my_script). Since appinspect 1.6.1, check that no files
have nt write permissions for all users.
FAILURE: A posix world-writable file was found. File:
appserver/templates/base.html
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/1.js
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/5.js
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/entry_page.licenses.txt
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/entry_page.js
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/1.licenses.txt
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/0.licenses.txt
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/3.js
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/3.licenses.txt
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/0.js
FAILURE: A posix world-writable file was found. File:
appserver/static/js/build/4.js
FAILURE: A posix world-writable file was found. File:
default/data/ui/views/inputs.xml
FAILURE: A posix world-writable file was found. File:
default/data/ui/views/configuration.xml
However, when running the same build with splunk-add-on-ucc-framework version 5.5.5, AppInspect succeeds and these failures do not occur. Have you already seen this behaviour? Is this a known issue?
Currently testing migrating to the UCC for alert actions purposes, I am stuck with Python import errors even with the default testing Addon:
04-08-2021 09:14:42.370 INFO sendmodalert - Invoking modular alert action=test_alert for search="" sid="1617873282.12" in app="Splunk_TA_UCCExample" owner="admin" type="inline"
04-08-2021 09:14:42.413 INFO PreviewExecutor - Preview Enforcing initialization done
04-08-2021 09:14:42.550 ERROR sendmodalert - action=test_alert STDERR - Traceback (most recent call last):
04-08-2021 09:14:42.550 ERROR sendmodalert - action=test_alert STDERR - File "/opt/splunk/etc/apps/Splunk_TA_UCCExample/bin/test_alert.py", line 9, in <module>
04-08-2021 09:14:42.550 ERROR sendmodalert - action=test_alert STDERR - from splunktaucclib.alert_actions_base import ModularAlertBase
04-08-2021 09:14:42.550 ERROR sendmodalert - action=test_alert STDERR - File "/opt/splunk/etc/apps/Splunk_TA_UCCExample/lib/splunktaucclib/alert_actions_base.py", line 5, in <module>
04-08-2021 09:14:42.550 ERROR sendmodalert - action=test_alert STDERR - from logging_helper import get_logger
04-08-2021 09:14:42.550 ERROR sendmodalert - action=test_alert STDERR - ModuleNotFoundError: No module named 'logging_helper'
04-08-2021 09:14:42.571 INFO sendmodalert - action=test_alert - Alert action script completed in duration=179 ms with exit code=1
04-08-2021 09:14:42.571 WARN sendmodalert - action=test_alert - Alert action script returned error code=1
04-08-2021 09:14:42.571 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
Steps to reproduce:
I can dirty fix the import error by editing the file:
Splunk_TA_UCCExample/lib/splunktaucclib/alert_actions_base.py
With:
import sys
sys.path.append('/opt/splunk/etc/apps/Splunk_TA_UCCExample/lib/splunktaucclib')
from logging_helper import get_logger
After running the alert, I see more errors which are not related to Python import:
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - Traceback (most recent call last):
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - File "/opt/splunk/etc/apps/Splunk_TA_UCCExample/bin/test_alert.py", line 44, in <module>
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - exitcode = AlertActionWorkertest_alert("Splunk_TA_UCCExample", "test_alert").run(sys.argv)
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - File "/opt/splunk/etc/apps/Splunk_TA_UCCExample/lib/splunktaucclib/alert_actions_base.py", line 219, in run
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - self.prepare_meta_for_cam()
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - File "/opt/splunk/etc/apps/Splunk_TA_UCCExample/lib/splunktaucclib/alert_actions_base.py", line 205, in prepare_meta_for_cam
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - for num, result in enumerate(csv.DictReader(rf)):
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - File "/opt/splunk/lib/python3.7/csv.py", line 111, in __next__
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - self.fieldnames
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - File "/opt/splunk/lib/python3.7/csv.py", line 98, in fieldnames
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - self._fieldnames = next(self.reader)
04-08-2021 09:21:42.556 ERROR sendmodalert - action=test_alert STDERR - _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
04-08-2021 09:21:42.582 INFO sendmodalert - action=test_alert - Alert action script completed in duration=397 ms with exit code=1
04-08-2021 09:21:42.583 WARN sendmodalert - action=test_alert - Alert action script returned error code=1
04-08-2021 09:21:42.583 ERROR sendmodalert - Error in 'sendalert' command: Alert script returned error code 1.
Thanks
It is hard for the beginners to start using UCC framework by the lack of the documentation and examples. That is what I will be working on for some time in the future.
But for now, we can also introduce a ucc-gen init command that creates a very minimal version of the add-on that can be loaded to Splunk and should be working without any additional modifications.
This command should accept the following arguments: TBD
Commands to execute:
python3 -m venv .venv
source .venv/bin/activate
pip install splunk-add-on-ucc-framework
pip install splunk-packaging-toolkit
ucc-gen init <params to be defined>
ucc-gen --ta-version=0.0.1
slim package output/<addon_name>
The additional property becomes problematic when python client is created from openapi.json
It forces to use the property that is not valid when creating input
Recreation scenario:
Create some directory and cd to it
mkdir out and copy openapi.json to it
Run:
docker run --rm -v ${PWD}:/local swaggerapi/swagger-codegen-cli-v3 generate -i /local/openapi.json -l python -o /local/out/python
python subdirectory appears in the out dir - that’s our client lib; open it and run
python setup.py install --user
create test.py script as below:
(example for Jira Cloud TA; based on README from the client code)
from __future__ import print_function
import time
import swagger_client
from swagger_client.rest import ApiException
from pprint import pprint
# Configure HTTP basic authorization: BasicAuth
configuration = swagger_client.Configuration()
configuration.host = configuration.host.replace('{domain}','localhost')
configuration.host = configuration.host.replace('{port}','8089')
configuration.verify_ssl = False
configuration.username = 'admin'
configuration.password = # your password here
# create an instance of the API class
api_instance = swagger_client.DefaultApi(swagger_client.ApiClient(configuration))
output_mode = 'json' # str | The name of the item to operate on (optional)
try:
api_response = api_instance.splunk_ta_jira_cloud_settings_logging_post(loglevel='DEBUG', output_mode=output_mode)
pprint(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->splunk_ta_jira_cloud_settings_logging_post: %s\n" % e)
input = {}
input['domain'] = {}
input['domain']['name'] = # your domain
input['token'] = {}
input['token']['name'] = 'tkn01'
input['token']['domain'] = input['domain']['name']
input['token']['username'] = # your username
input['token']['value'] = # your token value
input['input'] = {}
input['input']['name'] = 'in02'
input['input']['token'] = input['token']['name']
input['input']['from'] = '2022-01-01T00:00:00'
input['input']['interval'] = '60'
input['input']['index'] = 'default'
input['input']['disabled'] = True
name = input['domain']['name']
try:
api_response = api_instance.splunk_ta_jira_cloud_domain_post(name=name, output_mode=output_mode)
pprint(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->splunk_ta_jira_cloud_domain_post: %s\n" % e)
name = input['token']['name']
domain = input['token']['domain']
username = input['token']['username']
token = input['token']['value']
try:
api_response = api_instance.splunk_ta_jira_cloud_api_token_post(domain=domain, name=name, username=username, token=token, output_mode=output_mode)
pprint(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->splunk_ta_jira_cloud_api_token_name_post: %s\n" % e)
api_token = input['input']['token'] # 'api_token_example' # str | (optional)
name = input['input']['name'] # str | (optional)
_from = input['input']['from'] # str | (optional)
interval = input['input']['interval'] # str | (optional)
index = input['input']['index'] # str | (optional)
# disabled = input['input']['disabled'] # str | (optional)
disabled = None
try:
api_response = api_instance.splunk_ta_jira_cloud_jira_cloud_input_post(api_token=api_token, name=name, _from=_from, interval=interval, index=index, disabled=disabled, output_mode=output_mode)
pprint(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->splunk_ta_jira_cloud_jira_cloud_input_post: %s\n" % e)
Once the test.py script is run, the TA Configuration is created as expected
Input is not created and error appears:
HTTP response body: b'{"messages":[{"type":"ERROR","text":"Unexpected error \"<class 'splunktaucclib.rest_handler.error.RestError'>\" from python handler: \"REST Error [400]: Bad Request -- HTTP 400 Bad Request -- b'{\"messages\":[{\"type\":\"ERROR\",\"text\":\"Argument \\\\\"disabled\\\\\" is not supported by this handler.\"}]}'\". See splunkd.log/python.log for more details."}]}'
No response
No response
If globalConfig.json contains multiple helplink objects with an empty label, UCC breaks, example:
"field": "config1_help_link",
"label": "",
"type": "helpLink",
"options": {
"text": "Add-on configuration documentation",
"link": "https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/configuration.html"
}
},
{
"field": "config2_help_link",
"label": "",
"type": "helpLink",
"options": {
"text": "SSL configuration documentation",
"link": "https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/configuration.html"
}
},
Leads to:
This can be workaround by changing the second label to an empty space:
{
"field": "config2_help_link",
"label": " ",
"type": "helpLink",
"options": {
"text": "SSL configuration documentation",
"link": "https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/configuration.html"
}
},
Then everything works fine.
Thanks!
With current release, mod alert works with Splunk 8.x but fail with Splunk 7.3.x (and probably priors)
If you include a modified version of the bin/_helper.py and I modify the import such that:
import modalert_<my_mod_alert>_helper
instead of:
from ta_myta import modalert_<my_mod_alert>_helper
Likely a Python 2.7.x / 3.x compatibility issue.
It would be helpful to have a list of the JavaScript libraries included by ucc-gen when it is building our apps, maybe by dropping the lock file in /lib?
Current work around:
It would be preferred if the ucc-gen would perform these functions or similar automatically.
Thanks for considering.
On unknown circumstances, ucc-gen is blowing up with the following error:
ValueError: Version '1-post.3+7029d83' does not conform to the Semantic Versioning style
It was reported by @okashaev-splunk for one of the projects.
Once you press enter on some dialog, the dialog is closed with action as on "Cancel" while action like on "Add" would be expected.
Thanks @dfederschmidt for this feedback.
meta.version is being modified after an invocation of ucc-gen, which should not be the case.
If globalConfig.json has an alert configuration, ucc-gen should create alert_actions.con file providing default alert icon. Otherwise, Appinspect CLI complains about missing icon_path.
I used 2 different globalConfig.json files to generate add-on. I've uploaded them here (Github does not allow to upload JSON files, so they are in .txt format).
Steps to reproduce:
ucc-genslim package output/Splunk_TA_mysqlWhen using globalConfig_1 - it gives the first error.
When using globalConfig_2 - it gives the second error.
Expectations:
ucc-gen reports that the config is invalid and does not create a package
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
@babel/preset-env, @splunk/react-ui, @splunk/splunk-utils, @storybook/addon-essentials, @storybook/addon-interactions, @storybook/addon-links, @storybook/blocks, @storybook/react, @storybook/react-webpack5, @testing-library/jest-dom, @types/node, @typescript-eslint/eslint-plugin, @typescript-eslint/parser, actions/cache, css-loader, eslint-plugin-jest, importlib-metadata, jsonschema, mkdocs-material, msw, msw-storybook-addon, node, postcss, pytest-rerunfailures, pytest-splunk-addon, react-router-dom, splunk/appinspect-cli-action, storybook, style-loader).github/actions/cached-ui-deps/action.yml
actions/cache v3.github/workflows/agreements.yaml
splunk/addonfactory-github-workflows v1.github/workflows/build-test-release.yml
actions/checkout v4splunk/addonfactory-test-matrix-action v1.13actions/checkout v4actions/upload-artifact v4actions/checkout v4actions/setup-python v5pre-commit/action v3.0.0actions/checkout v4semgrep/semgrep-action v1actions/checkout v4apache/skywalking-eyes v0.5.0actions/checkout v4actions/setup-python v5actions/download-artifact v4actions/download-artifact v4actions/upload-artifact v4actions/checkout v4actions/setup-python v5actions/checkout v4actions/setup-python v5actions/download-artifact v4actions/checkout v4actions/setup-python v5actions/download-artifact v4actions/upload-artifact v4actions/upload-artifact v4actions/checkout v4actions/setup-python v5actions/download-artifact v4actions/upload-artifact v4dorny/test-reporter v1actions/checkout v4actions/setup-python v5splunk/appinspect-cli-action v2.0actions/checkout v4actions/setup-python v5actions/download-artifact v4actions/download-artifact v4splunk/semantic-release-action v1.3.github/workflows/build-ui.yml
actions/checkout v4actions/setup-node v4actions/upload-artifact v4.github/workflows/docs.yml
actions/checkout v4actions/setup-python v5actions/setup-node v4
ui/package.json
@splunk/react-icons ^4.2.0@splunk/react-page ^6.3.3@splunk/react-toast-notifications ^0.11.3@splunk/react-ui ^4.23.0@splunk/splunk-utils ^2.3.4@splunk/themes ^0.16.4@splunk/ui-utils ^1.6.0axios ^1.6.5immutability-helper ^3.1.1license-webpack-plugin ^4.0.2lodash ^4.17.21prop-types ^15.8.1react ^16.14.0react-dom ^16.14.0react-is ^16.13.1react-router-dom ^6.21.1styled-components ^5.3.11uuid ^9.0.1zod ^3.22.4@babel/core ^7.23.7@babel/eslint-parser ^7.23.3@babel/plugin-transform-runtime ^7.23.7@babel/preset-env ^7.23.7@babel/preset-react ^7.23.3@splunk/babel-preset ^4.0.0@splunk/eslint-config ^4.0.0@splunk/webpack-configs ^7.0.2@storybook/addon-essentials ^7.6.7@storybook/addon-interactions ^7.6.7@storybook/addon-links ^7.6.7@storybook/blocks ^7.6.7@storybook/react ^7.6.7@storybook/react-webpack5 ^7.6.7@storybook/testing-library ^0.2.2@testing-library/jest-dom ^6.1.6@testing-library/react ^12.1.5@testing-library/user-event ^14.5.2@types/jest ^29.5.11@types/js-yaml ^4.0.9@types/node ^20.10.6@types/react ^16.14.51@typescript-eslint/eslint-plugin ^6.17.0@typescript-eslint/parser ^6.17.0babel-eslint ^10.1.0babel-jest ^29.7.0babel-loader ^8.3.0cross-env ^7.0.3css-loader ^6.8.1eslint ^7.32.0eslint-config-airbnb ^18.2.1eslint-config-prettier ^7.2.0eslint-import-resolver-typescript ^3.6.1eslint-plugin-import ^2.29.1eslint-plugin-jest ^27.6.1eslint-plugin-jsx-a11y ^6.8.0eslint-plugin-prettier ^4.2.1eslint-plugin-react ^7.33.2eslint-plugin-react-hooks ^4.6.0eslint-plugin-storybook ^0.6.15fork-ts-checker-webpack-plugin ^9.0.2jest ^29.7.0jest-environment-jsdom ^29.7.0js-yaml ^4.1.0msw ^2.0.11msw-storybook-addon 2.0.0--canary.122.06f0c92.0prettier ^2.8.8querystring-es3 ^0.2.1storybook ^7.6.7style-loader ^3.3.3stylelint ^14.16.1ts-node ^10.9.2typescript ^5.3.3undici ^5.28.2url ^0.11.3webpack ^5.89.0webpack-cli ^5.1.4webpack-merge ^5.10.0node >=20.10.0yarn ^1.22.21@npmcli/git ^2.0.8@types/react ^16.14.35css-what ^5.0.1glob-parent ^5.1.2postcss ^8.4.32semver ^7.5.4strip-ansi ^6.0.1string-width ^4.2.2
pyproject.toml
poetry >=1.0.2
pyproject.toml
python ^3.7jinja2 >=2,<4addonfactory-splunk-conf-parser-lib *dunamai ^1.9.0jsonschema ^4.4.0PyYAML ^6.0openapi3 ^1.7.0defusedxml ^0.7.1requests ^2.31.0urllib3 <2colorama ^0.4.6mkdocs ^1.4.2importlib-metadata *pytest ^7.2.1pytest-splunk-addon ^5.0.0pytest-splunk-addon-ui-smartx 2.5.4pytest-rerunfailures ^11.1.1mkdocs-material ^9.1.3pytest-cov ^4.0.0covdefaults ^2.3.0
Hi!
Currently trying to migrate an existing Add-on to be using the super cool features on the framework, I am encountering a failure when attempting to save an account, with:
Cannot perform action "POST" without a target name to act on.
As bellow:
Testing with the UCC Example TA works perfectly fine so I am certain this is coming from the config in my Addon, thus I have not been able to locate where the issue is.
The code is available here:
https://github.com/guilhemmarchand/ucc-test-TA-jira-service-desk-simple-addon
Dev tools does not offer much help:
I can see the endpoints:
So far I am a little blocked, any idea what's going on?
Thanks in advance for any help!
Here it is a few problems glued together. Logic is like following
init.py
line 626 - we get a version from Git, if the work folder does not have a git initialized we fail
line 635 - we create an argument --ta-version
line 643 - if ta-version argument was passed we completely ignore the version that we got in line 626
Problems:
--ta-version we completely ignore the version that we got earlierOne of the solutions that improves logic, without chaging it
temp_version = "definitely not a version that someone gonna use"
parser.add_argument(
"--ta-version",
type=str,
help="Version of TA, Deafult version is version specified in the package such as app.manifest, app.conf, and globalConfig.json",
default = temp_version
)
args = parser.parse_args()
ta_version = args.ta_version.strip()
if not ta_version or ta_version == temp_version:
version = Version.from_git()
if not version.stage:
stage = 'R'
else:
stage = version.stage[:1]
version_str = version.serialize(metadata=True, style=Style.SemVer)
ta_version = f"{version.base}{stage}{version.commit}"
if not os.path.exists(args.source):
raise NotADirectoryError("{} not Found.".format(os.path.abspath(args.source)))
Make sure button with to indicator is available for TA developers
Swagger (https://swagger.io/) or similar tool would improve TA developer and TA user experience
Appinspect warning with web.conf due to pattern exposed that does not correspond to any match stanza in restmap.conf
An Add-on created with ucc-gen results gets a warning message from Appinsect as follows:
check_web_conf_expose_patterns_have_restmap_matches
web.conf found with a `pattern` exposed that does not correspond to any `match` stanza in restmap.conf. Apps should only expose endpoints that they define. Pattern: `/data/*`. Please remove or edit this stanza: [expose:_splunkd_data]. File: default/web.conf Line Number: 13
The check refers to this section in the generated web.conf:
[expose:_splunkd_data]
pattern = data/*
methods = GET
The restmaps.conf contains (generated by ucc-gen):
[admin:trackme]
match = /
members = trackme_settings
[admin_external:trackme_settings]
handlertype = python
python.version = python3
handlerfile = trackme_rh_settings.py
handleractions = edit, list
handlerpersistentmode = true
It looks like we should have a corresponding stanza in restmaps.conf ?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.