samsung / universum Goto Github PK

Originally created on Thu, 16 Mar 2017 16:16:15 +0900

1. Describe all configuration properties (keys in dict)
2. Write "Combining configurations" section
3. One more example, including variations
4. Describe configuration file debug ('dump' function, run from console from exact place) and place before section "combining"
5. Explain how to use code style checks in configs.py
6. Place somewhere earlier: section about paths, 'make' commands, getting paths from configuration_support (different from CI system and configs.py debug mode), etc.
7. Place as second section: comand line is not a string, but a list of one-word parameters, examples
8. 'configuration_support.py' documentation inside file itself, add links to the function descriptions in the file from main documentation

Originally created on Tue, 28 Mar 2017 21:36:44 +0900

Report testing coverage of all new code right to the corresponding lines of code

Originally created on Thu, 16 Mar 2017 16:08:15 +0900

Create tests for CI system in isolated environment (docker):
Create docker images for perforce, swarm and target system
Implement tests based on functionality and on regression test cases

Originally created on Tue, 16 May 2017 19:25:04 +0900

Check code with pylint and process results

Originally created on Wed, 22 Feb 2017 20:03:50 +0900

Sometimes, especially for debug reasons, all file removing in finalize() functions should be disabled.

Originally created on Thu, 16 Mar 2017 16:04:09 +0900

Create unit-tests in separate test scripts.
Use whatever unit-testing and mocking framework is appropriate.

Originally created on Thu, 16 Mar 2017 16:00:28 +0900

Currently there is a script created by SQA team. It must be rewritten in python and incorporated into CI system.

Main functionality:

1. The new script should support follow checkers:
   • "Dos end" file format checking
   • Tab checking (exclude for Makefile)
   • Trailing whitespaces
     Code style checker (Uncrustify or something else)
2. Script should have a possibility to check only the new patch changes (shelve) or all project tree
3. All code style reports should be generate in colourful html files
4. Code style (Uncrustify) report should be a compare line-by-line between latest version (left side) in repository and new changes (right side). As a feature: compare only the changed code instead of the entire source file. For the latter the following tool can be used: https://github.com/llvm-mirror/clang/blob/master/tools/clang-format/clang-format-diff.py
5. Code style (Uncrustify) should use the default config files for C/C++ and JAVA languages. Also must be default config file for Linux Kernel and possibility to upload custom config file for each project
6. html reports should be attached as links at the Swarm Review in comments. Vote Up and Down is enabled or disabled by needs

Originally created on Wed, 22 Feb 2017 19:10:23 +0900

TeamCity cannot interpret escape sequences, so they should not be shown there at all.

Originally created on Tue, 21 Feb 2017 20:02:11 +0900

Right now Swarm module gets build result from Output.build_failures, which creates unwanted dependecies.
Build results should be collected and processed in a separate module.

Originally created on Tue, 21 Feb 2017 19:11:49 +0900

CI system needs example project with example 'config.py' script and example usage script for developers to refer to.

Originally created on Mon, 6 Mar 2017 19:18:01 +0900

Some projects need sources from more than one directory on P4. Support of multiple mappings should be added

Originally created on Tue, 21 Feb 2017 18:58:22 +0900

Most exceptions should not cause "Unexpected error" failure message, but require special processing.

Originally created on Tue, 21 Feb 2017 19:09:02 +0900

Although class Main for 'universum.py' functionality is already created, most of code is still not included in it. This should be fixed.

Originally created on Fri, 21 Apr 2017 22:27:43 +0900

Now all project steps are launched in working directory if not stated otherwice.
This directory must be changed to settings.launcher.actual_project_root.

Originally created on Wed, 22 Feb 2017 18:45:39 +0900

The absence of 'configs.py', or any issues on its contents should be noticed in time and reported to user with recommendations on further actions.

Originally created on Tue, 28 Mar 2017 21:34:04 +0900

Add report of Static Analysis results into Code Review system:

1. On each change collect Static Analysis data (issues) from the latest sources without the change
2. Collext list of issues from Static Analsys from the latest sources with the change
3. Calculate the difference
4. Report each new issue right into Code Review system into corresponding line of code

Originally created on Wed, 29 Mar 2017 21:07:49 +0900

Right now, when encountering several configurations with same names while writing logs to files, CriticalException is raised with notifying user that log with this name already exists.
TODO: if log with the same name was created during this CI system run, it should be continued; otherwice, exception should be raised.

Originally created on Mon, 3 Apr 2017 22:36:43 +0900

In case of exceptions all sh stderr is printed in one place, including all previous warnings.
In case of no exceptions sh stderr is not printed at all.

Originally created on Tue, 21 Feb 2017 18:45:17 +0900

Sometimes collected artifacts should be left unpacked for further processing after build ends.
Special option should be added for this purpose (e.g. command line argument)

Originally created on Tue, 21 Feb 2017 19:15:00 +0900

Some settings must be passed to several modules, and it makes sense to implement it in our common argument parser infrastructure

Originally created on Tue, 28 Mar 2017 21:35:44 +0900

Collect cyclomatic complexity before and after the change is applied and report to the Code Review system to the corresponding line of code if there is a significant increase.

Originally created on Tue, 23 May 2017 21:57:38 +0900

Collect and run all docstring tests in CI module.

Originally created on Wed, 29 Mar 2017 22:19:48 +0900

Swarm varibles can be passed and should be checked via argparse module

Originally created on Thu, 25 Jan 2018 12:17:16 +0200

We would like to have the ability to run steps of the same project in parallel on several machines to improve build latency.
Possible choices: docker clusters, jenkins, mesos, other.

Input from customers

1. Launch and execution of the project must not differ from the launch without using distributed build option. This means, logs and output must be almost identical.
2. We should avoid creating new jobs/configurations if possible.
3. This feature must include and absorb the background steps feature. This means all configuration, usage and logs must be identical for both features. Difference between background and distributed should not be defined by different syntax or logic. Rather, it should be some global or per-step config option.
4. Parallelism must not be based on the steps hierarchy.
5. There must be ability to specify what steps can be done sequentially to other steps, and what steps can be done in parallel.
6. Discretionary waiting for steps must be supported.

Proposal for controlling parallelism

1. Introduce thread id to config.
2. Steps with the same id are always sequential
3. Steps with different ids can be launched in parallel
4. Steps without thread id obtain default (e.g. 0) and are executed sequentially
5. There must explicit or implicit splitting (thread creation) and joining (thread waiting) points. Probably, initial splitting can be based on having non-default thread id. For example, all steps without id can work as implicit join.

Ideally, the parallel steps must be parallel in Blue Ocean

Originally created on Thu, 16 Mar 2017 16:12:55 +0900

Create initial release notes in RST format and include them in the documentation.
From now on regenerate documentation and update release notes during release.
Place generated documentation into the 'release' branch.

Originally created on Wed, 29 Mar 2017 23:34:07 +0900

During project building with TC agent
This error occurs:

Error: 'ascii' codec can't encode character u'\u2018' in position 0: ordinal not in range(128)

Build fails on 3d step, so previous steps passed.
BUT when I build on the same server with my account manually , with tse same environment - all ok, successful building.
I have tried to put into config file

#!/usr/bin/env python
coding: UTF-8

but it doesn't seem to help.

Originally created on Fri, 19 May 2017 23:46:20 +0900

CI system should report to Swarm only if there were some build steps ran that need reporting. Reports on unsuccessful unshelving, config parsing, etc. do not involve changes from the review and should not be sent back to Swarm.

Originally created on Thu, 2 Mar 2017 17:04:07 +0900

A mechanism to pass some build-context parameter to configs.py is required.
For example, we can pass some subset of command-line parameters and environment variables.
This functionality can be used for implementing exclusion/inclusion.

Originally created on Wed, 22 Feb 2017 20:13:36 +0900

Instead of using predefined workspaces switch to newly created disposable ones; also, for downloading with p4, switch to 'export' command instead of 'sync' and avoid using workspaces at all.

Originally created on Tue, 28 Feb 2017 18:59:12 +0900

We need to implement support for launching entire build in dry run mode.
Dry run must not perform any real actions, but must print all usual logs and command-lines of any external commands.
This can be controlled by special command-line parameter, for example, --dry-run.

Originally created on Tue, 11 Apr 2017 21:30:39 +0900

The CI system must support setting runtime dependencies for each build step (configuration). It must be supplied in the form of reference to the Docker image in the repository for this configuration.
The CI system must then launch this step in the container created from that image.
Note, the CI script must be launched in its own container anyway. That means that probably the dockerization and current build with dependencies on the build server side cannot be mixed.

Originally created on Wed, 22 Feb 2017 19:53:48 +0900

Add MarkDown-compatible README file to CI documentation.

Originally created on Wed, 22 Feb 2017 19:15:16 +0900

Logs, created by CI systems, are mostly .txt files that do not support any formatting by escape sequences, so these symbols should not be written to log files.

Originally created on Wed, 19 Apr 2017 21:52:53 +0900

When performing a Swarm check, "CL already committed" should exit the script without voting and commenting review.

Originally created on Tue, 21 Feb 2017 23:22:17 +0900

If a source directory is the same as CI working directory, an infinite copying cycle is started.

Originally created on Tue, 28 Feb 2017 19:01:09 +0900

There must be a possibility to launch script with fault injection enabled.
Fault injection must fail some parts of the script by some criteria.
For example, we can specify faults ratio, such as "fail each command out of 5" or something like that. Also, it would be nice if we could fail only some steps or groups of steps.

Originally created on Tue, 7 Mar 2017 23:50:45 +0900

Proposed items:

• Exception handling
  • no crashes when handling p4 issues
  • no crashes when handling os, environment, file system issues
  • no crashes when handling connectivity issues
  • no crashes when processing configs.py
• Setting working directory
  Mapping with multiple lines for the same depot path
• Handling of parameters for build_teamcity.py and p4-poll.py (e.g. P4CLIENT is not needed for p4-poll.py)

Originally created on Thu, 16 Mar 2017 16:10:05 +0900

Create needed configuration on TeamCity to include all testing types

Originally created on Mon, 10 Apr 2017 20:39:49 +0900

The following bugs are observed:

1. The p4 where must be supplied full project path with ellipsis (...). Otherwise p4 where fails with "files not in client view" message.

   1. Steps to reproduce:
      create mapping with some project, e.g.//depot/path/... and pass the same path as P4_PATH
   2. Expected result:
      Project is processed
   3. Actual result:
      CI system fails on 'p4 where' with "files not in client view" error
   4. How to fix:
      do not remove ellipsis

2. Only last entry from the client view is used for getting local path, which is incorrect

   1. Steps to reproduce:
      create mapping with folder and subfolder, for example //depot/path/... and //depot/path/some/... and pass //depot/path/... as P4_PATH

   2. Expected result:
      Project is processed

   3. Actual result:
      The CI system only uses local path for //depot/path/some/... and ignores the real path //depot/path/...

   4. How to fix:

      • compare depot path to be identical to P4_PATH
      • take the 'unmap' dictionary key into account - ignore records with this key (the list of dictionaries is returned by p4.run_where)

3. Remapped locations must produce exception

Originally created on Thu, 23 Feb 2017 00:07:23 +0900

Log files are created in the current directory using relative file path, regardless of working dir option.
Also REPOSITORY_DIFFERENCE.txt is created in the current directory.

Originally created on Tue, 21 Feb 2017 18:55:18 +0900

'context.settings' structure is no longer needed and should be removed

Originally created on Tue, 21 Feb 2017 01:20:23 +0900

Instead creating a copy of the repository with fixed code style and listing the files, CI system is expected to add comments on issues to fix rigth in files where they were found.

Originally created on Thu, 2 Mar 2017 17:02:20 +0900

A way to build some steps selectively is requested by our customers

Originally created on Tue, 21 Feb 2017 19:14:29 +0900

TeamCity can display errors, happening in the build, differently from all other logs.
This requires printing error logs via stderr stream.

Originally created on Mon, 3 Apr 2017 22:39:16 +0900

As new TC version (since 9.1) supports multicoloured logs, change errors color into red.

Originally created on Thu, 16 Mar 2017 16:02:43 +0900

Analyze the project and add doctest-style comments where appropriate.
Also test them with both doctest and Sphinx (current documentation system).

Originally created on Wed, 5 Apr 2017 01:45:29 +0900

1. Some steps might be launched while other are still executed
2. Ci system should proceed to artifact collecting and finalizing only when all steps are completed.

Originally created on Tue, 21 Feb 2017 18:39:58 +0900

'rss_parser.py' and 'build_teamcity.py' scripts are now included to TC artifacts, because TC artifacts now contain every file found in working directory.
This should be fixed (e.g. via creating special folder to put all atrifacts to).

Originally created on Thu, 16 Mar 2017 16:11:17 +0900

We need to refer to different versions of the CI system by verison number.

Originally created on Thu, 4 May 2017 00:50:01 +0900

Steps to reproduce:
Specify sync cl via -p4h or set SYNC_CL environment variable

Actual result:

[Warning]: '//depot/blahblahblah/...123 - no such file(s).'

Expected result:
Successfull build by this CL