artefactual-labs / archivematica-acceptance-tests Goto Github PK

With Xenial vagrant/ansible deploys, the vagrant/vagrant username/password assumption no longer holds, which breaks scp-reliant tests, which require password-authenticated scp.

The solution would seem to be:

1. add another option (and some documentation) to the acceptance tests, e.g., -D ssh_identity_file=/path/to/.vagrant/machines/am-local/virtualbox/private_key
2. modify scp_server_file_to_local of archivematicaselenium.py to use ssh_identity_file if present instead of the current server_password
3. add documentation explaining this

See artefactual/deploy-pub#42 for more details.

in core/performance-stdout-no-write.feature (WIP)

It's unclear how Molly would be alerted to missing PRONOM characterizations, if at all. These are frequent sources of errors in Columbia content.

This could be specified in a JSON/YAML file:

{
    "installations": [
        {
            "archivematica": "99922b55bf3c741d409957cb04f520cf989868a6",
            "archivematica-storage-service": "4ad0f042039810849e3f12047a1fc12cdb3b78c4",
            "archivematica-sampledata": "ab40cbfe9b17d0f2e23369f4ff176d149c0a7fcd"
        },
        {
            "archivematica": "qa/1.x",
            "archivematica-storage-service": "qa/0.x",
            "archivematica-sampledata": "master"
        }
}

The commit 3777b1c added a change to the get_am_sel_cli() method in environment.py. This method requires an input parameter now. It is called from two helper scripts (close_all_transfers.py and close_all_ingest.py), those scripts don't work now.

If only one feature can be run in CI/CD system, this should be the one. It should do and test the following:

• Create an AIP from a transfer source (using processing configuration or making decisions?)
  • normalize for preservation
• Verify that at least some MS outputs are as expected, e.g., by clicking on the gear icon of file identification and verifying 0 exit codes and substrings in stdout
• Wait for AIP to be created and inspect it and make assertions about it:
  • METS file is XSD-valid and Schematron-valid
  • preservation derivatives exist on disk
  • METS file has crucial state correct (PREMIS events, relationships, ...)
  • ...
• Attempt a re-ingest? (Breaking re-ingest seems a common consequence of new AM dev work...)

Profiling of Archivematica's MCPClient indicates that it takes roughly 1.25 seconds for a (Python-based) client script to be started in a new Python process and for it to import Django (and other heavy modules) and create a database connection. Given a package creation event with very many files each requiring several Pythonic client script process forks, this can result in significant inefficiencies.

In many cases this "cold startup" cost is unnecessary when the following could be done instead:

1. Python client scripts could be directly (dynamically) imported into the long-running Python (MCP Client) Gearman worker process instead of forking new Python processes for the short-lived client script.

2. If the client script must run in a separate process, then we could avoid the Django import and db connection by having the long-running MCPClient process provide to the client scripts the state (e.g., database data, Django settings, unit variables) that they need. Once done, the client script could pass back (e.g., as JSON printed to stdout) directives for the MCPClient process to mutate the database appropriately.

At present, the ArchivematicaUser logs are written to amuser/amuser.log and the step file logs are written to features/steps/steps.log. The tests should be changed so that it is easier to control where the logs get written. Maybe the default should be to log to stdout.

@sevein I think you have been doing work that may have been hampered by this issue. Do you have suggestions or a PR?

The work that solves AM issue 763 needs an executable feature file to test for correct behaviour. This issue asserts the need for such a feature file.

The Then the transfer on disk is encrypted step can fail because even though the And the user waits for the DIP to appear in transfer backlog step has passed, the transfer is not actually yet stored at the expected path on disk:

File "/Users/user/.pyenv/versions/3.4.2/lib/python3.4/subprocess.py", line 620, in check_output
      raise CalledProcessError(retcode, process.args, output=output)
  subprocess.CalledProcessError: Command '['docker', 'cp', '2664c8ab5cc9f902a34db8155f41964cd5736b96434687aa0ce28751dfc0785b:/var/archivematica/sharedDirectory/www/AIPsStore/transferBacklogEncrypted/originals/BagTransfer_1525201451-ec84292f-7192-4cdc-93b0-53f4db26561d', '/Users/user/Documents/Artefactual/am/src/archivematica-acceptance-tests/.amsc-tmp/BagTransfer_1525201451-ec84292f-7192-4cdc-93b0-53f4db26561d']' returned non-zero exit status 1

Create a feature file that describes anticipated behaviour when the same transfer is run through two different Archivematica deployments which are identical except that one has client scripts that pass stdout/stderr back to MCPServer to be stored in the database and the other simply discards such stdout/stderr. The feature file should express an expectation that the stdout/stderr-discarding system should have significant relative performance gains.

This is because these features/tests require scp/ssh and docker containers don's allow that.

Cf.:

Given that the user has ensured that the default processing config is in its default state                                                                                       # features/steps/steps.py:43 270.139s
    And the reminder to add metadata is enabled                                                                                                                                      # features/steps/steps.py:49 18.912s
    When a transfer is initiated on directory ~/archivematica-sampledata/SampleTransfers/Images/pictures                                                                             # features/steps/steps.py:374 47.116s
    Traceback (most recent call last):
        File "/Users/jesus/archivematica-acceptance-tests/.env/lib/python3.6/site-packages/behave/model.py", line 1329, in run
        match.run(runner.context)
        File "/Users/jesus/archivematica-acceptance-tests/.env/lib/python3.6/site-packages/behave/matchers.py", line 98, in run
        self.func(context, *args, **kwargs)
        File "features/steps/steps.py", line 376, in step_impl
        utils.initiate_transfer(context, transfer_path)
        File "/Users/jesus/archivematica-acceptance-tests/features/steps/utils.py", line 143, in initiate_transfer
        accession_no=accession_no, transfer_type=transfer_type))
        File "/Users/jesus/archivematica-acceptance-tests/amuser/am_browser_transfer_ability.py", line 70, in start_transfer
        self.approve_transfer(transfer_div_elem, approve_option_uuid)
        File "/Users/jesus/archivematica-acceptance-tests/amuser/am_browser_transfer_ability.py", line 235, in approve_transfer
        approve_transfer_option.click()
        File "/Users/jesus/archivematica-acceptance-tests/.env/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 80, in click
        self._execute(Command.CLICK_ELEMENT)
        File "/Users/jesus/archivematica-acceptance-tests/.env/lib/python3.6/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
        return self._parent.execute(command, params)
        File "/Users/jesus/archivematica-acceptance-tests/.env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
        self.error_handler.check_response(response)
        File "/Users/jesus/archivematica-acceptance-tests/.env/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
        raise exception_class(message, screen, stacktrace)
    selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
        (Session info: chrome=66.0.3359.117)
        (Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Linux 4.4.0-119-generic x86_64)

The feature files in this repo are not currently organized into namespaces or folders. It is hard to tell which features are 'core' Archivematica features and which are related to new development work.

E.g., this

Given directory <transfer_path> contains files that, when normalized, will all <do_files_conform> to <policy_file>

Needs a step implementation in steps.py. However, its ilk will require (assumedly) running a particular command locally/separately during test execution. How do we get the file? The easy way would be to hardcode the github URL of an AM command into the steps.py file...

Regarding am issue #2

There is a need for running acceptance tests on the archivematica docker hub ( https://github.com/artefactual-labs/am ).

A new branch called acceptance-tests-docker will be created on the acceptance-test repo that includes the dockerfile and scripts.

This issue should be used to document the purpose and usage of the performance-stdout-no-write feature. The end result may be a set of slides, additions to the docs/ folder of this repo or some other document that helps end users to make use of this performance-measuring feature (and similar future features).

There are many places in these tests where timeouts are used in an ad hoc, de-centralized way. In other cases, the tests run in an infinite loop until some expected event happens in the Archivematica GUI. This is problematic especially in a CI context where we need the tests to fail without human intervention.

In some cases the timeout should be written right into the feature file since the time estimate for a task depends on the nature of the transfer being processed.

In other cases, a centralized set of timeout variables should be used and these should be made configurable from behave user data flags.

In cases where there are infinite loops without logical breakpoints, we need to introduce a such a breakpoint and tie its timeout to one of the above two strategies.

@sevein if you have come across particular places in the tests where this type of issue has been a pain point, please note them here.

Columbia University Library is sponsoring development work in Archivematica to improve the performance of automated pipelines. The first phase of this work involves removing ElasticSearch from the components that are installed with Archivematica.

When Archivematica is run in an entirely automated configuration - with new content being ingested, processed and packaged into AIP's entirely without human intervention, using Elasticsearch does not provide any benefits. This configuration can be referred to as 'headless', in the sense that the dashboard component is not being used as a gui, but only for the REST API to control the flow of content through the system.

Excluding Elasticsearch from the list of components installed and used by Archivematica will reduce the operational complexity and the required compute resources.

We need a Feature File that describes the expected behaviour when installing or upgrading Archivematica, to be clear how the process should work to allow both this new mode to operate without breaking existing installations. The Feature File will also need to describe the expected behaviour when a headless system is running.

Thee things should be obvious:

• What is it?
• Why was it created?
• What is it good for?
• How is it built?
• How can I run the tests? (The tests have tests? Wow, so meta!)
• Where is the entry point if I want to start hacking on it?
• How is the code structured?

We need to know where strange encodings break Archivematica and when this is ok.

Sources of strange bytestrings:

• transfer names (from dashboard web form)
• names of directories constituting transfers
• names of files in transfers
• strings within CSV files (e.g., metadata.csv)

Types of bytestring strangeness:

• Unicode UTF-8
  • in NFC and NFD (http://unicode.org/reports/tr15/#Norm_Forms)
• other encodings: Windows-1252, Latin-1, Shift-JIS

One possible metric of performance changes is a lower connection timeout configuration.

The use of both master and qa/1.x as production branches is confusing. We should just have master and follow semver and have releases.

in Columbia/indexless.feature
manual and ansible are in the WIP document. It's not clear to us whether manual is intended to mean "via package, e.g. rpm". If not, package should be added to the scenario's methods.

After disabling elasticsearch at fresh install, improve the user interface so that it's clear in AM dashboard that indexing is turned off.

If a value such http://1.2.3.4:80 is passed to am_url where 1.2.3.4 is any network location and the port is 80 and it's not omitted then the tests misbehave when URLs are compared because the browser may omit the port 80 or 443 as they're standard ports.

At least this function is misbehaving:

Where self.driver.current_url and url don't match

• driver.current_url: http://1.2.3.4/administration/processing/edit/default/
• url: http://1.2.3.4:80/administration/processing/edit/default/

Workaround: pass -D am_url=http://1.2.3.4/ instead of -D am_url=http://1.2.3.4:80/.

The When the AIP is deleted step fails because driver.find_element_by_id('DataTables_Table_0_filter') in am_browser_ss_ability.py now returns a <div> when previously it returned an <input> (I believe). This triggers an exception and the following stacktrace:

@allow-orphaned-key-delete
Scenario: Richard wants to ensure that GPG deletion is never permitted if the key is associated to a space or if it is needed to decrypt an existing package. However, if all space associations are destroyed and all dependent packages deleted, then deletion of the (orphaned) key should be permitted.  # features/core/aip-encryption.feature:89
  ...
  When the AIP is deleted                                                                                                                                                                                                                                                                                    # features/steps/steps.py:378 2.818s
    Traceback (most recent call last):
      File "features/steps/steps.py", line 382, in step_impl
        context.am_user.browser.approve_aip_delete_request(uuid_val)
      File "/Users/user/Documents/Artefactual/am/src/archivematica-acceptance-tests/amuser/am_browser_ss_ability.py", line 33, in approve_aip_delete_request
        self.driver.find_element_by_id('DataTables_Table_0_filter').send_keys(aip_uuid)
    selenium.common.exceptions.WebDriverException: Message: unknown error: cannot focus element
      (Session info: chrome=65.0.3325.181)
      (Driver info: chromedriver=2.38.552518 (183d19265345f54ce39cbb94cf81ba5f15905011),platform=Mac OS X 10.13.4 x86_64)

Need a feature file that describes how to upgrade (as opposed to a fresh install) Archivematica without Elasticsearch.

See https://github.com/artefactual-labs/ansible-archivematica-src/blob/stable/1.6.x/tasks/acceptance-tests.yml

So animate them!

Solution: Write a script that runs the tests at human-comprehension speed with the Gherkin statements added in a subtitles srt file.

• implement a "human speed" switch in archivematicaselenium.py with minimal code intrusion and which has an intelligent metric for determining how long to pause where. HARD PART.
• Extract time indices from behave output.
• Generate .srt file.
• Automate screen capture in tandem with CI (vnc2swf script http://www.unixuser.org/~euske/vnc2swf/). Other hard part.

The get_mets method in am_browser_ingest_ability.py assumes that when a METS file is clicked on, the new window with the XML in it will appear immediately. However, on production (remotely served) applications this is a false assumption and the "XML" extracted will simply be the HTML source of a not-yet-fully-loaded page.

If a file is so far down in the transfer source file browser that it is not visible, then the Selenium logic is currently failing to click on it. Note: this may be fixed for clicking to open a directory but not clicking to select a directory.

Hello,

As part of the feature testing, we'd like to save settings that we have toggled on and off in the dashboard to ensure reproducibility between our environment and the test environments we're setting up. This would be to ensure that we're running the same pipeline that we would normally run on our data in prod.

The documentation under Processing Configuration indicates that these dashboard settings should be defined in a file named processingMCP.xml (whose file name is configurable in the MCPServer config file).

I was unable to find processingMCP.xml on our current prod server. The filename assigned to the processingXMLFile field under MCPServer in the MCPServer config does not seem to have been changed from the default so if the config were written out somewhere based on the value of this parameter it should be where it would normally be in a stock Archivematica installation.

There is a file located within the directory assigned to the sharedDirectoryMounted in the MCPClient config that looks similar to what processingMCP.xml should look like- this file is found at ./sharedMicroServiceTasksConfigs/processingMCPConfigs/defaultProcessingMCP.xml under directory defined by sharedDirectoryMounted. The chief difference between this file and what we'd expect from processingMCP.xml, however, is that whereas we'd expect processingMCP.xml to use prose-based names for the applies and goToChain fields (like here), instead it populates these fields with UUIDs.

This leads to the following questions on our end:

• Is defaultProcessingMCP.xml the correct file we'd want to use to ensure our pipeline settings are preserved on the test hosts?
• Are the UUIDs assigned to the microservices on our prod host (and used within the XML config) portable across hosts? Can we drop in defaultProcessingMCP.xml on the test hosts and have it work without any further configuration/tweaking?

There are a number of reasons for this:

1. Lack of guidelines on how to contribute.

2. Code organization is lacking, e.g.,

   • modules are too big
   • concerns are not separated, e.g., GUI/Selenium calls mixed with REST API calls
   • steps file too big, should be split into many

in core/performance-stdout-no-write.feature (WIP)

the AIP output METS should be identical between instance A and instance B, but the process METS would be smaller.

SS PR 320 introduced the posix_move method of the Space class, which moves files directly from one location to another without the intermediary move to a staging path. We need a feature file that describes our expectation that this change results in a performance improvement and we need that feature file to provide an automated confirmation of that expectation.

We need to institute and implement standard methods/workflows for indicating how a specific feature (or feature file) is to be run. This should be in the documentation and ideally also in the feature files themselves as comments and also in a makefile or shell script that can run all features or individual ones.

Feature file(s) needed to describe the virus testing that needs to be done for the following scenarios:

1. files < 20M that have virus
2. files < 20M that don’t have virus
3. files > 20 M that have virus
4. files > 20M that don’t have virus

And also for this issue: JiscSD/rdss-archivematica#104

Related: artefactual/archivematica#772

in Columbia/indexless.feature
there is an ambiguity about whether review is possible without the Index (transfer) service - the decision point suggests it is, but the mechanics are not clear to us. Can the scenario be clarified as to whether review is actually possible?

Current feature file only includes manual method of disabling elasticsearch. Need to also automate this process.

This would appear to be a regression.

AM PR 1012 aimed to increase the efficiency of checksum verification. We need a feature file that describes our expectation that this change results in a performance improvement and we need that feature file to provide an automated confirmation of that expectation.

After analysing how micro-services work in parallel when creating an AIP in a test environment, now need to determine how those parallels play out using Columbia data

There are too many ways being used currently to do the following:

1. Perform some action, typically initiating a transfer/ingest
2. Make a series of decisions until something happens:
   • an AIP is created
   • an AIP is available for review
   • a particular micro-service has completed
3. Perform some second action, given the system/AIP is in the state guaranteed by (2)

We need a clear and memorable way of expressing, with minimal specification, common steps. For example:

When the user makes standard decisions to go from decision point "<DECISON_POINT_A>" to "<DECISON_POINT_B>"

or:

Given a processing config with standard decisions to go from decision point "<DECISON_POINT_A>" to "<DECISON_POINT_B>"

Where "to go from" implicitly assumes a particular sub-chain of the workflow, and "standard decisions" are defined how?

We have a project for the Canadian Centre for Architecture that has two deliverables:

• a script to create DIPs in a particular structure for CCA researchers
• a script to upload those DIPs to an instance of Atom.

Both of these scripts need acceptance tests to specify the desired functionality and supported automated testing of that functionality.

Update to Selenium 3.x

This may result in inconsistent or unexpected results. A different transfer source should probably be used.