Need to include these & also figure out why / how these get updated. Pinging @bibliotechy on this for the latter

Closes tulibraries/ansible-playbook-airflow#19

This nullifies linode-terraform client rebuild issues & allows us to address #30 via S3 object expiration. Nullifies need for something like Ceph on linode as well, meaning any service can grab data from s3 as long as it knows the timestamp (via db query?).

When the bashoperator runs a bash script, even as user airflow, it runs bash as a non-interactive, non-login shell; thus, the bash script cannot access rbenv (or any of the airflow users bash environment setup).

For now, this is temporarily fixed by sourcing bashrc in the bash scripts themselves; in the future, what should we do to fix this? bash -l for the BashOperator? Is there a missing profile to be loaded?

Add unit tests in this repo that test methods in tasks files.

Like:

• extract_members
• extract_child
  etc

We need to be extra careful when we are running a full re-index for production. Production currently uses two solr instances to manage full re-indexing. We run a full reindex on the instance that is not being used and then swap it with the instance that is being used.

Create a check that ssh's into production and retrieves the current $SOLR_URL and exit out of the full reindex process if by some mistake we are trying to run a full reindex job on the current solr instance.

We currently use both AZ_CORE and AZ_CONFIGSET. They hold essentially the same type of data which we use to generate the AZ solr_url.

Once we move over completely to Solrcloud we should remove AZ_CORE and just use AZ_CONFIGSET.

The ingest web content dag is currently failing to ingest some records but not reporting the failure.

• question of database backend
• database replication question
• airflow generic install with syncing only for DAGs & related scripts

• Confirm Alma OAI's last_updated field(s) per record in the ADM enhanced field accurately reflects the last_updated date of the bibliographic record, holdings, and possibly items. A test set of these records, with new enrichment added, is available on the FTP server (file name test_alma_bibs_2019041016_11157338920003811_new.1.xml)
• Use an updated traject mapping in tul_cob; see tulibraries/tul_cob#1113
• Shift OAI harvest time ranges to cover possible dropping of records between Alma OAI publication jobs (every 6 hours)

We have document versioning on in Solr; this can sometimes cause Solr to respond 409 for a full reindex; in current process, doesn't stop the traject process, just throws errors in the logfile;
but for Airflow sees that error code and considers the task failing;
can the airflow script process become a wrapper around traject to swallow the error;
and traject in that issue tries to retry individual + slows process.

• Full reindex using Airflow QA to Solr Dev 1
• Full reindex using Jenkins to Solr Dev 2

Hold off for turning on partial (OAI) updates on these for dev review; Requires tulibraries/grittyOps#59

If we move all the dags to a folder called "dags" this will be easier to dockerize.

See this file structure as a good example: https://github.com/puckel/docker-airflow

We need a DAG to hit API endpoints from the new website and push that data into solr.

Make sure BashOperator will actually report back failures.

Currently this is a manual process, since the entire Alma sftp export is manual anyway.
Do we want to automate it with logrotate (my preference)? Or have an airflow task do it?
And how many do we keep? (Disk space vs ability to look back considerations.)
Keep in mind we may greatly increase the frequency of full reindexing.

so the indexing step (bash operator) needs to create presigned S3 urls to pass to traject instead of the current http urls.

Looks like we hit solrcloud url with http vs. https which could be exposing our basic auth ... this happens because in connection.host we dont' specify http/http and instead add that programmatically.

In the boundwith implementation pull Variable.get calls to the top level of the dag.

Switch the boundwith implementation in #91 to use s3 instead of local storage.

This should effect the OAI harvesting and http api calls.

Additionally, update the reads as well to use the s3 location.

Steps:

• Gets number of solr docs for Web from SolrCloud Web Alias
• Creates new Web SolrCloud collection
• Indexes to new Web SolrCloud collection (ending with Web SolrCloud collection healthcheck)
• Swaps Web SolrCloud Alias (same name as configset) to point to newly created & indexed SolrCloud collection (see funcake dags for example of command)
• Gets number of solr docs for Web from SolrCloud Web Alias

Stetch goals:

• remove delete all records step from indexing process
• used tulflow shared Slack tasks instead of locally defined tasks
• Abstract out SolrCloud HTTP Operator tasks for reuse across cob_datapipeline (and then maybe to tulflow for reuse across all possible DAGs)

Error from webserver logs: "GET /static/appbuilder/fonts/glyphicons-halflings-regular.woff2 HTTP/1.1" 404 3695 "https://airflow.qa.tul-infra.page/static/appbuilder/css/bootstrap.min.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0"

The current OAI harvest task, almaoai_harvest.py, has a lot of baked in assumptions about the data that is coming in that block it from being reused for non Alma OAI feeds.

This includes, but is not limited to:

• wrapping the data in a collection tag
• Sickle harvest arguments

The harvest task should be refactored into a more generic OAI harvester, allowing extra steps to be added via parametrization

Steps:

• Gets number of solr docs for AZ from SolrCloud AZ Alias
• Creates new AZ SolrCloud collection
• Indexes to new AZ SolrCloud collection (ending with AZ SolrCloud collection healthcheck)
• Swaps AZ SolrCloud Alias (same name as configset) to point to newly created & indexed SolrCloud collection (see funcake dags for example of command)
• Gets number of solr docs for AZ from SolrCloud AZ Alias

Stetch goals:

• remove delete all records step from indexing process
• used tulflow shared Slack tasks instead of locally defined tasks

Most of the code dealing with sftp dumps assumes there will be only one set of files present at a time. For a variety of reasons, that assumption can fail and bad things can happen as a result.

A regex for extracting the date from an export file exists; could we use that in other places to ensure we only operate on the latest dump?

On the other hand, reasoning about the state of sftp exports is just a heuristic and probably prone to edge cases. Ideally we'd want to start with having the sftp server pexpect code only download the latest export (and have that date used for all subsequent steps), but that process is hacky enough already.

Get the fork of traject curently being used in airlfow onto github in a separate repository so it can be used in automation / be maintained.

Solr currently returns nonzero if there was a solr internal failure, but not if the query was not matched and nothing was deleted.
It would be nice to know for sanity checking if delete records are actually deleted, and also when they are preempted by another create record in the same harvest.

Is there a small fix here via getting the right file permissions for that bash script checked into the git repo itself?

• VM for Airflow
• Mount / Volume for Airflow VM
• Run ansible-server-bootstrap playbook on above
• Build the above in terraform using remote state

See output @relaxing experienced:

Apr  9 21:13:47 li1241-105 airflow: /bin/sh: line 0: exec: bash: not found
Apr  9 21:13:47 li1241-105 airflow: [2019-04-09 21:13:47,172] {local_executor.py:91} ERROR - Failed to execute task Command 'exec bash -c 'airflow run tul_cob_reindex get_num_solr_docs_pre 2019-04-09T20:59:52.378315+00:00 --local -sd /var/lib/airflow/airflow/dags/cob_datapipeline/tul_cob_fullreindex_dag.py'' returned non-zero exit status 127..

Create a DAG that follows the patterns of the ALMA OAI harvesting dag to harvest data from Combine's OAI endpoint and sends it to Solr for Blacklight.

Blocked by #82

Need to rotate out old archived dumps, whether that involves logrotate or not.

Full reindex or partial index should not include it for now.