gsa / catalog.data.gov Goto Github PK
View Code? Open in Web Editor NEWDevelopment environment for catalog.data.gov
Home Page: https://catalog.data.gov
Development environment for catalog.data.gov
Home Page: https://catalog.data.gov
Catalog: https://admin-catalog.data.gov/dataset/annual-performance-reports-aprs
Sandbox: https://catalog-next.sandbox.datagov.us/dataset/annual-performance-reports-aprs
You click on Search datasets within this collection within a collection and should be able to see the children datasets like this one
When you click Search datasets within this collection within a collection you see a page with 0 datasets like this one
Workflow with Issue: restart application
Job Failed: restart-prod
Last Commit: 768a246
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3388445701
Expected Behavior
Actual Behavior
After we import harvest sources we need to validate we have all the information we expect
At the final report about harvest sources in the new catalog we found some error that make us think that we fail while import this harfest sources.
Some sources faileds with validation errors:
source | url | config | added | updated | nulls | errors |
---|---|---|---|---|---|---|
epa-sciencehub | https://pasteur.epa.gov/metadata.json | {"private_datasets": "False"} | 1665 | 0 | -1 | Title: EPA True NO2 ground site measurements. 1 Error(s) Found. ### ERROR 1: 'distribution':[{'accessURL': 'http://www-air.larc.nasa.gov/cgi-bin/ArcView/disco ... |
mcc-data-json | https://data.mcc.gov/data.json | {"private_datasets": "False"} | 199 | 0 | -1 | Title: Ghana - Agriculture - Land Tenure; 1 Error(s) Found. ### ERROR 1: 'distribution':[{'downloadURL': 'http://microdata.worldbank.org/index.php/cata ... |
healthdata-gov | https://healthdata.gov/data.json | {"private_datasets": "False"} | 410 | 0 | 1302 | Title: Commercial Medical Insurance 1 Error(s) Found. ### ERROR 1: 'references':['https://wwwdev.cdc.gov/visionhealth/vehss/data/claims/marketscan.html; https://marketscan.truvenhealth.com/marketscanportal/'] is not valid under any of the given schemas ... |
energystar | http://data.energystar.gov/data.json | {"private_datasets": "False"} | 54 | 0 | -1 | Title: ENERGY STAR Certified Ceiling Fans Light Kits; 1 Error(s) Found. ### ERROR 1: 'keyword' is a required proper ... |
Workflow with Issue: restart application
Job Failed: restart-prod
Last Commit: 783acbb
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3607526730
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: 752fe44
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3274120278
We need a long term strategy for testing multiple versions of CKAN in the extensions
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: 799961c
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3281798821
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: 783acbb
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3639953041
Source with config: '{"validator_profiles": ["fgdc_minimal"]}
fails with Unknown validation profile(s): fgdc_minimal
Harvest source failing: fema-r10
Run the importer for this source:
python import_harvest_sources.py --names=fema-r10 --destination_api_key=b396d3d9-1f24-4b72-92c1-78f981009a92
Full importer error
****** creating fema-r10: 7 of 289 sources
remote_ckan.lib - Get organization data https://catalog.data.gov/api/3/action/organization_show {'id': 'fema-gov'}
remote_ckan.lib - organization fema-gov saved at /checks
remote_ckan.lib - New extra found at org [email protected]
[email protected]
[email protected]
remote_ckan.lib - New extra found at org organization_type=Federal Government
remote_ckan.lib - Creating organization fema-gov
remote_ckan.lib - Get organization data https://catalog-next.sandbox.datagov.us/api/3/action/organization_show {'id': 'fema-gov'}
remote_ckan.lib - organization fema-gov saved at /checks
remote_ckan.lib - Request OK https://catalog-next.sandbox.datagov.us/api/3/action/organization_update
remote_ckan.lib - Config found in extras: {'value': '{"validator_profiles": ["fgdc_minimal"], "private_datasets": false}', 'key': 'config'}
remote_ckan.lib - Creating harvest source FEMA-R10
remote_ckan.lib - ERROR status: 409
content:b'{"help": "https://catalog-next.sandbox.datagov.us/api/3/action/help_show?name=harvest_source_create", "success": false, "error": {"__type": "Validation Error", "config": ["Error parsing the configuration options: Unknown validation profile(s): fgdc_minimal"]}}'
sent: {'name': 'fema-r10', 'owner_org': 'fema-gov', 'title': 'FEMA-R10', 'url': 'https://hazards.fema.gov/filedownload/metadata/R10/', 'notes': '', 'source_type': 'waf', 'frequency': 'WEEKLY', 'config': '{"validator_profiles": ["fgdc_minimal"], "private_datasets": false}', 'extras': [], 'id': 'fd47c02e-3ef2-42fb-b4e5-40560b136a08'}
After fix #43 we are not able to get inside resources of a spatial dataset
Related to #307
Duplicated from #1690
How to reproduce
Add a spatial dataset (e.g. use a harvest source as in #43)
Full error log
HelperError: Helper 'archiver_is_resource_broken_line' has not been defined.
View as: Interactive (full) | Text (full) | XML (full)
Module ckan.controllers.package:1147 in resource_read view
>> return render(template, extra_vars=vars)
Module ckan.lib.base:125 in render view
>> return cached_template(template_name, renderer)
Module pylons.templating:249 in cached_template view
>> return render_func()
Module ckan.lib.base:162 in render_template view
>> return render_jinja2(template_name, globs)
Module ckan.lib.base:94 in render_jinja2 view
>> return template.render(**extra_vars)
Module jinja2.environment:989 in render view
>> return self.environment.handle_exception(exc_info, True)
Module jinja2.environment:754 in handle_exception view
>> reraise(exc_type, exc_value, tb)
Module /srv/app/src/ckan/ckanext/datastore/templates/package/resource_read.html:1 in top-level template code view
>> {% ckan_extends %}
Module /usr/lib/python2.7/site-packages/ckanext/datagovtheme/templates/package/resource_read.html:46 in top-level template code view
>> {% set pkg = c.pkg_dict %}
Module /srv/app/src/ckan/ckan/templates/package/resource_read.html:3 in top-level template code view
>> {% set res = c.resource %}
Module /usr/lib/python2.7/site-packages/ckanext/datagovtheme/templates/package/base.html:3 in top-level template code view
>> {% set pkg = c.pkg_dict %}
Module /usr/lib/python2.7/site-packages/ckanext/datagovtheme/templates/page.html:1 in top-level template code view
>> {% ckan_extends %}
Module /srv/app/src/ckan/ckan/templates/page.html:1 in top-level template code view
>> {% extends "base.html" %}
Module /usr/lib/python2.7/site-packages/ckanext/harvest/plugin/../templates/base.html:1 in top-level template code view
>> {% ckan_extends %}
Module /usr/lib/python2.7/site-packages/ckanext/datagovtheme/templates/base.html:1 in top-level template code view
>> {% ckan_extends %}
Module /srv/app/src/ckan/ckan/templates/base.html:101 in top-level template code view
>> {%- block page %}{% endblock -%}
Module /srv/app/src/ckan/ckan/templates/page.html:19 in block "page" view
>> {%- block content %}
Module /srv/app/src/ckan/ckan/templates/page.html:22 in block "content" view
>> {% block main_content %}
Module /srv/app/src/ckan/ckan/templates/page.html:53 in block "main_content" view
>> {% block pre_primary %}
Module /srv/app/src/ckan/ckan/templates/package/resource_read.html:22 in block "pre_primary" view
>> {% block resource %}
Module /srv/app/src/ckan/ckan/templates/package/resource_read.html:24 in block "resource" view
>> {% block resource_inner %}
Module /srv/app/src/ckan/ckan/templates/package/resource_read.html:27 in block "resource_inner" view
>> {% block resource_actions %}
Module /srv/app/src/ckan/ckan/templates/package/resource_read.html:29 in block "resource_actions" view
>> {% block resource_actions_inner %}
Module /srv/app/src/ckan/ckanext/datastore/templates/package/resource_read.html:4 in block "resource_actions_inner" view
>> {{ super() }}
Module /usr/lib/python2.7/site-packages/ckanext/datagovtheme/templates/package/resource_read.html:25 in block "resource_actions_inner" view
>> {{ h.archiver_is_resource_broken_line(c.resource) }}
Module jinja2.environment:412 in getattr view
>> return obj[attribute]
Module ckan.lib.helpers:87 in __getitem__ view
>> key=key
HelperError: Helper 'archiver_is_resource_broken_line' has not been defined.
One difference between harvest upstream version and the GSA fork is that the fork changes the order in which we harvest.
The DCAT-US harvester adds datajson_collection to source config to be used at the harvester ext to divide the harvest process first for parents and then for children.
First, we harvest father datasets and then the children.
We already have a test case for DCAT-US with collections but this test does not cover this feature.
Workflow with Issue: publish & deploy
Job Failed: deploy-prod
Last Commit: a79947f
Number of times run: 1
Last run by: robert-bryson
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3347622218
Workflow with Issue: publish & deploy
Job Failed: deploy-prod
Last Commit: 52602ab
Number of times run: 2
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3277106748
Analyze options to update the integration tests for this repo
Technology | Pros | Cons |
---|---|---|
Bats | It's already working | Is not easy to create new tests |
Cypress | Is what upstream CKAN is using now. We can re-use some preexisting tests for CKAN | Require to migrate current tests |
Regular CKAN test | We current use this for extensions | Require to migrate current tests |
Selenium | Is a more general tool and could be used with python | Require to migrate current tests |
Workflow with Issue: publish & deploy
Job Failed: deploy-staging
Last Commit: 7fe90d9
Number of times run: 2
Last run by: robert-bryson
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3363518037
GSA/data.gov#2004 moved some things around. The instructions in the Usage section of the README in GSA/catalog.data.gov should be updated to reflect the new locations.
How to reproduce
When using the import script
$ pipenv run python import_harvest_sources.py --source_ty=datajson --limit=3 --destination_url=https://catalog-next.sandbox.datagov.us --destination_api_key=$CKAN_API_KEY
https://catalog.data.gov/api/3/action/harvest_source_show?id=11cd07d8-f5a7-4207-9852-6160e09ff240
gives a 500 error, the name is u-s-forest-service-geospatial-data-discovery Harvest Source
Expected behavior
Harvest source should be imported
Actual behavior
https://catalog.data.gov/api/3/action/harvest_source_show?id=11cd07d8-f5a7-4207-9852-6160e09ff240
gives a 500 error, the name is u-s-forest-service-geospatial-data-discovery Harvest Source
Workflow with Issue: publish & deploy
Job Failed: deploy-prod
Last Commit: fa16430
Number of times run: 1
Last run by: robert-bryson
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3379442931
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: 1023c85
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3275169714
I updated the datagovtheme
extension today and after run make update-dependencies
I didn't see the new version in the requirement file
Logs:
make update-dependencies
docker-compose run --rm -T ckan freeze-requirements.sh 1000 1000
Starting catalogdatagov_redis_1 ... done
Starting catalogdatagov_db_1 ... done
Running poetry lock ...
Creating virtualenv ckan-catalog-C0o-n6lY-py2.7 in /root/.cache/pypoetry/virtualenvs
Using virtualenv: /root/.cache/pypoetry/virtualenvs/ckan-catalog-C0o-n6lY-py2.7
Updating dependencies
Resolving dependencies...
1: fact: ckan-catalog is 0.1.0
1: derived: ckan-catalog
1: fact: ckan-catalog depends on ckanext-datagovcatalog (0.0.1 git)
1: fact: ckan-catalog depends on repoze.who (2.3)
1: fact: ckan-catalog depends on psycopg2 (2.7.3.2)
1: fact: ckan-catalog depends on Flask (0.12.4)
1: fact: ckan-catalog depends on OWSLib (0.8.6)
1: fact: ckan-catalog depends on cffi (1.12.3)
1: fact: ckan-catalog depends on pyutilib.component.core (4.6.4)
1: fact: ckan-catalog depends on fanstatic (0.12)
1: fact: ckan-catalog depends on messytables (>=0.15.2)
1: fact: ckan-catalog depends on gevent (*)
1: fact: ckan-catalog depends on ckanext-report (0.1 git)
1: fact: ckan-catalog depends on GeoAlchemy (>=0.6)
1: fact: ckan-catalog depends on newrelic (*)
1: fact: ckan-catalog depends on pyparsing (>=2.1.10)
1: fact: ckan-catalog depends on pysolr (3.6.0)
1: fact: ckan-catalog depends on xlrd (>=1.0.0)
1: fact: ckan-catalog depends on ckan (2.8.6 git)
1: fact: ckan-catalog depends on sqlparse (0.2.2)
1: fact: ckan-catalog depends on lxml (>=2.3)
1: fact: ckan-catalog depends on rq (0.6.0)
1: fact: ckan-catalog depends on ckanext-googleanalyticsbasic (0.1 git)
1: fact: ckan-catalog depends on passlib (1.7.3)
1: fact: ckan-catalog depends on greenlet (0.4.12)
1: fact: ckan-catalog depends on polib (1.0.7)
1: fact: ckan-catalog depends on sqlalchemy-migrate (0.10.0)
1: fact: ckan-catalog depends on WebHelpers (1.3)
1: fact: ckan-catalog depends on redis (2.10.6)
1: fact: ckan-catalog depends on google_compute_engine (2.8.13)
1: fact: ckan-catalog depends on click (6.7)
1: fact: ckan-catalog depends on celery (3.1.25)
1: fact: ckan-catalog depends on ckanext-qa (2.0 git)
1: fact: ckan-catalog depends on gunicorn (*)
1: fact: ckan-catalog depends on WebTest (1.4.3)
1: fact: ckan-catalog depends on simplejson (3.10.0)
1: fact: ckan-catalog depends on kombu (3.0.37)
1: fact: ckan-catalog depends on werkzeug (~0.15.3)
1: fact: ckan-catalog depends on Babel (2.3.4)
1: fact: ckan-catalog depends on python-magic (0.4.15)
1: fact: ckan-catalog depends on Jinja2 (~2.10.1)
1: fact: ckan-catalog depends on ckanext-geodatagov (0.1 git)
1: fact: ckan-catalog depends on ckanext-harvest (1.3.2 git)
1: fact: ckan-catalog depends on Paste (1.7.5.1)
1: fact: ckan-catalog depends on SQLAlchemy (1.1.11)
1: fact: ckan-catalog depends on Genshi (0.7.3)
1: fact: ckan-catalog depends on cryptography (2.7)
1: fact: ckan-catalog depends on rfc3987 (*)
1: fact: ckan-catalog depends on urllib3 (1.25.9)
1: fact: ckan-catalog depends on ckanext-archiver (2.0.0 git)
1: fact: ckan-catalog depends on ply (3.4)
1: fact: ckan-catalog depends on ckanext-spatial (0.2 git)
1: fact: ckan-catalog depends on lepl (*)
1: fact: ckan-catalog depends on pytz (2016.7)
1: fact: ckan-catalog depends on tzlocal (1.3)
1: fact: ckan-catalog depends on PasteScript (2.0.2)
1: fact: ckan-catalog depends on bleach (~3.1.1)
1: fact: ckan-catalog depends on python-dateutil (<2.0.0,>=1.5.0)
1: fact: ckan-catalog depends on repoze.who-friendlyform (1.0.8)
1: fact: ckan-catalog depends on Shapely (>=1.2.13)
1: fact: ckan-catalog depends on Routes (1.13)
1: fact: ckan-catalog depends on argparse (*)
1: fact: ckan-catalog depends on pyyaml (*)
1: fact: ckan-catalog depends on ckanext-envvars (*)
1: fact: ckan-catalog depends on pika (>=1.1.0)
1: fact: ckan-catalog depends on boto (*)
1: fact: ckan-catalog depends on Flask-Babel (0.11.2)
1: fact: ckan-catalog depends on pyOpenSSL (18.0.0)
1: fact: ckan-catalog depends on WebOb (1.0.8)
1: fact: ckan-catalog depends on Pylons (0.9.7)
1: fact: ckan-catalog depends on ckantoolkit (0.0.3)
1: fact: ckan-catalog depends on GeoAlchemy2 (0.5.0)
1: fact: ckan-catalog depends on zope.interface (4.3.2)
1: fact: ckan-catalog depends on progressbar (2.3)
1: fact: ckan-catalog depends on ofs (0.4.2)
1: fact: ckan-catalog depends on Pairtree (0.7.1-T)
1: fact: ckan-catalog depends on unicodecsv (>=0.9)
1: fact: ckan-catalog depends on ckanext-datagovtheme (0.1 git)
1: fact: ckan-catalog depends on ckanext-datajson (0.1 git)
1: fact: ckan-catalog depends on requests (~2.20.0)
1: fact: ckan-catalog depends on Markdown (~3.1)
1: fact: ckan-catalog depends on vdm (0.14)
1: fact: ckan-catalog depends on jsonschema (2.4.0)
1: selecting ckan-catalog (0.1.0)
1: derived: jsonschema (2.4.0)
1: derived: vdm (0.14)
1: derived: Markdown (~3.1)
1: derived: requests (~2.20.0)
1: derived: ckanext-datajson (0.1 git)
1: derived: ckanext-datagovtheme (0.1 git)
1: derived: unicodecsv (>=0.9)
1: derived: Pairtree (0.7.1-T)
1: derived: ofs (0.4.2)
1: derived: progressbar (2.3)
1: derived: zope.interface (4.3.2)
1: derived: GeoAlchemy2 (0.5.0)
1: derived: ckantoolkit (0.0.3)
1: derived: Pylons (0.9.7)
1: derived: WebOb (1.0.8)
1: derived: pyOpenSSL (18.0.0)
1: derived: Flask-Babel (0.11.2)
1: derived: boto (*)
1: derived: pika (>=1.1.0)
1: derived: ckanext-envvars (*)
1: derived: pyyaml (*)
1: derived: argparse (*)
1: derived: Routes (1.13)
1: derived: Shapely (>=1.2.13)
1: derived: repoze.who-friendlyform (1.0.8)
1: derived: python-dateutil (<2.0.0,>=1.5.0)
1: derived: bleach (~3.1.1)
1: derived: PasteScript (2.0.2)
1: derived: tzlocal (1.3)
1: derived: pytz (2016.7)
1: derived: lepl (*)
1: derived: ckanext-spatial (0.2 git)
1: derived: ply (3.4)
1: derived: ckanext-archiver (2.0.0 git)
1: derived: urllib3 (1.25.9)
1: derived: rfc3987 (*)
1: derived: cryptography (2.7)
1: derived: Genshi (0.7.3)
1: derived: SQLAlchemy (1.1.11)
1: derived: Paste (1.7.5.1)
1: derived: ckanext-harvest (1.3.2 git)
1: derived: ckanext-geodatagov (0.1 git)
1: derived: Jinja2 (~2.10.1)
1: derived: python-magic (0.4.15)
1: derived: Babel (2.3.4)
1: derived: werkzeug (~0.15.3)
1: derived: kombu (3.0.37)
1: derived: simplejson (3.10.0)
1: derived: WebTest (1.4.3)
1: derived: gunicorn (*)
1: derived: ckanext-qa (2.0 git)
1: derived: celery (3.1.25)
1: derived: click (6.7)
1: derived: google_compute_engine (2.8.13)
1: derived: redis (2.10.6)
1: derived: WebHelpers (1.3)
1: derived: sqlalchemy-migrate (0.10.0)
1: derived: polib (1.0.7)
1: derived: greenlet (0.4.12)
1: derived: passlib (1.7.3)
1: derived: ckanext-googleanalyticsbasic (0.1 git)
1: derived: rq (0.6.0)
1: derived: lxml (>=2.3)
1: derived: sqlparse (0.2.2)
1: derived: ckan (2.8.6 git)
1: derived: xlrd (>=1.0.0)
1: derived: pysolr (3.6.0)
1: derived: pyparsing (>=2.1.10)
1: derived: newrelic (*)
1: derived: GeoAlchemy (>=0.6)
1: derived: ckanext-report (0.1 git)
1: derived: gevent (*)
1: derived: messytables (>=0.15.2)
1: derived: fanstatic (0.12)
1: derived: pyutilib.component.core (4.6.4)
1: derived: cffi (1.12.3)
1: derived: OWSLib (0.8.6)
1: derived: Flask (0.12.4)
1: derived: psycopg2 (2.7.3.2)
1: derived: repoze.who (2.3)
1: derived: ckanext-datagovcatalog (0.0.1 git)
PyPI: 1 packages found for jsonschema 2.4.0
PyPI: No release information found for vdm-0.8, skipping
PyPI: 1 packages found for vdm 0.14
PyPI: No release information found for markdown-1.6, skipping
PyPI: 2 packages found for markdown >=3.1,<3.2
PyPI: No release information found for requests-0.12.01, skipping
PyPI: No release information found for requests-2.15.0, skipping
PyPI: No release information found for requests-0.0.1, skipping
PyPI: 2 packages found for requests >=2.20.0,<2.21.0
PyPI: No release information found for unicodecsv-0.9.0, skipping
PyPI: 9 packages found for unicodecsv >=0.9
PyPI: 1 packages found for pairtree 0.7.1-T
PyPI: 1 packages found for ofs 0.4.2
PyPI: No release information found for progressbar-2.3-dev, skipping
PyPI: 1 packages found for progressbar 2.3
PyPI: No release information found for zope.interface-3.0.0b1, skipping
PyPI: 1 packages found for zope.interface 4.3.2
PyPI: 1 packages found for geoalchemy2 0.5.0
PyPI: 1 packages found for ckantoolkit 0.0.3
PyPI: 1 packages found for pylons 0.9.7
PyPI: 1 packages found for webob 1.0.8
PyPI: No release information found for pyopenssl-0.11, skipping
PyPI: 1 packages found for pyopenssl 18.0.0
PyPI: 1 packages found for flask-babel 0.11.2
PyPI: 81 packages found for boto *
PyPI: 1 packages found for pika >=1.1.0
PyPI: 1 packages found for ckanext-envvars *
PyPI: No release information found for pyyaml-3.09, skipping
PyPI: No release information found for pyyaml-3.08, skipping
PyPI: No release information found for pyyaml-3.03, skipping
PyPI: No release information found for pyyaml-3.02, skipping
PyPI: No release information found for pyyaml-3.01, skipping
PyPI: No release information found for pyyaml-3.07, skipping
PyPI: No release information found for pyyaml-3.06, skipping
PyPI: No release information found for pyyaml-3.05, skipping
PyPI: No release information found for pyyaml-3.04, skipping
PyPI: 10 packages found for pyyaml *
PyPI: No release information found for argparse-0.0.1, skipping
PyPI: No release information found for argparse-1.2, skipping
PyPI: 17 packages found for argparse *
PyPI: No release information found for routes-2.4.0, skipping
PyPI: 1 packages found for routes 1.13
PyPI: 45 packages found for shapely >=1.2.13
PyPI: 1 packages found for repoze.who-friendlyform 1.0.8
PyPI: No release information found for python-dateutil-1.0, skipping
PyPI: No release information found for python-dateutil-1.1, skipping
PyPI: No release information found for python-dateutil-1.2, skipping
PyPI: No release information found for python-dateutil-0.1, skipping
PyPI: No release information found for python-dateutil-0.3, skipping
PyPI: No release information found for python-dateutil-0.5, skipping
PyPI: No release information found for python-dateutil-0.4, skipping
PyPI: No release information found for python-dateutil-2.0, skipping
PyPI: 1 packages found for python-dateutil >=1.5.0,<2.0.0
PyPI: No release information found for bleach-0.1, skipping
PyPI: No release information found for bleach-0.2, skipping
PyPI: No release information found for bleach-0.1.1, skipping
PyPI: No release information found for bleach-0.1.2, skipping
PyPI: 5 packages found for bleach >=3.1.1,<3.2.0
PyPI: 1 packages found for pastescript 2.0.2
PyPI: 1 packages found for tzlocal 1.3
PyPI: 1 packages found for pytz 2016.7
PyPI: 44 packages found for lepl *
PyPI: No release information found for ply-1.6, skipping
PyPI: No release information found for ply-1.8, skipping
PyPI: No release information found for ply-3.3, skipping
PyPI: No release information found for ply-3.1, skipping
PyPI: No release information found for ply-2.2, skipping
PyPI: No release information found for ply-2.1, skipping
PyPI: No release information found for ply-2.0, skipping
PyPI: No release information found for ply-2.5, skipping
PyPI: No release information found for ply-2.4, skipping
PyPI: 1 packages found for ply 3.4
PyPI: No release information found for urllib3-0.3, skipping
PyPI: No release information found for urllib3-0.3.1, skipping
PyPI: No release information found for urllib3-0.4.1, skipping
PyPI: No release information found for urllib3-0.4.0, skipping
PyPI: 1 packages found for urllib3 1.25.9
PyPI: 11 packages found for rfc3987 *
PyPI: 1 packages found for cryptography 2.7
PyPI: No release information found for genshi-0.5, skipping
PyPI: No release information found for genshi-0.3.2, skipping
PyPI: No release information found for genshi-0.3.3, skipping
PyPI: No release information found for genshi-0.3, skipping
PyPI: No release information found for genshi-0.3.1, skipping
PyPI: No release information found for genshi-0.3.6, skipping
PyPI: No release information found for genshi-0.4, skipping
PyPI: No release information found for genshi-0.3.4, skipping
PyPI: No release information found for genshi-0.3.5, skipping
PyPI: No release information found for genshi-0.4.1, skipping
PyPI: No release information found for genshi-0.5.1, skipping
PyPI: No release information found for genshi-0.4.3, skipping
PyPI: No release information found for genshi-0.4.2, skipping
PyPI: No release information found for genshi-0.4.4, skipping
PyPI: 1 packages found for genshi 0.7.3
PyPI: 1 packages found for sqlalchemy 1.1.11
PyPI: 1 packages found for paste 1.7.5.1
PyPI: 3 packages found for jinja2 >=2.10.1,<2.11.0
PyPI: No release information found for python-magic-0.4.1, skipping
PyPI: 1 packages found for python-magic 0.4.15
PyPI: No release information found for babel-0.9, skipping
PyPI: No release information found for babel-0.8, skipping
PyPI: No release information found for babel-0.9.1, skipping
PyPI: No release information found for babel-0.9.2, skipping
PyPI: No release information found for babel-0.9.3, skipping
PyPI: No release information found for babel-0.9.4, skipping
PyPI: No release information found for babel-0.9.5, skipping
PyPI: No release information found for babel-0.8.1, skipping
PyPI: 1 packages found for babel 2.3.4
PyPI: No release information found for werkzeug-0.10.3, skipping
PyPI: 4 packages found for werkzeug >=0.15.3,<0.16.0
PyPI: No release information found for kombu-3.0.17-20140602, skipping
PyPI: 1 packages found for kombu 3.0.37
PyPI: No release information found for simplejson-2.1.0rc3, skipping
PyPI: 1 packages found for simplejson 3.10.0
PyPI: 1 packages found for webtest 1.4.3
PyPI: No release information found for gunicorn-20.0.1, skipping
PyPI: 79 packages found for gunicorn *
PyPI: 1 packages found for celery 3.1.25
PyPI: 1 packages found for click 6.7
PyPI: 1 packages found for google-compute-engine 2.8.13
PyPI: 1 packages found for redis 2.10.6
PyPI: 1 packages found for webhelpers 1.3
PyPI: No release information found for sqlalchemy-migrate-0.1, skipping
PyPI: No release information found for sqlalchemy-migrate-0.2.2, skipping
PyPI: No release information found for sqlalchemy-migrate-0.2.1, skipping
PyPI: No release information found for sqlalchemy-migrate-0.2.0, skipping
PyPI: No release information found for sqlalchemy-migrate-0.4.0, skipping
PyPI: 1 packages found for sqlalchemy-migrate 0.10.0
PyPI: No release information found for polib-0.5.4, skipping
PyPI: No release information found for polib-0.4.2, skipping
PyPI: No release information found for polib-0.5.0, skipping
PyPI: No release information found for polib-0.4.1, skipping
PyPI: No release information found for polib-0.4.0, skipping
PyPI: 1 packages found for polib 1.0.7
PyPI: 1 packages found for greenlet 0.4.12
PyPI: 1 packages found for passlib 1.7.3
PyPI: 1 packages found for rq 0.6.0
PyPI: No release information found for lxml-2.3alpha2, skipping
PyPI: No release information found for lxml-2.3alpha1, skipping
PyPI: No release information found for lxml-1.3.1, skipping
PyPI: 67 packages found for lxml >=2.3
PyPI: 1 packages found for sqlparse 0.2.2
PyPI: 3 packages found for xlrd >=1.0.0
PyPI: 1 packages found for pysolr 3.6.0
PyPI: No release information found for pyparsing-1.1.2, skipping
PyPI: No release information found for pyparsing-1.2, skipping
PyPI: No release information found for pyparsing-1.3.3, skipping
PyPI: 14 packages found for pyparsing >=2.1.10
PyPI: No release information found for newrelic-0.5.52.109, skipping
PyPI: No release information found for newrelic-1.10.1.36, skipping
PyPI: No release information found for newrelic-0.5.48.104, skipping
PyPI: No release information found for newrelic-2.16.0.12, skipping
PyPI: No release information found for newrelic-1.9.0.21, skipping
PyPI: No release information found for newrelic-1.0.5.156, skipping
PyPI: No release information found for newrelic-1.10.2.38, skipping
PyPI: No release information found for newrelic-0.5.47.103, skipping
PyPI: No release information found for newrelic-0.5.58.122, skipping
PyPI: No release information found for newrelic-1.6.0.13, skipping
PyPI: No release information found for newrelic-1.11.0.55, skipping
PyPI: No release information found for newrelic-1.0.3.138, skipping
PyPI: No release information found for newrelic-1.1.0.192, skipping
PyPI: No release information found for newrelic-1.7.0.31, skipping
PyPI: No release information found for newrelic-1.10.0.28, skipping
PyPI: No release information found for newrelic-0.5.49.105, skipping
PyPI: No release information found for newrelic-1.4.0.137, skipping
PyPI: No release information found for newrelic-1.2.1.265, skipping
PyPI: No release information found for newrelic-1.3.0.289, skipping
PyPI: No release information found for newrelic-1.8.0.13, skipping
PyPI: No release information found for newrelic-1.2.0.246, skipping
PyPI: No release information found for newrelic-1.13.0.30, skipping
PyPI: No release information found for newrelic-1.12.0.56, skipping
PyPI: No release information found for newrelic-1.0.2.130, skipping
PyPI: No release information found for newrelic-0.5.50.107, skipping
PyPI: No release information found for newrelic-1.5.0.103, skipping
PyPI: No release information found for newrelic-2.86.3.69, skipping
PyPI: 112 packages found for newrelic *
PyPI: 4 packages found for geoalchemy >=0.6
PyPI: 46 packages found for gevent *
PyPI: 1 packages found for messytables >=0.15.2
PyPI: 1 packages found for fanstatic 0.12
PyPI: 1 packages found for pyutilib.component.core 4.6.4
PyPI: 1 packages found for cffi 1.12.3
PyPI: No release information found for owslib-0.10.2, skipping
PyPI: 1 packages found for owslib 0.8.6
PyPI: 1 packages found for flask 0.12.4
PyPI: No release information found for psycopg2-2.0.7, skipping
PyPI: No release information found for psycopg2-2.0.6, skipping
PyPI: No release information found for psycopg2-2.0.4, skipping
PyPI: No release information found for psycopg2-2.0.3, skipping
PyPI: No release information found for psycopg2-2.0.2, skipping
PyPI: No release information found for psycopg2-2.0.5.1, skipping
PyPI: No release information found for psycopg2-2.0.8, skipping
PyPI: 1 packages found for psycopg2 2.7.3.2
PyPI: 1 packages found for repoze.who 2.3
PyPI: Getting info for jsonschema (2.4.0) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: jsonschema-2.4.0-py2.py3-none-any.whl
1: selecting jsonschema (2.4.0)
PyPI: Getting info for vdm (0.14) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: vdm-0.14.tar.gz
1: selecting vdm (0.14)
1: selecting ckanext-datajson (0.1 2de3535)
1: selecting ckanext-datagovtheme (0.1 04d32b7)
PyPI: Getting info for pairtree (0.7.1-T) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: Pairtree-0.7.1-T.tar.gz
1: selecting pairtree (0.7.1-T)
PyPI: Getting info for ofs (0.4.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: ofs-0.4.2.tar.gz
1: fact: ofs (0.4.2) depends on argparse (*)
1: selecting ofs (0.4.2)
PyPI: Getting info for progressbar (2.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: progressbar-2.3.tar.gz
1: selecting progressbar (2.3)
PyPI: Getting info for zope.interface (4.3.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: zope.interface-4.3.2.tar.gz
1: selecting zope.interface (4.3.2)
PyPI: Getting info for geoalchemy2 (0.5.0) from PyPI
1: fact: geoalchemy2 (0.5.0) depends on SQLAlchemy (>=0.8)
1: selecting geoalchemy2 (0.5.0)
PyPI: Getting info for ckantoolkit (0.0.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: ckantoolkit-0.0.3.tar.gz
1: selecting ckantoolkit (0.0.3)
PyPI: Getting info for pylons (0.9.7) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: Pylons-0.9.7.tar.gz
1: fact: pylons (0.9.7) depends on Routes (>=1.10.3)
1: fact: pylons (0.9.7) depends on WebHelpers (>=0.6.4)
1: fact: pylons (0.9.7) depends on Beaker (>=1.2.2)
1: fact: pylons (0.9.7) depends on Paste (>=1.7.2)
1: fact: pylons (0.9.7) depends on PasteDeploy (>=1.3.3)
1: fact: pylons (0.9.7) depends on PasteScript (>=1.7.3)
1: fact: pylons (0.9.7) depends on FormEncode (>=1.2.1)
1: fact: pylons (0.9.7) depends on simplejson (>=2.0.8)
1: fact: pylons (0.9.7) depends on decorator (>=2.3.2)
1: fact: pylons (0.9.7) depends on nose (>=0.10.4)
1: fact: pylons (0.9.7) depends on Mako (>=0.2.4)
1: fact: pylons (0.9.7) depends on WebOb (>=0.9.6.1)
1: fact: pylons (0.9.7) depends on WebError (>=0.10.1)
1: fact: pylons (0.9.7) depends on WebTest (>=1.1)
1: fact: pylons (0.9.7) depends on Tempita (>=0.2)
1: selecting pylons (0.9.7)
1: derived: Tempita (>=0.2)
1: derived: WebError (>=0.10.1)
1: derived: Mako (>=0.2.4)
1: derived: nose (>=0.10.4)
1: derived: decorator (>=2.3.2)
1: derived: FormEncode (>=1.2.1)
1: derived: PasteDeploy (>=1.3.3)
1: derived: Beaker (>=1.2.2)
PyPI: 5 packages found for tempita >=0.2
PyPI: No release information found for weberror-0.8dev-20071109, skipping
PyPI: 7 packages found for weberror >=0.10.1
PyPI: 43 packages found for mako >=0.2.4
PyPI: No release information found for nose-1.3.5, skipping
PyPI: 13 packages found for nose >=0.10.4
PyPI: No release information found for decorator-4.0.8, skipping
PyPI: No release information found for decorator-3.4.1, skipping
PyPI: 25 packages found for decorator >=2.3.2
PyPI: No release information found for formencode-1.2.4dev, skipping
PyPI: 8 packages found for formencode >=1.2.1
PyPI: No release information found for pastedeploy-0.9.7dev-r5510, skipping
PyPI: 9 packages found for pastedeploy >=1.3.3
PyPI: No release information found for beaker-1.6.5, skipping
PyPI: 27 packages found for beaker >=1.2.2
PyPI: Getting info for webob (1.0.8) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: WebOb-1.0.8.zip
1: selecting webob (1.0.8)
PyPI: Getting info for pyopenssl (18.0.0) from PyPI
1: fact: pyopenssl (18.0.0) depends on cryptography (>=2.2.1)
1: fact: pyopenssl (18.0.0) depends on six (>=1.5.2)
1: selecting pyopenssl (18.0.0)
1: derived: six (>=1.5.2)
PyPI: 15 packages found for six >=1.5.2
PyPI: Getting info for flask-babel (0.11.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: Flask_Babel-0.11.2-py2.py3-none-any.whl
1: fact: flask-babel (0.11.2) depends on Flask (*)
1: fact: flask-babel (0.11.2) depends on Babel (>=2.3)
1: fact: flask-babel (0.11.2) depends on Jinja2 (>=2.5)
1: selecting flask-babel (0.11.2)
PyPI: Getting info for pika (1.1.0) from PyPI
1: selecting pika (1.1.0)
PyPI: Getting info for ckanext-envvars (0.0.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: ckanext-envvars-0.0.1.tar.gz
1: selecting ckanext-envvars (0.0.1)
PyPI: Getting info for routes (1.13) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: Routes-1.13.tar.gz
1: fact: routes (1.13) depends on repoze.lru (>=0.3)
1: selecting routes (1.13)
1: derived: repoze.lru (>=0.3)
PyPI: 5 packages found for repoze.lru >=0.3
PyPI: Getting info for repoze.who-friendlyform (1.0.8) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: repoze.who-friendlyform-1.0.8.tar.gz
1: fact: repoze.who-friendlyform (1.0.8) depends on repoze.who (>=1.0)
1: fact: repoze.who-friendlyform (1.0.8) depends on zope.interface (*)
1: fact: repoze.who-friendlyform (1.0.8) depends on WebOb (>=0.9.7)
1: selecting repoze.who-friendlyform (1.0.8)
PyPI: Getting info for python-dateutil (1.5) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: python-dateutil-1.5.tar.gz
1: selecting python-dateutil (1.5)
PyPI: Getting info for pastescript (2.0.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: PasteScript-2.0.2-py2.py3-none-any.whl
1: fact: pastescript (2.0.2) depends on Paste (>=1.3)
1: fact: pastescript (2.0.2) depends on PasteDeploy (*)
1: fact: pastescript (2.0.2) depends on six (*)
1: selecting pastescript (2.0.2)
PyPI: Getting info for tzlocal (1.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: tzlocal-1.3.tar.gz
1: fact: tzlocal (1.3) depends on pytz (*)
1: selecting tzlocal (1.3)
PyPI: Getting info for pytz (2016.7) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: pytz-2016.7-py2.py3-none-any.whl
1: selecting pytz (2016.7)
1: selecting ckanext-spatial (0.2 4ac25f1)
PyPI: Getting info for ply (3.4) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: ply-3.4.tar.gz
1: selecting ply (3.4)
1: fact: ckanext-archiver (2.0.0) depends on SQLAlchemy (>=0.6.6)
1: fact: ckanext-archiver (2.0.0) depends on requests (>=1.1.0)
1: fact: ckanext-archiver (2.0.0) depends on progressbar (*)
1: fact: ckanext-archiver (2.0.0) depends on ckanext-report (*)
1: selecting ckanext-archiver (2.0.0 4cb10ac)
PyPI: Getting info for urllib3 (1.25.9) from PyPI
1: selecting urllib3 (1.25.9)
PyPI: Getting info for cryptography (2.7) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: cryptography-2.7.tar.gz
1: fact: cryptography (2.7) depends on asn1crypto (>=0.21.0)
1: fact: cryptography (2.7) depends on six (>=1.4.1)
1: fact: cryptography (2.7) depends on cffi (>=1.8,<1.11.3 || >1.11.3)
1: fact: cryptography (2.7) depends on enum34 (*)
1: fact: cryptography (2.7) depends on ipaddress (*)
1: selecting cryptography (2.7)
1: derived: ipaddress (*)
1: derived: enum34 (*)
1: derived: asn1crypto (>=0.21.0)
PyPI: 22 packages found for ipaddress *
PyPI: 30 packages found for enum34 *
PyPI: 11 packages found for asn1crypto >=0.21.0
PyPI: Getting info for genshi (0.7.3) from PyPI
1: selecting genshi (0.7.3)
PyPI: Getting info for sqlalchemy (1.1.11) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: SQLAlchemy-1.1.11.tar.gz
1: selecting sqlalchemy (1.1.11)
PyPI: Getting info for paste (1.7.5.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: Paste-1.7.5.1.tar.gz
1: selecting paste (1.7.5.1)
1: selecting ckanext-harvest (1.3.2 8cde93c)
1: selecting ckanext-geodatagov (0.1 9df6669)
PyPI: Getting info for python-magic (0.4.15) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: python_magic-0.4.15-py2.py3-none-any.whl
1: selecting python-magic (0.4.15)
PyPI: Getting info for babel (2.3.4) from PyPI
1: fact: babel (2.3.4) depends on pytz (>=0a)
1: selecting babel (2.3.4)
PyPI: Getting info for kombu (3.0.37) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: kombu-3.0.37-py2.py3-none-any.whl
1: fact: kombu (3.0.37) depends on anyjson (>=0.3.3)
1: fact: kombu (3.0.37) depends on amqp (>=1.4.9,<2.0)
1: selecting kombu (3.0.37)
1: derived: amqp (>=1.4.9,<2.0)
1: derived: anyjson (>=0.3.3)
PyPI: No release information found for amqp-0.0.1, skipping
PyPI: 1 packages found for amqp >=1.4.9,<2.0
PyPI: 1 packages found for anyjson >=0.3.3
PyPI: Getting info for simplejson (3.10.0) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: simplejson-3.10.0.tar.gz
1: selecting simplejson (3.10.0)
PyPI: Getting info for webtest (1.4.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: WebTest-1.4.3.zip
1: fact: webtest (1.4.3) depends on WebOb (*)
1: selecting webtest (1.4.3)
1: fact: ckanext-qa (2.0) depends on ckanext-archiver (>=2.0)
1: fact: ckanext-qa (2.0) depends on ckanext-report (*)
1: fact: ckanext-qa (2.0) depends on SQLAlchemy (>=0.6.6)
1: fact: ckanext-qa (2.0) depends on requests (*)
1: fact: ckanext-qa (2.0) depends on xlrd (>=0.8.0)
1: fact: ckanext-qa (2.0) depends on messytables (>=0.8)
1: fact: ckanext-qa (2.0) depends on python-magic (>=0.4)
1: fact: ckanext-qa (2.0) depends on progressbar (*)
1: fact: ckanext-qa (2.0) depends on six (>=1.9)
1: selecting ckanext-qa (2.0 d7d384c)
1: derived: six (>=1.9)
PyPI: Getting info for celery (3.1.25) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: celery-3.1.25-py2.py3-none-any.whl
1: fact: celery (3.1.25) depends on billiard (>=3.3.0.23,<3.4)
1: fact: celery (3.1.25) depends on kombu (>=3.0.37,<3.1)
1: fact: celery (3.1.25) depends on pytz (>0.0-dev)
1: selecting celery (3.1.25)
1: derived: billiard (>=3.3.0.23,<3.4)
PyPI: 1 packages found for billiard >=3.3.0.23,<3.4
PyPI: Getting info for click (6.7) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: click-6.7-py2.py3-none-any.whl
1: selecting click (6.7)
PyPI: Getting info for google-compute-engine (2.8.13) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: google-compute-engine-2.8.13.tar.gz
1: fact: google-compute-engine (2.8.13) depends on boto (*)
1: fact: google-compute-engine (2.8.13) depends on distro (*)
1: selecting google-compute-engine (2.8.13)
1: derived: distro (*)
PyPI: 12 packages found for distro *
PyPI: Getting info for redis (2.10.6) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: redis-2.10.6-py2.py3-none-any.whl
1: selecting redis (2.10.6)
PyPI: Getting info for webhelpers (1.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: WebHelpers-1.3.tar.gz
1: fact: webhelpers (1.3) depends on MarkupSafe (>=0.9.2)
1: selecting webhelpers (1.3)
1: derived: MarkupSafe (>=0.9.2)
PyPI: 18 packages found for markupsafe >=0.9.2
PyPI: Getting info for sqlalchemy-migrate (0.10.0) from PyPI
1: fact: sqlalchemy-migrate (0.10.0) depends on pbr (>=1.3,<2.0)
1: fact: sqlalchemy-migrate (0.10.0) depends on SQLAlchemy (>=0.7.8,<0.9.5 || >0.9.5)
1: fact: sqlalchemy-migrate (0.10.0) depends on decorator (*)
1: fact: sqlalchemy-migrate (0.10.0) depends on six (>=1.7.0)
1: fact: sqlalchemy-migrate (0.10.0) depends on sqlparse (*)
1: fact: sqlalchemy-migrate (0.10.0) depends on Tempita (>=0.4)
1: selecting sqlalchemy-migrate (0.10.0)
1: derived: Tempita (>=0.4)
1: derived: pbr (>=1.3,<2.0)
PyPI: 10 packages found for pbr >=1.3,<2.0
PyPI: Getting info for polib (1.0.7) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: polib-1.0.7-py2.py3-none-any.whl
1: selecting polib (1.0.7)
PyPI: Getting info for greenlet (0.4.12) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: greenlet-0.4.12.tar.gz
1: selecting greenlet (0.4.12)
PyPI: Getting info for passlib (1.7.3) from PyPI
1: selecting passlib (1.7.3)
1: selecting ckanext-googleanalyticsbasic (0.1 54647da)
PyPI: Getting info for rq (0.6.0) from PyPI
1: fact: rq (0.6.0) depends on redis (>=2.7.0)
1: fact: rq (0.6.0) depends on click (>=3.0)
1: selecting rq (0.6.0)
PyPI: Getting info for sqlparse (0.2.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: sqlparse-0.2.2-py2.py3-none-any.whl
1: selecting sqlparse (0.2.2)
1: selecting ckan (2.8.6 eca78d5)
PyPI: Getting info for pysolr (3.6.0) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: pysolr-3.6.0-py2.py3-none-any.whl
1: fact: pysolr (3.6.0) depends on requests (>=2.9.1)
1: selecting pysolr (3.6.0)
1: selecting ckanext-report (0.1 b67875b)
PyPI: Getting info for messytables (0.15.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: messytables-0.15.2.tar.gz
1: fact: messytables (0.15.2) depends on xlrd (>=0.8.0)
1: fact: messytables (0.15.2) depends on python-magic (>=0.4.12)
1: fact: messytables (0.15.2) depends on chardet (>=2.3.0)
1: fact: messytables (0.15.2) depends on python-dateutil (>=1.5.0)
1: fact: messytables (0.15.2) depends on lxml (>=3.2)
1: fact: messytables (0.15.2) depends on requests (*)
1: fact: messytables (0.15.2) depends on html5lib (*)
1: fact: messytables (0.15.2) depends on json-table-schema (>=0.2,<=0.2.1)
1: selecting messytables (0.15.2)
1: derived: json-table-schema (>=0.2,<=0.2.1)
1: derived: html5lib (*)
1: derived: lxml (>=3.2)
1: derived: chardet (>=2.3.0)
PyPI: 2 packages found for json-table-schema >=0.2,<=0.2.1
PyPI: 18 packages found for html5lib *
PyPI: 6 packages found for chardet >=2.3.0
PyPI: Getting info for fanstatic (0.12) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: fanstatic-0.12.tar.gz
1: fact: fanstatic (0.12) depends on Paste (*)
1: fact: fanstatic (0.12) depends on WebOb (*)
1: selecting fanstatic (0.12)
PyPI: Getting info for pyutilib.component.core (4.6.4) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: pyutilib.component.core-4.6.4-py2-none-any.whl
1: fact: pyutilib.component.core (4.6.4) depends on six (*)
1: selecting pyutilib.component.core (4.6.4)
PyPI: Getting info for cffi (1.12.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: cffi-1.12.3.tar.gz
1: fact: cffi (1.12.3) depends on pycparser (*)
1: selecting cffi (1.12.3)
1: derived: pycparser (*)
PyPI: 20 packages found for pycparser *
PyPI: Getting info for owslib (0.8.6) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: OWSLib-0.8.6.tar.gz
1: fact: owslib (0.8.6) depends on python-dateutil (>=1.5)
1: fact: owslib (0.8.6) depends on pytz (*)
1: selecting owslib (0.8.6)
PyPI: Getting info for flask (0.12.4) from PyPI
1: fact: flask (0.12.4) depends on Werkzeug (>=0.7)
1: fact: flask (0.12.4) depends on Jinja2 (>=2.4)
1: fact: flask (0.12.4) depends on itsdangerous (>=0.21)
1: fact: flask (0.12.4) depends on click (>=2.0)
1: selecting flask (0.12.4)
1: derived: itsdangerous (>=0.21)
PyPI: 5 packages found for itsdangerous >=0.21
PyPI: Getting info for psycopg2 (2.7.3.2) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: psycopg2-2.7.3.2.tar.gz
1: selecting psycopg2 (2.7.3.2)
PyPI: Getting info for repoze.who (2.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: repoze.who-2.3-py2-none-any.whl
PyPI: Downloading wheel: repoze.who-2.3-py3-none-any.whl
1: fact: repoze.who (2.3) depends on WebOb (*)
1: fact: repoze.who (2.3) depends on zope.interface (*)
1: selecting repoze.who (2.3)
1: selecting ckanext-datagovcatalog (0.0.1 2c724d8)
PyPI: Getting info for amqp (1.4.9) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: amqp-1.4.9-py2.py3-none-any.whl
1: selecting amqp (1.4.9)
PyPI: Getting info for anyjson (0.3.3) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: anyjson-0.3.3.tar.gz
1: selecting anyjson (0.3.3)
PyPI: Getting info for billiard (3.3.0.23) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading sdist: billiard-3.3.0.23.tar.gz
1: selecting billiard (3.3.0.23)
PyPI: Getting info for markdown (3.1.1) from PyPI
1: selecting markdown (3.1.1)
PyPI: Getting info for requests (2.20.1) from PyPI
1: fact: requests (2.20.1) depends on chardet (>=3.0.2,<3.1.0)
1: fact: requests (2.20.1) depends on idna (>=2.5,<2.8)
1: fact: requests (2.20.1) depends on urllib3 (>=1.21.1,<1.25)
1: fact: requests (2.20.1) depends on certifi (>=2017.4.17)
1: derived: not requests (2.20.1)
PyPI: Getting info for requests (2.20.0) from PyPI
1: fact: requests (2.20.0) depends on chardet (>=3.0.2,<3.1.0)
1: fact: requests (2.20.0) depends on idna (>=2.5,<2.8)
1: fact: requests (2.20.0) depends on urllib3 (>=1.21.1,<1.25)
1: fact: requests (2.20.0) depends on certifi (>=2017.4.17)
1: derived: not requests (2.20.0)
1: fact: no versions of requests match >2.20.0,<2.20.1 || >2.20.1,<2.21.0
1: conflict: no versions of requests match >2.20.0,<2.20.1 || >2.20.1,<2.21.0
1: ! requests (>2.20.0,<2.20.1 || >2.20.1,<2.21.0) is partially satisfied by not requests (2.20.0)
1: ! which is caused by "requests (2.20.0) depends on urllib3 (>=1.21.1,<1.25)"
1: ! thus: requests (>=2.20.0,<2.20.1 || >2.20.1,<2.21.0) requires urllib3 (>=1.21.1,<1.25)
1: fact: requests (>=2.20.0,<2.20.1 || >2.20.1,<2.21.0) requires urllib3 (>=1.21.1,<1.25)
1: derived: not requests (>=2.20.0,<2.20.1 || >2.20.1,<2.21.0)
1: derived: certifi (>=2017.4.17)
1: conflict: requests (2.20.1) depends on urllib3 (>=1.21.1,<1.25)
1: ! requests (2.20.1) is partially satisfied by not requests (>=2.20.0,<2.20.1 || >2.20.1,<2.21.0)
1: ! which is caused by "requests (>=2.20.0,<2.20.1 || >2.20.1,<2.21.0) requires urllib3 (>=1.21.1,<1.25)"
1: ! thus: requests (>=2.20.0,<2.21.0) requires urllib3 (>=1.21.1,<1.25)
1: ! not urllib3 (>=1.21.1,<1.25) is satisfied by urllib3 (1.25.9)
1: ! which is caused by "ckan-catalog depends on urllib3 (1.25.9)"
1: ! thus: requests is forbidden
1: ! requests (>=2.20.0,<2.21.0) is satisfied by requests (~2.20.0)
1: ! which is caused by "ckan-catalog depends on requests (~2.20.0)"
1: ! thus: version solving failed
1: Version solving took 366.890 seconds.
1: Tried 1 solutions.
[SolverProblemError]
Because no versions of requests match >2.20.0,<2.20.1 || >2.20.1,<2.21.0
and requests (2.20.0) depends on urllib3 (>=1.21.1,<1.25), requests (>=2.20.0,<2.20.1 || >2.20.1,<2.21.0) requires urllib3 (>=1.21.1,<1.25).
And because requests (2.20.1) depends on urllib3 (>=1.21.1,<1.25), requests (>=2.20.0,<2.21.0) requires urllib3 (>=1.21.1,<1.25).
So, because ckan-catalog depends on both urllib3 (1.25.9) and requests (~2.20.0), version solving failed.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/clikit/console_application.py", line 131, in run
status_code = command.handle(parsed_args, io)
File "/usr/lib/python2.7/site-packages/clikit/api/command/command.py", line 120, in handle
status_code = self._do_handle(args, io)
File "/usr/lib/python2.7/site-packages/clikit/api/command/command.py", line 171, in _do_handle
return getattr(handler, handler_method)(args, io, self)
File "/usr/lib/python2.7/site-packages/cleo/commands/command.py", line 92, in wrap_handle
return self.handle()
File "/usr/lib/python2.7/site-packages/poetry/console/commands/lock.py", line 28, in handle
return installer.run()
File "/usr/lib/python2.7/site-packages/poetry/installation/installer.py", line 74, in run
self._do_install(local_repo)
File "/usr/lib/python2.7/site-packages/poetry/installation/installer.py", line 161, in _do_install
ops = solver.solve(use_latest=self._whitelist)
File "/usr/lib/python2.7/site-packages/poetry/puzzle/solver.py", line 36, in solve
packages, depths = self._solve(use_latest=use_latest)
File "/usr/lib/python2.7/site-packages/poetry/puzzle/solver.py", line 190, in _solve
raise SolverProblemError(e)
Running poetry export ...
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.
argparse 1.4.0 Python command-line parsing library
babel 2.3.4 Internationalization utilities
`-- pytz >=0a
bleach 3.1.5 An easy safelist-based HTML-sanitizing tool.
|-- packaging *
| |-- pyparsing >=2.0.2
| `-- six *
|-- six >=1.9.0
`-- webencodings *
boto 2.49.0 Amazon Web Services Library
celery 3.1.25 Distributed Task Queue
|-- billiard >=3.3.0.23,<3.4
|-- kombu >=3.0.37,<3.1
| |-- amqp >=1.4.9,<2.0
| `-- anyjson >=0.3.3
`-- pytz >0.0-dev
cffi 1.12.3 Foreign Function Interface for Python calling C code.
`-- pycparser *
ckan 2.8.6 CKAN Software
ckanext-archiver 2.0.0 Archives resources in CKAN (CKAN Extension)
|-- ckanext-report *
|-- progressbar *
|-- requests >=1.1.0
| |-- certifi >=2017.4.17
| |-- chardet >=3.0.2,<3.1.0
| |-- idna >=2.5,<2.8
| `-- urllib3 >=1.21.1,<1.25
`-- sqlalchemy >=0.6.6
ckanext-datagovcatalog 0.0.1 Catalog customizations
ckanext-datagovtheme 0.1 Datagov Theme
ckanext-datajson 0.1 CKAN extension to generate /data.json
ckanext-envvars 0.0.1 CKAN configuration settings available from env vars
ckanext-geodatagov 0.1
ckanext-googleanalyticsbasic 0.1 Basic extension to add google analytics tracking code in page header
ckanext-harvest 1.3.2 Harvesting interface plugin for CKAN
ckanext-qa 2.0 Quality Assurance plugin for CKAN
|-- ckanext-archiver >=2.0
| |-- ckanext-report *
| |-- progressbar *
| |-- requests >=1.1.0
| | |-- certifi >=2017.4.17
| | |-- chardet >=3.0.2,<3.1.0
| | |-- idna >=2.5,<2.8
| | `-- urllib3 >=1.21.1,<1.25
| `-- sqlalchemy >=0.6.6
|-- ckanext-report *
|-- messytables >=0.8
| |-- chardet >=2.3.0
| |-- html5lib *
| | |-- six >=1.9
| | `-- webencodings *
| |-- json-table-schema >=0.2,<=0.2.1
| |-- lxml >=3.2
| |-- python-dateutil >=1.5.0
| |-- python-magic >=0.4.12
| |-- requests *
| | |-- certifi >=2017.4.17
| | |-- chardet >=3.0.2,<3.1.0 (circular dependency aborted here)
| | |-- idna >=2.5,<2.8
| | `-- urllib3 >=1.21.1,<1.25
| `-- xlrd >=0.8.0
|-- progressbar *
|-- python-magic >=0.4
|-- requests *
| |-- certifi >=2017.4.17
| |-- chardet >=3.0.2,<3.1.0
| |-- idna >=2.5,<2.8
| `-- urllib3 >=1.21.1,<1.25
|-- six >=1.9
|-- sqlalchemy >=0.6.6
`-- xlrd >=0.8.0
ckanext-report 0.1 Framework for defining reports in CKAN
ckanext-spatial 0.2 Geo-related plugins for CKAN
ckantoolkit 0.0.3 UNKNOWN
click 6.7 A simple wrapper around optparse for powerful command line utilities.
cryptography 2.7 cryptography is a package which provides cryptographic recipes and primitives to Python developers.
|-- asn1crypto >=0.21.0
|-- cffi >=1.8,<1.11.3 || >1.11.3
| `-- pycparser *
|-- enum34 *
|-- ipaddress *
`-- six >=1.4.1
fanstatic 0.12 Flexible static resources for web applications.
|-- paste *
`-- webob *
flask 0.12.4 A microframework based on Werkzeug, Jinja2 and good intentions
|-- click >=2.0
|-- itsdangerous >=0.21
|-- jinja2 >=2.4
| `-- markupsafe >=0.23
`-- werkzeug >=0.7
flask-babel 0.11.2 Adds i18n/l10n support to Flask applications
|-- babel >=2.3
| `-- pytz >=0a
|-- flask *
| |-- click >=2.0
| |-- itsdangerous >=0.21
| |-- jinja2 >=2.4
| | `-- markupsafe >=0.23
| `-- werkzeug >=0.7
`-- jinja2 >=2.5
`-- markupsafe >=0.23
genshi 0.7.3 A toolkit for generation of output for the web
geoalchemy 0.7.2 Using SQLAlchemy with Spatial Databases
`-- sqlalchemy >=0.6.1
geoalchemy2 0.5.0 Using SQLAlchemy with Spatial Databases
`-- sqlalchemy >=0.8
gevent 1.2.2 Coroutine-based network library
`-- greenlet >=0.4.10
google-compute-engine 2.8.13 Google Compute Engine
|-- boto *
|-- distro *
`-- setuptools *
greenlet 0.4.12 Lightweight in-process concurrent programming
gunicorn 19.10.0 WSGI HTTP Server for UNIX
jinja2 2.10.3 A very fast and expressive template engine.
`-- markupsafe >=0.23
jsonschema 2.4.0 An implementation of JSON Schema validation for Python
kombu 3.0.37 Messaging library for Python
|-- amqp >=1.4.9,<2.0
`-- anyjson >=0.3.3
lepl 5.1.3 A Parser Library for Python 2.6+/3+: Recursive Descent; Full Backtracking
lxml 4.6.1 Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API.
markdown 3.1.1 Python implementation of Markdown.
`-- setuptools >=36
messytables 0.15.2 Parse messy tabular data in various formats
|-- chardet >=2.3.0
|-- html5lib *
| |-- six >=1.9
| `-- webencodings *
|-- json-table-schema >=0.2,<=0.2.1
|-- lxml >=3.2
|-- python-dateutil >=1.5.0
|-- python-magic >=0.4.12
|-- requests *
| |-- certifi >=2017.4.17
| |-- chardet >=3.0.2,<3.1.0
| |-- idna >=2.5,<2.8
| `-- urllib3 >=1.21.1,<1.25
`-- xlrd >=0.8.0
newrelic 5.20.1.150 New Relic Python Agent
ofs 0.4.2 OFS - provides plugin-orientated low-level blobstore.
`-- argparse *
owslib 0.8.6 OGC Web Service utility library
|-- python-dateutil >=1.5
`-- pytz *
pairtree 0.7.1-T Pairtree FS implementation.
passlib 1.6.5 comprehensive password hashing framework supporting over 30 schemes
paste 1.7.5.1 Tools for using a Web Server Gateway Interface stack
pastescript 2.0.2 A pluggable command-line frontend, including commands to setup package file layouts
|-- paste >=1.3
|-- pastedeploy *
| `-- setuptools *
`-- six *
pika 1.1.0 Pika Python AMQP Client Library
ply 3.4 Python Lex & Yacc
polib 1.0.7 A library to manipulate gettext files (po and mo files).
progressbar 2.3 Text progress bar library for Python.
psycopg2 2.7.3.2 psycopg2 - Python-PostgreSQL Database Adapter
pylons 0.9.7 Pylons Web Framework
|-- beaker >=1.2.2
| `-- funcsigs *
|-- decorator >=2.3.2
|-- formencode >=1.2.1
| `-- six *
|-- mako >=0.2.4
| `-- markupsafe >=0.9.2
|-- nose >=0.10.4
|-- paste >=1.7.2
|-- pastedeploy >=1.3.3
| `-- setuptools *
|-- pastescript >=1.7.3
| |-- paste >=1.3
| |-- pastedeploy *
| | `-- setuptools *
| `-- six *
|-- routes >=1.10.3
| `-- repoze.lru >=0.3
|-- simplejson >=2.0.8
|-- tempita >=0.2
|-- weberror >=0.10.1
| |-- paste >=1.7.1
| |-- pygments *
| |-- tempita *
| `-- webob *
|-- webhelpers >=0.6.4
| `-- markupsafe >=0.9.2
|-- webob >=0.9.6.1
`-- webtest >=1.1
`-- webob *
pyopenssl 18.0.0 Python wrapper module around the OpenSSL library
|-- cryptography >=2.2.1
| |-- asn1crypto >=0.21.0
| |-- cffi >=1.8,<1.11.3 || >1.11.3
| | `-- pycparser *
| |-- enum34 *
| |-- ipaddress *
| `-- six >=1.4.1
`-- six >=1.5.2
pyparsing 2.4.7 Python parsing module
pysolr 3.6.0 Lightweight python wrapper for Apache Solr.
`-- requests >=2.9.1
|-- certifi >=2017.4.17
|-- chardet >=3.0.2,<3.1.0
|-- idna >=2.5,<2.8
`-- urllib3 >=1.21.1,<1.25
python-dateutil 1.5 Extensions to the standard python 2.3+ datetime module
python-magic 0.4.15 File type identification using libmagic
pytz 2016.7 World timezone definitions, modern and historical
pyutilib.component.core 4.6.4 The PyUtilib Component Architecture.
`-- six *
pyyaml 5.3.1 YAML parser and emitter for Python
redis 2.10.6 Python client for Redis key-value store
repoze.who 2.3 repoze.who is an identification and authentication framework for WSGI.
|-- setuptools *
|-- webob *
`-- zope.interface *
`-- setuptools *
repoze.who-friendlyform 1.0.8 Collection of repoze.who friendly form plugins
|-- repoze.who >=1.0
| |-- setuptools *
| |-- webob *
| `-- zope.interface *
| `-- setuptools * (circular dependency aborted here)
|-- webob >=0.9.7
`-- zope.interface *
`-- setuptools *
requests 2.20.1 Python HTTP for Humans.
|-- certifi >=2017.4.17
|-- chardet >=3.0.2,<3.1.0
|-- idna >=2.5,<2.8
`-- urllib3 >=1.21.1,<1.25
rfc3987 1.3.8 Parsing and validation of URIs (RFC 3986) and IRIs (RFC 3987)
routes 1.13 Routing Recognition and Generation Tools
`-- repoze.lru >=0.3
rq 0.6.0 RQ is a simple, lightweight, library for creating background jobs, and processing them.
|-- click >=3.0
`-- redis >=2.7.0
shapely 1.7.1 Geometric objects, predicates, and operations
simplejson 3.10.0 Simple, fast, extensible JSON encoder/decoder for Python
sqlalchemy 1.1.11 Database Abstraction Library
sqlalchemy-migrate 0.10.0 Database schema migration for SQLAlchemy
|-- decorator *
|-- pbr >=1.3,<2.0
|-- six >=1.7.0
|-- sqlalchemy >=0.7.8,<0.9.5 || >0.9.5
|-- sqlparse *
`-- tempita >=0.4
sqlparse 0.2.2 Non-validating SQL parser
tzlocal 1.3 tzinfo object for the local timezone
`-- pytz *
unicodecsv 0.14.1 Python2's stdlib csv module is nice, but it doesn't support unicode. This module is a drop-in replacement which *does*.
urllib3 1.24.3 HTTP library with thread-safe connection pooling, file post, and more.
vdm 0.14 A versioned domain model framework.
webhelpers 1.3 Web Helpers
`-- markupsafe >=0.9.2
webob 1.0.8 WSGI request and response object
webtest 1.4.3 Helper to test WSGI applications
`-- webob *
werkzeug 0.15.6 The comprehensive WSGI web application library.
xlrd 1.2.0 Library for developers to extract data from Microsoft Excel (tm) spreadsheet files
zope.interface 4.3.2 Interfaces for Python
`-- setuptools *
cp requirements/requirements.txt ckan/requirements.txt
As a data.gov developer I want to test that the existing collections being harvested from a CSW harvest source are being harvested correctly
Acceptance Criteria
When I search for a parent dataset (need live example of collection from CSW harvest source)
Then I see it in search results
When I search for a child dataset (need live example of collection from CSW harvest source)
Then I do not see the child dataset in the search results, but I do see the parent dataset in the results (need live example of collection from CSW harvest source)
WHEN I click on a collection
Task-list
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: eea39e9
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3431319348
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: 1c9b292
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3526579166
Workflow with Issue: restart application
Job Failed: restart-prod
Last Commit: e9000b7
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3276645097
Workflow with Issue: publish & deploy
Job Failed: deploy-prod
Last Commit: f9ee3cf
Number of times run: 1
Last run by: robert-bryson
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3387956439
Workflow with Issue: publish & deploy
Job Failed: deploy-staging
Last Commit: 1023c85
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3274265068
Error
Viewing datasets of type "dataset" is not supported
('Template spatial/snippets/dataset_map_sidebar.html cannot be found').
How to reproduce
Add a WAF source from the URL http://meta.geo.census.gov/data/existing/decennial/GEO/CPMB/boundary/2016gz/kml_aiannh_500/
Add config:
{"validator_profiles": ["iso19139ngdc"], "private_datasets": false}
(you can also import WAF sources from the import script)
This source was harvested last time in 2016: validate we expected to work
Run the harvest process, you will get 2 datasets. You'll see the error trying to open any of these datasets.
#related to #307
Same error for New Mexico Resource Geographic Information System (NM RGIS) source
URL: http://rgismetadata.unm.edu/19115/collections/
Config
{"validator_profiles": ["iso19139ngdc"], "private_datasets": false}
We can get the datasets list but we can get inside
Related to #307
Error import WAF source:
Error parsing the configuration options: Unknown validation profile(s): fgdc_minimal
To reproduce
python import_harvest_sources.py \
--origin_url=https://catalog.data.gov \
--destination_url=http://ckan:5000 \
--destination_api_key=xxxxx \
--source_type=waf \
--limit=2
Error log.
Creating organization Federal Geographic Data Committee
Request OK http://ckan:5000/api/3/action/organization_create
Config found in extras: {'value': '{"validator_profiles": ["fgdc_minimal"], "private_datasets": false}', 'key': 'config'}
Creating harvest source FGDC WAF (Hosted by DOI for Geoplatform.gov)
http://data.doi.gov/WAF/FGDC
{"validator_profiles": ["fgdc_minimal"], "private_datasets": false}
ERROR status: 409
content:b'{"help": "http://ckan:5000/api/3/action/help_show?name=harvest_source_create", "success": false, "error": {"__type": "Validation Error", "config": ["Error parsing the configuration options: Unknown validation profile(s): fgdc_minimal"]}}'
Tasks
Error at DCAT-US harvest sources while we harvest them (report here).
ckan.logic.ValidationError:
{
'spatial': [
u'Error decoding JSON object: Extra data: line 1 column 7 - line 1 column 23 (char 6 - 22)'
]
}
This issue stems from #312
This source is harvested in production:
spatial
fieldThis error happens in the spatial extension, here.
The GSA fork also includes the same code and should also fail
The GeoDataGov ext avoid the error by doing a roll up extras before package_create
here so the spatial ext is not able to find the broken spatial extra.
if action_name in self.ROLLUP_SAVE_ACTIONS:
extras_rollup = {}
new_extras = []
for extra in data_dict.get('extras', []):
if extra['key'] in self.EXTRAS_ROLLUP_KEY_IGNORE:
new_extras.append(extra)
else:
extras_rollup[extra['key']] = extra['value']
if extras_rollup:
new_extras.append({'key': 'extras_rollup',
'value': json.dumps(extras_rollup)})
data_dict['extras'] = new_extras
This feature avoids the spatial
validation for the spatial extra.
At GSA CANK 2.3 we allow to override before_action
and after_action
functions to all extensions. This is not available at CKAN 2.8
The GeoDataGov extension uses this function to roll up extras.
Notes
package_update
and package_create
but the class IPackageController seems not include a before_create
or before_update
hooks.spatial
as a string or deal with this in a different way. Avoid a validation is not a real fix here.Workflow with Issue: publish & deploy
Job Failed: deploy-prod
Last Commit: 560586e
Number of times run: 1
Last run by: FuhuXia
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3576947744
Workflow with Issue: publish & deploy
Job Failed: deploy-staging
Last Commit: eea39e9
Number of times run: 2
Last run by: robert-bryson
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3422611674
Workflow with Issue: publish & deploy
Job Failed: deploy-staging
Last Commit: eea39e9
Number of times run: 1
Last run by: robert-bryson
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3422611674
Workflow with Issue: publish & deploy
Job Failed: deploy-prod
Last Commit: 852fed6
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3281817910
CVE-2019-1010083
low severity
Vulnerable versions: < 1.0.0
Patched version: 1.0.0
The Pallets Project Flask before 1.0 is affected by: unexpected memory usage. The impact is: denial of service. The attack vector is: crafted encoded JSON data. The fixed version is: 1. NOTE: this may overlap CVE-2018-1000656.
Note that CKAN 2.8.x does not appear to be compatible with Flask >= 1.0.0, although CKAN 2.9 is.
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: c68d3eb
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3269368438
The DataGovTheme depends on ckanext-archiver
and ckanext-qa
and uses its helpers:
archiver
qa
Since this new catalog move extension to the upstream version we miss these functions and the Theme is not able to show resources inside datasets.
We hide these calls in a PR.
This issue is to validate if we want these features and recover them.
Related issues:
Should be part of the analysis at:
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: 1c9b292
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3490674881
Implement similar enhancements to Inventory logging config (#474)
Workflow with Issue: restart application
Job Failed: restart-staging
Last Commit: 4178a23
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3308699566
If I try to open /dataset page I have this error:
IOError: [Errno 2] No such file or directory: '/var/tmp/ckan/logos/bureau.csv'
Module /srv/app/src/ckanext-datagovtheme/ckanext/datagovtheme/templates/snippets/facet_list.html:70 in top-level template code
{% set label = h.get_bureau_info(label)['title'] if h.get_bureau_info(label) else 'Code ' + label %}
Module ckanext.datagovtheme.helpers:493 in get_bureau_info
file_obj = open(filename)
Workflow with Issue: publish & deploy
Job Failed: deploy-prod
Last Commit: aad399d
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3321595798
Workflow with Issue: restart application
Job Failed: restart-prod
Last Commit: c1a48a9
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3368981918
This issue is blocking Sandblox-PR#1792.
We can't create waf-collection
harvest source in the new harvest source form
This is because we are not using the custom form to create this type of source.
The catalog classic use datagov_harvest
instead harvest
in the plugin list. This is defined at geodatagov extension. This class override the form to create harvest sources.
If we move plugin from harvest
to datagov_harvest
we still have an error
harvest
extension with datagov_harvest
in the plugins configuration (production.ini)datagov_harvest
and fix the error or move this new form (and the other features) to a different extension preserving the harvest
plugin in use. (discussion here)This new plugin was added for Z3950 harvester 7 years ago.
The order of plugins is different:
Using datagov_harvest
we see this error log:
Module ckan.logic.action.create:170 in package_create
check_data_dict None
context {'__auth_audit': [], '__auth_user_obj_checked': True, 'auth_user_obj': <User id=6fb43a58-a883-47c b-8e19-8503d7f32064 name=admin password=$pbkdf2-sha512$25000$WgvhHOOc897bO6dUau09xw$waMJ6mmbgBynQpo8aep6ijGRm8EqijCjlQ5aeMPqA5XfjxDKE.QSXO8czsiBHUnEisH8oxC8QutjPWgItmFsig fullname=None [email protected] apikey=f1bb38d5-8841-48cb-a414-cda128f811ab created=2020-06-23 15:39:44.994457 reset_key=None about=None activity_streams_email_notifications=False sysadmin=True state=active>, 'message': '', 'model': <module 'ckan.model' from '/srv/app/src/ckan/ckan/model/__init__.pyc'>, 'save': True, 'session': <sqlalchemy.orm.scoping.scoped_session object at 0x7f292908ef50>, 'user': u'admin'}
data_dict {'collection_metadata_url': u'https://meta.geo.census.gov/data/existing/dec ennial/GEO/GPMB/TIGERline/TIGER2018/SeriesInfo/SeriesCollection_tl_2018_concity.shp.iso.xml', 'database': u'', 'extra_search_criteria': u'', 'extras': [], 'frequency': u'MANUAL', 'name': u'gpb-5', 'notes': u'', 'owner_org': u'1c4b46eb-afe8-4d4c-a974-822f9d7df76a', 'port': u'', 'private_datasets': u'False', 'save': u'Save', 'source_type': u'waf-collection', 'title': u'GPB 5b', 'type': 'harvest', 'url': u'https://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2018/concity/', 'validator_profiles': u'', 'validator_schema': u''}
model <module 'ckan.model' from '/srv/app/src/ckan/ckan/model/__init__.pyc'>
package_plugin <Plugin DataGovHarvest 'datagov_harvest'>
schema {'__extras': [<function harvest_source_extra_validator at 0x7f29261ceed8>], 'config': [<fun ction ignore_missing at 0x7f29263740c8>, <function harvest_source_config_validator at 0x7f29261cede8>, <function convert_to_extras at 0x7f29262bcde8>], 'extras': {'__extras': [<function ignore at 0x7f292636cf50>], 'deleted': [<function ignore_missing at 0x7f29263740c8>], 'id': [<function ignore at 0x7f292636cf50>], 'key': [<function not_empty at 0x7f292636cd70>, <function extra_key_not_in_root_schema at 0x7f29274fe578>, <function unicode_safe at 0x7f29263742a8>], 'revision_timestamp': [<function ignore at 0x7f292636cf50>], 'state': [<function ignore at 0x7f292636cf50>], 'value': [<function not_missing at 0x7f292636ccf8>]}, 'frequency': [<function ignore_missing at 0x7f29263740c8>, <type 'unicode'>, <function harvest_source_frequency_exists at 0x7f29261d60c8>, <function convert_to_extras at 0x7f29262bcde8>], 'name': [<function not_empty at 0x7f292636cd70>, <type 'unicode'>, <function name_validator at 0x7f29274fd500>, <function package_name_validator at 0x7f29274fd578>], 'notes': [<function ignore_missing at 0x7f29263740c8>, <type 'unicode'>], 'organization': [<function ignore_missing at 0x7f29263740c8>], 'owner_org': [<function owner_org_validator at 0x7f29274eeb90>, <type 'unicode'>], 'private': [<function ignore_missing at 0x7f29263740c8>, <function boolean_validator at 0x7f29274eede8>, <function datasets_with_no_organization_cannot_be_private at 0x7f29274fe230>], 'save': [<function ignore at 0x7f292636cf50>], 'source_type': [<function not_empty at 0x7f292636cd70>, <type 'unicode'>, <function harvest_source_type_exists at 0x7f29261ced70>, <function convert_to_extras at 0x7f29262bcde8>], 'state': [<function ignore_missing at 0x7f29263740c8>], 'title': [<function callable at 0x7f292127e7d0>, <type 'unicode'>], 'type': [<function dataset_type_exists at 0x7f29261d6140>, <type 'unicode'>], 'url': [<function not_empty at 0x7f292636cd70>, <type 'unicode'>, <function harvest_source_url_validator at 0x7f29261cecf8>]}
user u'admin'
Module ckan.lib.navl.dictization_functions:452 in unflatten
convert_to_list []
current_pos [{'key': 'frequency', 'value': u'MANUAL'}]
data {
('__extras',): {
'collection_metadata_url': u'https://meta.geo.census.gov/data/e xisting/decennial/GEO/GPMB/TIGERline/TIGER2018/SeriesInfo/SeriesCollection_tl_2018_concity.shp.iso.xml',
'database': u'',
'extra_search_criteria': u'',
'extras': [],
'port': u'',
'private_datasets': u'False',
'validator_profiles': u'',
'validator_schema': u''
},
('collection_metadata_url',): u'https://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2018/SeriesInfo/SeriesCollection_tl_2018_concity.shp.iso.xml',
('config',): '{
"collection_metadata_url": "https://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2018/SeriesInfo/SeriesCollection_tl_2018_concity.shp.iso.xml",
"private_datasets": false
}',
('extras',): [
{'key': 'frequency', 'value': u'MANUAL'}
],
('extras', 0, 'key'): 'frequency', ('extras', 0, 'value'): u'MANUAL',
('extras', 1, 'key'): 'source_type', ('extras', 1, 'value'): u'waf-collection',
('frequency',): u'MANUAL',
('name',): u'gpb-5',
('notes',): u'',
('owner_org',): u'1c4b46eb-afe8-4d4c-a974-822f9d7df76a',
('private_datasets',): False,
('source_type',): u'waf-collection',
('title',): u'GPB 5b',
('type',): u'harvest',
('url',): u'https://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2018/concity/'
}
flattend_key ('extras', 1, 'key')
key 1
unflattened {'__extras': {'collection_metadata_url': u'https://meta.geo.census.gov/data/exis ting/decennial/GEO/GPMB/TIGERline/TIGER2018/SeriesInfo/SeriesCollection_tl_2018_concity.shp.iso.xml', 'database': u'', 'extra_search_criteria': u'', 'extras': [], 'port': u'', 'private_datasets': u'False', 'validator_profiles': u'', 'validator_schema': u''}, 'collection_metadata_url': u'https://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2018/SeriesInfo/SeriesCollection_tl_2018_concity.shp.iso.xml', 'config': '{"collection_metadata_url": "https://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2018/SeriesInfo/SeriesCollection_tl_2018_concity.shp.iso.xml", "private_datasets": false}', 'extras': [{'key': 'frequency', 'value': u'MANUAL'}], 'frequency': u'MANUAL', 'name': u'gpb-5', 'notes': u'', 'owner_org': u'1c4b46eb-afe8-4d4c-a974-822f9d7df76a', 'private_datasets': False, 'source_type': u'waf-collection', 'title': u'GPB 5b', 'type': u'harvest', 'url': u'https://meta.geo.census.gov/data/existing/decennial/GEO/GPMB/TIGERline/TIGER2018/concity/'}
view
>> current_pos = current_pos[key]
IndexError: list index out of range
When I search for a parent dataset (need live example of collection from WAF-collection type harvest source)
Then I see it in search results
When I search for a child dataset (need live example of collection from WAF-collection type harvest source)
Then I do not see the child dataset in the search results, but I do see the parent dataset in the results (need live example of collection from WAF-collection type harvest source)
WHEN I click on a collection
Task-list
Allow optional execution of saml2 extension locally
localhost
Document the steps to run the tests for any extension.
Related to #301
We need a way to full test extensions inside this new catalog app.
For instance, to test our new catalog extension we run:
docker-compose exec ckan bash -c \
"cd src_extensions/ckanext-catalogdatagov && nosetests -v -s --with-pylons=test.ini"
We detect errors trying to run these tests for ckanext-datajson and ckanext-harvest
In both cases, we mount a volume with full source and run tests
nosetests --ckan --with-pylons=test.ini ckanext/datajson/tests
The reset_db
function freezes the tests
fails at `test_datajsonharvester.py", line 42, in setup: reset_db()`
ERROR: ProgrammingError: (psycopg2.ProgrammingError)
relation "package_tag_revision" does not exist
LINE 1: delete from "package_tag_revision"
To test notification/email in harvest upstream ext inside this env: Fail when we try to use
nosetests --ckan --with-pylons=test.ini ckanext.harvest.tests.nose.test_action:TestHarvestErrorMail
Testing on inventory demonstrated that we want to use async workers with gunicorn.
As a data.gov developer I want to test that the existing collections being harvested from a WAF harvest source are being harvested correctly
Acceptance Criteria
When I search for a parent dataset (need live example of collection from WAF harvest source)
Then I see it in search results
When I search for a child dataset (need live example of collection from WAF harvest source)
Then I do not see the child dataset in the search results, but I do see the parent dataset in the results (need live example of collection from WAF harvest source)
WHEN I click on a collection
Task-list
Workflow with Issue: restart application
Job Failed: restart-prod
Last Commit: 783acbb
Number of times run: 2
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3585113234
Workflow with Issue: restart application
Job Failed: restart-prod
Last Commit: c325ebf
Number of times run: 1
Last run by: nickumia-reisys
Github Action Run: https://github.com/GSA/catalog.data.gov/actions/runs/3292182580
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.