openconceptlab / ocl_issues Goto Github PK

Feeds are intended to enable tracking of all updates to concepts, not just to indicate whether a concept has been updated. Therefore, atom feeds for sources and collections should include the full feed for each concept that they contain, not only the latest update. It appears that currently the atom feed for sources/collections includes only a single update (the most recent) for each, which is less like a feed and more just a list of the most current versions for each concept.

Based on REST API best practices, resources should contain URLs to all of their sub-resource endpoints. Sources are currently missing the mappings_url field. Verify that collections already have this attribute. For example:

"mappings_url": "/orgs/WHO/sources/ICD-10/mappings/"

As it stands now, every single update is committed to solr. Commits are quite slow in solr and it is better to make them in batches. We can make commits happen only every 15 seconds and soft-commits every second. In practice it means that it will take a second for an update to be visible in solr searchers and 15 seconds for the update to be persisted to disk. It will speed up response time of updates and batch operations such as deleting a source.

In the middle of a major import of CIEL v2015-08-24, users were trying to fetch an export for CIEL v2015-05-14. The first thing the API does when an export is requested is determine the last_child_update (see https://github.com/OpenConceptLab/oclapi/blob/master/django-nonrel/ocl/sources/models.py#L90).

However, when a concept version is updated, some of its metadata is modified before the new concept version is created. So even though users were requesting an older CIEL source version, every single time they requested the export, the last_child_update was different because the import process was modifying the old concept versions' metadata, which caused the export file to be re-created again and never returning an exportUrl to the user.

So basically, this means that it is not possible to retrieve exports that contain concept versions that are being updated during an import process. This is obviously not good!

Will need to brainstorm solutions, but here are some ideas:

• Speed up imports so that this is less of a problem! Currently it takes ~12 seconds during an import to update a single concept and ~6 seconds to add a new mapping.
• Default the export request to ALWAYS return an export file if it exists, even if its out-of-date, unless they include a request parameter -- could include an additional parameter that returns it unless they force date checking and if its not more than, say, 30 days old -- I like this approach the best because it requires fewer changes and gives control to the requesting client
• "Freeze" the old source version during the import process -- essentially, this means turn off checking of the last_child_update

Could use a text box with a very simple javascript date selector for this. Refer to the API documentation for the required format of the date when sent to the API.

When includeConcepts and includeMappings are used at the source or collection endpoints, paging is not currently supported. For example, this request returns the same concepts/mappings (the first page) regardless of the value of page.

GET http://api.dev.openconceptlab.com/orgs/CIEL/sources/CIEL/concepts/32/?includeMappings=true&includeInverseMappings=true&verbose=true&page=2

The page parameter must work, and the appropriate response headers (num_found, offset, num_returned, prev_url, next_url, etc.) should be returned.

When importing concepts, the locale_preferred is saved as false if using either true or false boolean values. However, it is saved as true if instead locale_preferred is set as a non-empty string that evaluates to true.

For example, I imported this:

{"datatype": "N/A", "concept_class": "Locale", "id": "abk", "names": [{"locale": "en", "locale_preferred": true, "name_type": "Fully Specified", "name": "Abkhazian"}, {"locale": "fr", "locale_preferred": true, "name_type": "Fully Specified", "name": "abkhaze"}, {"locale": "en", "locale_preferred": false, "name_type": "ISO 639-2", "name": "abk"}, {"locale": "en", "locale_preferred": false, "name_type": "ISO 639-1", "name": "ab"}]}

The import script debug output shows that locale_preferred is set correctly:

Adding new concept: {u'datatype': u'N/A', u'concept_class': u'Locale', u'id': u'ace', u'names': [{u'locale': u'en', u'locale_preferred': True, u'name_type': u'Fully Specified', u'name': u'Achinese'}, {u'locale': u'fr', u'locale_preferred': True, u'name_type': u'Fully Specified', u'name': u'aceh'}, {u'locale': u'en', u'locale_preferred': False, u'name_type': u'ISO 639-2', u'name': u'ace'}]}

But after creating the new concept, all locale_preferred fields are set to False. The same exact behavior occurs when doing an update.

However, if I instead change the true values to "locale_preferred":"True" (which evaluates to a boolean true), then it works fine. This is the short-term fix that I'm using so that I can continue importing content, but is still a bug that needs to be fixed.

Broken endpoints:
https://api.staging.openconceptlab.org/orgs/PEPFAR/collections/MER-T-Operating-Unit-Level-IM-FY17/v2017-10-02T101905.267394/export/

https://api.staging.openconceptlab.org/orgs/PEPFAR/sources/Mechanisms/v2018-01-13T004813.005885/export/

https://api.staging.openconceptlab.org/orgs/PEPFAR/collections/SIMS3-Facility/v2017-10-11T175154.910319/export/

Example use case is populating a drop down of locales that have a ISO 639-2 code.

https://app.openconceptlab.org/#/orgs/OCL/sources/Locales/concepts/zza/

There are 3 places where collection membership should be available:

• Retrieve list of collections that reference a resource:
  • This query will probably need a second optional parameter to specify how to handle resource versions -- i.e. above is the "head" of the concept, so should it return a match if a collection contains a specific version of this concept?
  • Example:

/orgs/PEPFAR/collections/?contains=/orgs/PEPFAR/sources/MER/concepts/HIV01-01/

[
    "/orgs/PEPFAR/collections/MyCollection/"
]

• Retrieve lists of collections that a concept is in along with the concept details:
  • This request will need a default scope (e.g. within the organization) to constrain which collections it returns -- in the future, if this is run on an ICD-10 code, it would return 1000s of collections! Meaning, that this should also expose an optional parameter to specify the scope of collections returned to point to another owner or namespace -- alternatively, we could only return collections within the current namespace for this request, and force a user to use the 1st query with the contains parameter to get collections from other namespaces
  • This requests needs a systematic way to handle (or ignore) concept versions as well

/orgs/PEPFAR/sources/MER/concepts/HIV01-01/[conceptVersion/]?verbose=true

{
    "collections": [
        "/orgs/PEPFAR/collections/My-Collection/"
    ],
    ...
}

• Retrieve lists of collections that a mapping is in along with the mapping details:
  • This is identical to the mappings one above

/orgs/PEPFAR/sources/MER/mappings/HIV01-01/[mappingVersion/]?verbose=true

@paynejd - This is just my opinion, but two things leap to mind for me:

• Does the information really belong in concept/mapping details? As you said, you could be returning a whole bunch of info that isn't, strictly speaking, the details of the resource. It seems more like extended information to me, that would be better suited to be returned by an additional call to something like
  /orgs/PEPFAR/sources/MER/concepts/HIV01-01/[conceptVersion/]collections

• Why would we need more than the first API call? Given completion of the first task to retrieve collections for a given resource, aren't the second and third items redundant?

Based on discussion today, let's focus in on just the collections endpoints:

• Retrieve list of collections for a single owner that contain a specific resource -- TOP PRIORITY -- this endpoint allows users to retrieve both public and private content (assuming that they have access to the private content)
  • contains parameter is a relative URL for a non-versioned or versioned concept or mapping. Example:

# Non-versioned Concept Relative URL:
?contains=/orgs/PEPFAR/sources/MER/concepts/HIV01-01/
# Versioned Mapping Relative URL:
?contains=/orgs/PEPFAR/sources/MER/mappings/09fj9j9fjf933/2/

* Examples:

/orgs/PEPFAR/collections/?contains=/orgs/PEPFAR/sources/MER/concepts/HIV01-01/
/users/JohnDoe/collections/?contains=/orgs/PEPFAR/sources/MER/concepts/HIV01-01/

• Retrieve list of collections globally -- MEDIUM PRIORITY --- note that the top-level endpoint only views public content
  • Example:

/collections/?contains=/orgs/PEPFAR/sources/MER/concepts/HIV01-01/

Finally, we still need to think about how to model a parameter that lets the client control how resource and repository versions are matched/returned. i.e. if contains is the "head" of the concept, should it return a match if a collection contains a specific version of this concept? Or vice-versa? What if an older version of a collection does contain the specified resource, but the HEAD of the collection does not?

With Rafal's help, I got the membership filter working on the collections view. However, I had to include a bit of a hack to get around some unexpected behavior applying an additional filter_backend. I'd like to meet to discuss this issue sometime soon so I can properly apply the membership filter without working around the filter_backend.

Pull request has been submitted.

Thank you @davetrig! Can you describe the exact behavior that was implemented? Was this implemented for the global collections endpoint as well as the orgs & users collections endpoint? How does it handle a versioned vs. non-versioned URI? How does it handle versions within the collections?

This is implemented for both the global and users/orgs endpoint (they both use the same view).
Each collection has a list of references to its contained resources, and those references contain the uri of the resource. So, if you specify a concept version, you will only get collections that contain that concept version. If you do not specify a version, you will only get collections that contain the unversioned concept.

It currently only handles the current state of collections. Collection versions are not searched.

I realized I had made a mistake in the first pull request. I think I should have only filtered on publicly viewable collections in the global search. I was applying that filter in both global and owner based searches. I submitted another pull request with that change.

Okay - I just tested on QA and appears to be working correctly. This approach will be sufficient for our demonstration today. I expect we will need to provide more flexibility in the handling of concept/mapping versions (i.e. through an additional parameter or two), but let's see how our conversation today goes with PEPFAR. Thanks!

This worked great during the presentation yesterday!

@davetrig Based on the implementation, can you propose the REST APIs that could be used to handle versions on the concept/mapping and/or collections side? These will be needed to fully support the use case we proposed walked through yesterday.

The concept details page provides a list mappings that are stored in the same source for which the fromConcept or toConcept matches the concept that is being viewed. This is the equivalent of setting verbose=true in the API request for the concept.

Ideally, we would like for users to be able to be able to view mappings that are contained in any source that have the same owner. For example, if PEPFAR has source A and source B, we want to optionally be able to see mappings that are stored within both source A and B when viewing a concept in either one.

In summary, the default behavior does not change, but we'd like to be able to set a URL parameter that expands the scope of the mappings that are returned to the owner (org or user) and not just the source. In the future, there may be other values supported, such as namespace or global. The default value is source.

I don't have a good name for the URL parameter yet -- any ideas? I wrote in mappingScope for now.

Also, I haven't thought through whether this affects collections or not...

Example:

/[ownerType]/[owner]/sources/[source]/concepts/[concept]/?mappingScope=org

This was already prototyped here: https://api.github.com/repos/OpenConceptLab/ocl_web/pulls/324

This includes:

• Dockerize OpenHIM with mediators
• Once system is running use admin account and add an account for self
• Apply configuration to new OpenHIM instance
• Document processes for (a) redeploying, (b) applying/modifying configuration (if not included in docker container), (c) redeploying updated sync/show scripts, and (d) redeploying updated mediator
• Point to sync mediator to DATIM server with production data (awaiting account from ICF)

Currently the bulk import functionality is only available on the server itself. Exposing through the API would be much better.

Currently downloads use the default sort of the export. Need to do some research to see how the records are sorted in the ZenDesk landing page downloads

This is especially pertinent for client applications that do not keep track of retired resources (such as OpenMRS for mappings). Note that separate cached exports would have to be created if supporting this.

At some point we decided not to implement the extras endpoint for orgs even though we use them. Instead extras are only an attribute of the organization document.

We want to implement extras for orgs the same way they are implemented for sources and concepts. For example:

• /orgs/:org/ -- extras returned as an attribute
• /orgs/:org/extras/
• /orgs/:org/extras/:extra-name/

Also, make sure that the same error doesn't occur for orgs that occurred for sources (see #133)

This was an oversight that it was not in the spec -- low priority, but documenting here

This is nearly done, but needs to be tested

HATEOAS links would be a much better way to handle all of the internal linking used in the API: https://spring.io/understanding/HATEOAS

For example, for a collection version it might look like this:

"_links": [
  { "rel": "self", "uri":"/orgs/OCL/collections/Community-MCH/1.5/" },
  { "rel": "collection", "uri": "/orgs/OCL/collections/Community-MCH/"},
  { "rel": "previous_version", "uri": "/orgs/OCL/collections/Community-MCH/1.4/"},
  { "rel": "root_version", "uri": "/orgs/OCL/collections/Community-MCH/1.0/"}
]

Note that this would require getting rid of the existing link properties.

Ideally, we would like more advanced functionality that would allow dictionary managers to view how much their dictionary is being accessed, but Google Analytics is a good start so that at least system admins can get at this info.

We want to support underscores in all resource IDs. Note that underscores are supported already in concept IDs.

It is currently not possible to name the inverse version of a mapping. For example, if a "parent of" mapping is created, it would be really helpful to be able to represent the inverse of that mapping as "child of"

Collections are performing badly when it comes to both reads and writes. It's unacceptable for the fetch of 25 items on a collection of 2000 items to take ~10 seconds. Imports are similarly slow. The issue also impacts solr indexing. Things get worse when the number of items grows.

Collections need to be remodeled similar as in https://github.com/OpenConceptLab/ocl_datim/issues/102. It's a bit more complex as we need to account for dynamic reference resolution of concepts and mappings belonging to a collection.

@paynejd, please confirm that I understand the logic correctly. When you create a new version of a collection and have references to concepts and not specific concept versions, then that collection version will be created using a specific concept version (latest at the time of version creation). In other words references will never be evaluated again for a collection version other than HEAD.

@rkorytkowski Sorry that I missed your question here! You are correct in your analysis -- references to HEAD are resolved to a specific concept/mapping version at the time of repo version creation.

There are apparently two copies of source extras, depending on how you access them. Need to verify that this isn't also happening for collections. Example:

/:ownerType/:owner/sources/:source/
/:ownerType/:owner/sources/:source/extras/

For example, if I do the following request:

POST /orgs/CIEL/sources/CIEL/extras/about/
{"about":"My about text goes here..."}

The above value will only be returned if I do a GET request on one of these:

GET /orgs/CIEL/sources/CIEL/extras/
GET /orgs/CIEL/sources/CIEL/extras/about/

However, if I do a GET request on the source, the about extra will not show up:

GET /orgs/CIEL/sources/CIEL/

Conversely, if I do the post on the source:

POST /orgs/CIEL/sources/CIEL/
{"extras":{"about":"My about text goes here..."}}

The new value will show up here:

GET /orgs/CIEL/sources/CIEL/

But the new value will not show up here:

GET /orgs/CIEL/sources/CIEL/extras/
GET /orgs/CIEL/sources/CIEL/extras/about/

Currently, DELETE calls only deactivate/retire a resource, effectively hiding it from view. Add a new "purge" parameter to all DELETE calls that actually causes a full DELETE of the resource (from both Mongo and Solr) when set to true.

Current limit is 100 for most downloads, but not clear when or where this limit is applied. We likely want to remove limit entirely on downloads for public repositories. Note that this is not a high priority because the clearinghouse IOL uses the export functionality, which does not have this limitation.

Extras for mappings has not been implemented.

Web creates its own user accounts that mirror the API user accounts, which creates the possibility of the accounts lists getting out of sync. We should eliminate the web side user accounts and expose any necessary functionality needed to allow for user account creation, management, and authentication using API only

There are API URL parameters that are activated simply if the parameter is included in the URL -- even if the parameter is set to false. For example, this request returns both concepts and mappings despite includeMappings and includeConcepts being set to false. All boolean URL parameters need to be tested and fixed.

GET /orgs/CIEL/sources/CIEL/?includeConcepts=false&includeMappings=false

There is currently no way to search across all repos in an org. This is especially problematic for private repositories, because then you can't use global search either, because global search only indexes public resources. For example, if a private org has several sources, a user could only search content by going to each of the sources individually.

Mappings can be queried in two ways at the concept level -- at the concept-id endpoint or the mappings endpoint:

GET http://api.dev.openconceptlab.com/orgs/CIEL/sources/CIEL/concepts/32/?includeMappings=true

GET http://api.dev.openconceptlab.com/orgs/CIEL/sources/CIEL/concepts/32/mappings/

Both of the above return 9 mappings. However, when using includeInverseMappings=true, a different number of inverse mappings are returned (2 and 1 additional inverse mappings, respectively):

GET http://api.dev.openconceptlab.com/orgs/CIEL/sources/CIEL/concepts/32/?includeMappings=true&includeInverseMappings=true

GET http://api.dev.openconceptlab.com/orgs/CIEL/sources/CIEL/concepts/32/mappings/?includeMappings=true&includeInverseMappings=true

We decided a long time ago that only mappings (direct or inverse) stored within the same source would be available from the concept or sub-concept endpoints, which means that both of these should be returning the same number of concepts. Currently, the concept-id endpoint is the one that is correct.

When using the /mappings/ endpoint, the response headers should be the same as with all other lists that are returned (e.g. should include offset, num_found, num_returned, etc.). It appears that paging was not implemented.

Multi-action imports occur when there is an update to a resource and simultaneously the resource's retired status is changed (because these are each updated as separate steps). Currently, if a concept import changes retired status and updates concept attributes (e.g. a concept is retired and its datatype is updated), then only the update is saved in the concept version history. In all other cases the concept version history is captured correctly (including if only the retired status changes).

Sources and collections should both allow you to see details of their concepts and mappings by using the includeConcepts and includeMappings parameters. For example, this works as expected at the sources endpoint:

https://api.staging.openconceptlab.org/orgs/MOH/sources/HMIS-Indicators/?includeMappings=true&includeConcepts=true

However, this is not working on the collections endpoint:

https://api.staging.openconceptlab.org/orgs/MOH/collections/HTS-TST-POSITIVE-F-15/?incluceMappings=true&includeConcepts=true

The above query is identical to what is contained in exports, so if this is not working it might also mean that the collections exports are not working -- please also check that.

Allow sign in using populare social media accounts

Ability to filter searches with a Custom Attribute value (e.g., find only concepts with a custom attribute of "Indicator Type" = "Result" or a collection with "QMAP-collection"=True). Custom attributes are stored in the extras field of a resource. Note that custom attributes may be of any type, eg. string, number, JSON, etc.

Eventually, we want to support filtering with many different operators (eg "equals", "does not equal", "contains", "ends with", "begins with", etc.), but minimum requirement is "equals". Operators not supported in the first round will be captured in a future ticket.

Also note that we will implement this feature for all resources that have custom attributes, but sources, collections and concepts are the top priority. If it makes sense to do this for all resources at the same time, then even better! Applicable resources include:

• orgs
• sources
• collections
• source versions
• collection versions
• concepts
• mappings

parentVersionUrl, previousVersionUrl and sourceUrl all need to be renamed to be underscore_spaced instead of camelCased. Change collections too if they are similar.

{
  'created_on': '2015-07-26T21:24:48.145',
  'description': 'Here is my description of this very exciting beta release that is just about nearly ready!!',
  'external_id': None,
  'extras': None,
  'id': 'v1.1.beta.2',
  'owner': 'CIEL',
  'owner_type': 'Organization',
  'owner_url': '/orgs/CIEL/',
  'parentVersionUrl': None,
  'previousVersionUrl': '/orgs/CIEL/sources/SNOMED-MVP/v1.1-beta/',
  'released': False,
  'sourceUrl': '/orgs/CIEL/sources/SNOMED-MVP/',
  'type': u'Source Version',
  'updated_on': '2015-07-26T21:24:49.421',
  'version_url': '/orgs/CIEL/sources/SNOMED-MVP/v1.1.beta.2/'
}

Note that this is lower priority...

Adding mappings/ to a concept URL returns a single concept's mappings that are stored in the same source, like this:

http://api.showcase.openconceptlab.org:8000/orgs/EthiopiaMoH-test-becfol/sources/HSTP-Indicators/concepts/C1.1.1.2/mappings/

However, this feature does not appear to have been implemented for concept versions:

http://api.showcase.openconceptlab.org:8000/orgs/EthiopiaMoH-test-becfol/sources/HSTP-Indicators/concepts/C1.1.1.2/5873f131aebccd0014c56e7e/mappings/

Primary concept name is populated automatically by the first concept name from when the concept was created and it is not editable. It was intended to be dynamically populated according to the user's preferred locale and the available concept synonyms. However, since we are not prioritizing localization for this development phase, we need to decide how to handle this in the short-term.

References in a collection often point to a specific version of a concept or mapping, which means that if the concept/mapping is modified at the source that the change will not be reflected in the collection. This behavior is by design, because collection curators will want to review and approve changes before adopting them in their collection. However, there is no method currently to review available updates and for a large collection finding and reviewing updates would be a significant burden.

An action button on the collection toolbar to identify, review and accept changes to concepts/mappings would solve this.

Note that a simple "diff" tool that compares and shows the difference between two resources (or two versions of a resource) would be an important component of this as well and will be described in a separate ticket.

This needs some design work and probably some help from @davetrig and @cmac35 to determine the best approach.

@rkorytkowski Can you confirm if a reference to a concept/mapping HEAD is automatically changed to the latest versioned concept/mapping? Or is that only happening upon repo version creation?

The issue we're trying to address here is that after resources are added to a collection, those resources can (and do sometimes) change at the source. How are those changes propagated to the collection? How does a user know that a new collection version needs to be created (e.g. to reflect those changes in an export)?

There are multiple parts to this (and not all will necessarily need to be implemented):

1. Method to determine if a reference points to the latest version of a resource
2. Method to update a reference to the latest version of a resource
3. Method to determine if updated resources are available for a collection
4. Method to update all references in a collection to the latest versions of the referenced resources

Related to this:

1. Method to determine if a repo version points to the latest versions of resources (if not, the user will know that some action needs to be taken to get the latest versions)

Note that versioned concepts/mappings should return an attribute that indicates whether it is the latest version of the resource. However, since mappings are stored separately from concepts, if a new mapping is added to a source, it would not be found just by looking for resources that have been updated, so there may need to be a way to find changes related mappings too.

The error appears in logs, but does not show in the UI.

api_1 | ERROR 2018/02/05 11:32:47 [base.py:210 handle_uncaught_exception()] Internal Server Error: /mappings/
api_1 | Traceback (most recent call last):
api_1 | File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 113, in get_response
api_1 | response = callback(request, *callback_args, **callback_kwargs)
api_1 | File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
api_1 | return self.dispatch(request, *args, **kwargs)
api_1 | File "/usr/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
api_1 | return view_func(*args, **kwargs)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 327, in dispatch
api_1 | response = self.handle_exception(exc)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 324, in dispatch
api_1 | response = handler(request, *args, **kwargs)
api_1 | File "/code/mappings/views.py", line 309, in get
api_1 | return self.list(request, *args, **kwargs)
api_1 | File "/code/oclapi/mixins.py", line 90, in list
api_1 | results = serializer.data
api_1 | File "/code/oclapi/serializers.py", line 17, in data
api_1 | self._populate_headers_and_data()
api_1 | File "/code/oclapi/serializers.py", line 32, in _populate_headers_and_data
api_1 | page_fields = self.to_native(obj)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/serializers.py", line 306, in to_native
api_1 | value = field.field_to_native(obj, field_name)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/serializers.py", line 357, in field_to_native
api_1 | return [self.to_native(item) for item in value]
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/serializers.py", line 306, in to_native
api_1 | value = field.field_to_native(obj, field_name)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/fields.py", line 967, in field_to_native
api_1 | value = getattr(self.parent, self.method_name)(obj)
api_1 | File "/code/mappings/serializers.py", line 184, in get_url
api_1 | return Mapping.objects.get(id=obj.versioned_object_id).url
api_1 | File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 143, in get
api_1 | return self.get_query_set().get(*args, **kwargs)
api_1 | File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 404, in get
api_1 | self.model._meta.object_name)
api_1 | DoesNotExist: Mapping matching query does not exist.
api_1 | ERROR 2018/02/05 11:32:47 [base.py:210 handle_uncaught_exception()] Internal Server Error: /mappings/
api_1 | Traceback (most recent call last):
api_1 | File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 113, in get_response
api_1 | response = callback(request, *callback_args, **callback_kwargs)
api_1 | File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
api_1 | return self.dispatch(request, *args, **kwargs)
api_1 | File "/usr/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 77, in wrapped_view
api_1 | return view_func(*args, **kwargs)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 327, in dispatch
api_1 | response = self.handle_exception(exc)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/views.py", line 324, in dispatch
api_1 | response = handler(request, *args, **kwargs)
api_1 | File "/code/mappings/views.py", line 309, in get
api_1 | return self.list(request, *args, **kwargs)
api_1 | File "/code/oclapi/mixins.py", line 90, in list
api_1 | results = serializer.data
api_1 | File "/code/oclapi/serializers.py", line 17, in data
api_1 | self._populate_headers_and_data()
api_1 | File "/code/oclapi/serializers.py", line 32, in _populate_headers_and_data
api_1 | page_fields = self.to_native(obj)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/serializers.py", line 306, in to_native
api_1 | value = field.field_to_native(obj, field_name)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/serializers.py", line 357, in field_to_native
api_1 | return [self.to_native(item) for item in value]
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/serializers.py", line 306, in to_native
api_1 | value = field.field_to_native(obj, field_name)
api_1 | File "/usr/local/lib/python2.7/site-packages/rest_framework/fields.py", line 967, in field_to_native
api_1 | value = getattr(self.parent, self.method_name)(obj)
api_1 | File "/code/mappings/serializers.py", line 184, in get_url
api_1 | return Mapping.objects.get(id=obj.versioned_object_id).url
api_1 | File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 143, in get
api_1 | return self.get_query_set().get(*args, **kwargs)
api_1 | File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 404, in get
api_1 | self.model._meta.object_name)
api_1 | DoesNotExist: Mapping matching query does not exist.

It's only on staging due to missing Mapping object for MappingVersion. Not sure when it happened. I'll add logging to identify which MappingVersion causes the issue.

Ethiopia requested the ability to backup and restore organizations (e.g. the entire Ethiopia NHDD) and repositories. This is an important risk mitigation feature for any country or organization, especially given that OCL is cloud-hosted. Currently you can export a specific repository version, but it is not in a format that could be imported without quite a few changes. We know that we will have to provide multiple options for exports and imports in the future, so we should start to think about what that would look like for an entire organization or an entire repository as well.

Currently only non-retired concepts/mappings are exported for full exports of sources and collections. It makes it hard for consumers to discover newly retired concepts/mappings and forces to omit mappings referring retired and non-retired concepts.

Exports should contain all concepts/mappings.

A simple semantic diff tool to compare 2 different resource versions (i.e. what changed between version 3 and version 4 of this concept or mapping?) or between 2 different resources (i.e. what is different between concept A and concept B?). Ideally this is built into the API, but a UI-only approach may also be appropriate.

Take a look at this json semantic diff tool for an example: jsondiff.com

The ability to compare 2 repo versions with each other

Ability to define a collection by an attribute or custom attribute value (e.g., create a collection of all MER Indicator concepts with the custom attribute "Indicator Type" = "Result). The dynamic references should be resolved to actual resources and frozen when a repo version is created. There are many other types of dynamic references that are possible, and this covers a single type.

A demo server would require dedicating additional infrastructure resources before there is a significant demand for this, so for the time being we will continue to use the QA server for this purpose. However, the QA server doesn't have enough content to make it very useful for training purposes. Need to prepare the import content and list of user accounts to automatically be generated on the QA server each time it is restored so that we can use it for training.

We had many requests in Ethiopia for a demo server with several pre-existing user accounts and test data that we could use for training purposes. The server should be reset every day or week. We used the QA and Staging servers for training this last week, which did the trick for now. However, requiring that 20 people create new user accounts was unnecessary, we didn't have useful test data loaded, the QA server is slow, and the staging server isn't really the right place to put practice content because we use it to show clients what the system and content will look like in production.

At some point, a demo/training server will be really helpful...