Code Monkey home page Code Monkey logo

admg-backend's Introduction

ADMG Project Repo

Welcome to the ADMG project repository. This repository contains the code for the ADMG web application, which is a Django web application that provides a user interface for the ADMG database. The application is built using the Django web framework and is designed to be deployed using Docker.

README Contents

  • Project Structure
  • Local Development
  • Additional Dev Tools
  • Automated Deployment
  • Configuring system to deploy CASEI

Project Structure

  • frontend: /app/admin_ui
  • backend: /app/admg_webapp

Local Development

Project Setup

  1. Install docker
  2. Copy .env.sample_local to .env
  3. Run docker compose
    docker compose up

If this is your first time setting up & running the application, you will also need to:

Run migrations to create the database schema:

docker compose run --rm web sh -c "python manage.py migrate"

Create a superuser to access the admin interface:

docker compose run --rm web sh -c "python manage.py createsuperuser"

Load a dump of the database to view the application with some data: Download the latest zip file of example data (or get this from one of the database maintainers) & load into the database. The following command will load the data into the database:

cat ./production_dump-2020.01.28.sql | psql admg_webapp

^ These commands should be run in a new terminal window, while the application is running.

Additional Dev Tools

Understanding python manage.py

python manage.py <command>

manage.py is your entry point into the django app. It has several commands, including:

  • test
  • migrate
  • makemigrations
  • runserver_plus
  • shell_plus
  • django extensions — third party modules

To run python manage.py commands using docker compose, use the following command structure:

docker compose run --rm -it web python manage.py <command>

Shell Access

Utilize Django's shell for experimentation with queries & other Django functionality:

docker compose run --rm -it web python manage.py shell_plus

Project URLs

List all of the URL patterns for the project:

docker compose run --rm it web python manage.py show_urls

Running Tests

Run your tests:

docker compose run --rm -it web pytest

Reporting test coverage

Run your tests with coverage:

docker compose run --rm -it web python -m coverage run -m pytest

Generate coverage report:

docker compose run --rm -it web python -m  coverage report -m --skip-covered

If you want to view coverage in your editor using, for example, VSCode's Coverage Gutters plugin, export the coverage report to a supported format:

docker compose run --rm -it web python -m coverage lcov -o coverage.lcov

Sass

To build Sass files for the project:

python manage.py sass admin_ui/static/scss admin_ui/static/css --watch

Automated deployment

Several automated workflows are already configured. These can be found within the .github/workflows directory.

Configuring system to deploy CASEI

The Maintenance Interface is able to initiate a deployment of CASEI. This works by triggering a workflow dispatch event on CASEI's deploy-to-production workflow. To allow the Maintenance Interface to trigger CASEI, a Personal Access Token with actions:write permissions should be provided via the CASEI_GH_TOKEN environment secret. The following environment variables may optionally be provided to override default configuration:

  • CASEI_GH_REPO, the repo to deploy. Defaults to NASA-IMPACT/admg-inventory
  • CASEI_GH_WORKFLOW_ID, the workflow to run. Defaults to deploy-to-production.yml
  • CASEI_GH_BRANCH, the branch to deploy. Defaults to production

admg-backend's People

Contributors

carsondavis avatar alukach avatar pkoiralap avatar rwegener2 avatar tammo-feldmann avatar johnhedman avatar code-geek avatar alicer avatar naomatheus avatar edkeeble avatar lint-action avatar alekhya-pottimuthi avatar svbagwell avatar ejwalker13 avatar xhagrg avatar heidimok avatar bitner avatar kamicut avatar emmalu avatar praveenphatate avatar smwingo avatar

Stargazers

Tobias Augspurger avatar BKN avatar

Watchers

Sajjad Anwar avatar  avatar  avatar Hanbyul Jo avatar Amy Lin avatar Karito Tenorio Palomino avatar Ge Peng avatar  avatar Lilly Thomas avatar  avatar Prakash Chaudhary avatar

admg-backend's Issues

Spatial bounds to deployment

WHY: These are on the deployment level. They are input on the whole for the campaign. Often, the deployment occurs over one region but another deployment would be a completely different location. If we combine those binding boxes, these could be really large areas of interest
It helps to compartmentalize the pieces of the campaign for users to see the different context pieces for the objectives for that deployment vs see specific areas

How is the MI passing spatial bounds to database? What happens when nothing is put into the MI in this field?

Technical Asks

  • Database adjustments to have fields entered at a different level
  • Changing where the field appears on the MI
  • Display changes that could affect the campaign page or deployment

EPIC: MI Workflow Enhancement - Release 1

Update Aug 25, 2022 with proposed stories and process documentation

Summary

Problem

  • The current maintenance interface design reflects how the data model works where everything is a record/draft.
  • For example, a create campaign draft or an update instrument draft.
  • As more drafts are added over time and update are created, the number of drafts that are part of the same “family” gets harder to keep track of because it’s all shown in one table.
  • While being able to reference past records can be nice, it’s not the same level of importance.

Goal

  • Improve curator workflow for finding, updating, and keeping track of published and edit drafts.
  • To a curator, we learned that the most important information is what is currently published, and whether or not there is an edit to that published version.
  • The scope is for campaigns, instruments, and platforms (not the limited fields, partner organization page) as they represent the main data categories.

Designs

Development

#437

User Requirements (not development stories)

The following stories will be used to track completion of the overall work. They are written from a user requirement standpoint, not technical steps. The actual development stories will be created during development and we will close out the user requirement stories as they are demoed in a staging environment.

Campaign Dashboard

Platform Dashboard

Instrument Dashboard

Other

Merge published tables and tables

There is a lot in common between published_tables.py and tables.py. They look like something that can be merged into a single structure. Potential solutions:

  1. Render columns using the config
  2. Abstract out the common elements in both tables and use it for the tables
  3. Combination of 1 and 2.

Numbers on summary page

Problem

Not all of the numbers are accurate

Explanation

  • The total value is set to none - was it meant to be manually provided? Can we do our own calculation on this?
  • The team is constantly adding C/P/I - we could submit those numbers and show the addition next to total
  • Could we click on a link that does what our sheet does for tracking this total calculation - this is a wish list item

Solution ideas

Display the # in review (this is currently inaccurate), # in admin, get rid of total . Awaiting review or in progress ?
Note: The numbers for each status should match the number of items we have

Design

ticket

No results:

Use autocomplete on Short names if possible
Clearly explain that there are no matching results
Offer a Path Forward

GCMD keywords auto update (SP: 1.5)

when a new version of GCMD keywords is released, we'd like the MI/CASEI to be consistent with that -- so the selectable GCMD keywords for fields that use them reflect the most current list of keywords

Note: Carson et al already working on this -- in parallel with effort for another impact project, i believe

Draft Campaigns are Missing Some Draft IOPs/Significant Events/CDPIs

image

MI is showing 5 deployments, which is correct, however it is missing several IOPs, Significant Events, and CDPIs which an analysis of the database showed to be correctly linked to the deployments.

image

Counts were made by gathering all 5 draft deployments linked to able's uuid, then finding all sig events, iops, and cdpis which were linked to one of those deployments.

Description text on Forms

Background

ADMG has requested some text changes in feedback, which are captured here.

Links

There’s a box for a short and long description on the campaign metadata form….These should be renamed to ‘draft description’ and ‘admin description’, while ensuring the admin description is the description that is displayed in the CASEI UI--currently it appears the draft description is being displayed

For deployments, need to ensure team names them with the campaign name at the beginning of deployment name (e.g., ARCTAS_dep_2009). Add needed guidance on structure “CampaignShortname_dep_yyyy(a or b if needed)”

Tables headers:

Add arrow icons to make it obvious that the header is clickable and make the whole area clickable (not just the link)

could this be 2 separate items?

  1. the arrow as an indicator
  2. clickable area
    idk if that makes one an easier lift to do?

Progressive Completion - Campaign level then Deployment level (SP:2)

Problem

After the curation staff enters a new campaign's campaign form info, the most likely next task is for that person to start entering deployment level info for that new campaign. Right now, we have to go back all the way to start and find the new campaign we just made and then add a new deployment.

Proposed Solution

Direct user to deployment creation page after completing the creation of a new campaign

Adding logos to campaigns

  • Getting error when add to published campaign
  • Lose access to logo form after upload -- error when validating campaign form once logo added
  • Do we need to publish logo separately, then re-publish campaign?

Can't update Logo on published Campaign

Imagine a scenario where a team member wants to add a new logo to a published Campaign. They would navigate to the the published Campaign, press “Create Edit Draft”...

image

click on “Add new Image”...
image

create a new Image draft, and then attempt to save their new draft of the Campaign...

image

However, this save fails because the draft does not pass validation...

image

We can see that the Published Edit Form runs validation on the models before creating a new Change model:

https://github.com/NASA-IMPACT/admg_webapp/blob/ee2d83f18197981a159a14d6c77bf6c6ef109609/admin_ui/published_views.py#L132-L149

However, I believe we should allow users to create Change records even if they don't pass validation. Instead, validation should only be applied when a user attempts to publish a Change record.

Alias Model Name API

Presently, when adding an alias through the APIs, it is necessary to give the integer primary key of the content_type. This is not a robust method, as the user should not be expected to know this integer, and it would be subject to change if the inventory was ever redeployed and ingested.

The preferred solution is that the user gives the string value of the short name of the linked model instead of the content_type. This functionality previously existed, but was disabled by an update to the maintenance interface. When addressing this issue, care should be taken to confirm that the MI is not impacted.

Change form does not support M2M fields

Currently, we customize our Change forms to support rendering FKs that reference other Change instances rather than published records:

https://github.com/NASA-IMPACT/admg_webapp/blob/ebfe3a2798ef9f1e220e683063a4f4f341a04d44/admin_ui/mixins.py#L16-L37

However, we don't offer this support for M2M fields (i.e. rendering Change instances rather than published records generating form options). This is problematic, particulalry when a user is working on a draft of a DOI:

https://github.com/NASA-IMPACT/admg_webapp/blob/ebfe3a2798ef9f1e220e683063a4f4f341a04d44/data_models/models.py#L1022-L1040

Obtain CollectionPeriod identifier's campaign name from join rather than 'update' prop

In #124, we introduced logic that generates a selection widget of CollectionPeriod drafts. To render the string representation as we do with real CollectionPeriod instances, we must join the records to a number of models. Obtaining the campaign name is particularly challenging, because we need to join across an intermediary models (i.e. CollectionPeriod -> Deployment -> Campaign). We avoided doing this by joining on the campaign value that happened to be in the CollectionPeriod dafts' update property:

https://github.com/NASA-IMPACT/admg_webapp/blob/7546c1aba2a091b4562023fef78954b979c8fae2/admin_ui/fields.py#L104-L113

However, this is not dependable. As @CarsonDavis wrote:

I think the inclusion of campaign is just a fluke of how the ingest works. They record it on the sheets, so the ingest brings it in. But since it isn't a real field, when stuff is published it gets ignored. You won't be able to count on having this field with the items created inside the MI.

As such, we should update the logic to properly join through the Deployment model to obtain the campaign name.


Note: It should be said that this was not initially done due to the fact this was difficult and holding up #124. I believe it is achievable but this is not a beginner task.

Layout Changes (SP: 2)

  • remove “Collection Period” phrase from the MI
  • Auto-adjust sizing/spacing on DOI page
  • view where user left off

Add indexes to Change.update properties

To make lookups of Change records performant, we should add indexes to the following properties of the update column on the Change table:

Common Lookups

  • campaign
  • deployment

Unsure of...

  • short_name - Used for filtering Changes by their short name

I'll be using this ticket to keep notes on which indexes to add. At this time, it is not to be considered as comprehensive.

Logo Forms Will Not Publish

Describe the bug
Logo forms still will not publish even when appropriate forms are filled out. When logo form is saved, image appears above the "Choose File" box, but it still says "No file chosen".

To Reproduce
Steps to reproduce the behavior:

  1. In the staging MI, add a new logo via form
  2. Fill out required elements
  3. Save the form
  4. Choose "Publish to Production" and see error

Expected behavior
When all the required fields are present, the form should publish. In addition, when the form is saved. we would expect to see the image name by the "Choose File" box.

Screenshots
MI_Logo_Form
Logo form -- see "No file chosen" under image

MI_Logo_Publication_Error
Error that displays when trying to publish logo

Desktop (please complete the following information):

  • OS: Windows
  • Browser: Chrome
  • Version: 97.0.4692.71 (Official Build) (64-bit)

Additional context
I am assigningAnthony to this issue, but please add anyone else who needs to be included.

Link to logo form in screenshots: https://admgstaging.nasa-impact.net/drafts/edit/333c3988-e993-448b-a62e-a6e90f755580

Table pagination - Go to Page element

Include ‘Go to Page’ element in the pagination so users can directly enter or select a page number and go directly to that page for viewing records.

Add Logos (SP: 1)

Right now there is a button to add a logo, but nothing happens/no way to actually add an image to upload.

Clean up links

Goal

Some of the redirects are not very intuitive. We would like to establish a clean and standard way of move between pages across the interface.

Details

Edit forms

Desired action & response: Save button should not redirect. Just keep the user on the same page. already complete
Desired action & response: Back button from any form should take you to the relevant table page (Ex. http://localhost:8000/organizations from a partner org form) EXCEPT for campaign items (campaign, deployment, se, iop, cdpi). These should take you back to the campaign details page Ex. http://localhost:8000/drafts/145c05dc-bb72-47af-933a-4b94f3169bb9

Notes

If anyone disagrees that this is the right flow I'd love to hear. @tania-pires in particular if you have opinions please jump in. If you want to revist the Miro board we could do that, if it is easier than a PR conversation.

Better widget for many to many selections

Background/problem

The default django widget for a user to select multiple values out of a pre-defined list has a few problems:

  • the user has to know to use the shift key
  • it is hard to see which fields have been selected if the list has more than a few options

Example

Proposed solution

In the django admin you can use a widget like this: https://i.stack.imgur.com/mJL3H.png
It seems worth investigating if we could use this same widget for the many to many relationships in our forms.

Custom widget to support uploading images and preview of images

Background and Goal

For each Campaign the inventory team would like to upload an image from their local computer to be used a the campaign logo for that campaign. This same feature is required for Platforms and Instruments.
Fields:

(These links should work when using the deployed version of the interface)

Currently these fields exist as dropdowns with no options.

Tech

The proposed solution is to develop a custom widget which supports the upload images and preview of images. There is likely a third-party plugin (or out of the box widget) to add this support.

We might have to coordinate with the backend team to make sure that the data model supports images/logos with the right type of field.

Website Types Clarification Info

Is your feature request related to a problem? Please describe.
Info below will be helpful to ADMG Staff when entering Websites info in the CASEI MI.

Describe the solution you'd like
The "Website Type" should be viewable through/on the CASEI UI site (and note that "website type" could be considered a "short name").
The "Website Type Description" should be visible only to curators/ADMG Staff within the MI to clarify what URLs are appropriate for each type.

(1) Website Type: Repository
Long Name: Repository/DAAC Website
Website Type Description: URL for Campaign Landing Page at the Assigned DAAC (if such a page exists)

(2) Website Type: Publication
Long Name: Overview Publication
Website Type Description: Most Recent, Fully Descriptive Peer-Reviewed Publication on the Campaign

(3) Website Type: Project
Long Name: Project Website
Website Type Description: Website built by Campaign Science Team ahead of/during the campaign. These sites may remain active for a long time following the Campaign.

(4) Website Type: Field Catalog
Long Name: Campaign Field/Event Catalog
Website Type Description: Listing of details related to each day or IOP during the campaign. May include raw data or images from the field, brief mission/science reports/drafts, data access links, and/or significant event overviews.

(5) Website Type: Additional
Long Name: Additional Campaign Resource
Website Type Description: Any other URL or DOI landing page with pertinent info for this campaign (e.g.: additional publications, partner institution sites describing the campaign, etc).

Describe alternatives you've considered
n/a

Additional context
This is intended for the MI and UI to help with organizing/grouping website URLs that are entered by the ADMG Staff.

Clean Up Sidebar UI

Context

This issue was repurposed to be part of EPIC: MI Workflow Redesign #270

Currently, the sidebar on drafts can be confusing to navigate and be somewhat messy. How can we improve this experience?

Current

image
https://admgstaging.nasa-impact.net/drafts/edit/a5ca9504-6f2c-4d65-8cf0-f0d5cfb6f31d

  • Example of an unpublished draft where the side bar is on the right-side

image
https://admgstaging.nasa-impact.net/drafts/edit/092e31e7-1227-47f3-af14-6c80ed178d96

  • Example of a published draft with the diff view where the side bar is at the bottom

New

image
https://www.figma.com/file/zY7BS9fR1huZ0CieHf3Dm8/MI-Workflow-Change?node-id=286%3A25633

image
https://www.figma.com/file/zY7BS9fR1huZ0CieHf3Dm8/MI-Workflow-Change?node-id=286%3A26852

  • Collapse information into an accordion
  • Only allow one accordion to open at a time (not like the mockup)
  • For the Dependent Drafts, created 1 indented section for each of:
    • Deployments
    • IOPs
    • Significant Events
    • CDPIs
  • Only the logo is not collapsed because it's a nice visual representation of the campaign, platform, or instrument

image
https://www.figma.com/file/zY7BS9fR1huZ0CieHf3Dm8/MI-Workflow-Change?node-id=286%3A25716

  • For the diff view, we can keep the current behaviour where the sidebar stacks at the bottom

image
https://www.figma.com/file/zY7BS9fR1huZ0CieHf3Dm8/MI-Workflow-Change?node-id=286%3A26004

  • For a published view, we can still show the logo and history

DOI Errors: Newly generated DOIs with both "keep" statuses displayed

Description: "Hadn’t worked on AfriSAR. got to DOI tab, ran “generate”. got list. why are both + & trash buttons already colored in."
The AfriSAR eg:  https://admg.nasa-impact.net/campaigns/dd804086-5fe2-494f-aac3-e63b5ff342ce/doi-approval

Screenshots, Example 1 AfriSAR
"This is my view for AfriSAR immediately after go to DOI tab today (after had "generated" yesterday, go same views). I haven't done anything on these, but the check/trash bins are all already colored in"
Capture_afriSAR_01
Capture_afriSAR_02

Deploy button fix (SP: .5)

@CarsonDavis reports:

Hey, I'm trying the deploy button on the Maintenance Interface, and I'm getting the following error

Failed to trigger deployment: {"message":"Bad credentials","documentation_url":"https://docs.github.com/rest"}

Add Websites (SP: 1)

Problem

Website is very unclear and since even I'm not sure how it actually works, we've instructed staff to not do them. perhaps can be mildly better with training, but the intuitiveness needs to be improved

Process

Review process for adding websites

  • map userflow
  • list any bugs or UI blockers
  • updated this ticket with analysis
  • identify key problems and propose solutions

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.