seed-platform / seed Goto Github PK

Sorting columns in Matching

When in the Matching screen, user should be able to sort on any of the columns, including the CONFIDENCE column, so that it is possible to see the lowest confidence matches together, which would make it easier to unmatch records by hand (by unchecking them)

Sometimes the program reports the number of matched records incorrectly in the popup windows that appears after matching. But the actual matches that were made are correct. Doesn’t always happen. May be hard to reproduce.

Right now only a few fields (the 30+ non-BEDES “hardcoded fields”) are the only fields in the building database record. The rest are stored in a JSON blob. Is this the right approach? Should those fields actually be put into the database? There are currently issues with filtering on the JSON blob fields, but that may be possible to solve with the current structure.

Field order of imported data in Mapping

The default order of the fields in the Mapping Screen should be the order of the fields in the original dataset. Right now it is not clear what order the fields are in -- it is not the order of the original data file, and it is not in alphabetical.

Add ability to edit mapping after it has been saved.

Once you have mapped your data, you can’t go back and change a mapped field if you made a mistake. You have to delete the data, reimport it, and remap it. Need to be able to change the mapping -- this will have cascading effects on all the child/associated data.

Allow user to clear type ahead list in Mapping
Need a way to clear out the mapping field list (possibly an admin function). The program (from the initial source code) create the initial field mapping list from the BEDES list (bedes.py) + hardcoded fields from (seed_mappings.py) . Then as users create their own fields (user defined fields), those get added to the field list. And on a field by field basis, the program should allow those user defined fields to be mapped to another field name, either existing or new, or deleted.
(LBNL Ticket # 274)

Understand how matching works with a dataset and across datasets. It is slightly different, some matching seems to only happen within datasets.

There are a number of existing issues that were identified by LBNL during acceptance testing of the SEED software. These need to be screened, cleaned up, and entered as new issues in the GitHub tracker. The plan is for this issue tracker to be the one place for the SEED community to log and track issues with the SEED software.

Have a central place to enter an organization name, main contact, and logo that will be used throughout a SEED installation, e.g., in the page footer of the GUI, in the email subject line of account invitation emails, etc. This can initially just be done through a text config file and maybe a directory to put static content.

User should be able to control where the export files are saved, i.e., a browse option to select the folder. Right now the files are always saved in the Downloads folder. Should also allow overwriting of existing file, with the standard warning.

Need an admin function (for Superusers) to delete Parent and Sub orgs. And possibly that same functionality for the owner of a Parent Org to delete Sub-Orgs. Right now it is not possible to delete either, even as a superuser. This needs some thought, because you don't want orphaned building records, so probably the records associated with those orgs would be deleted. Or possibly they could be move to another org before deleting their own org. Needs design before implementing.

Allow user to change % confidence threshold in Matching
Add an admin function that allows tweaking of the % confidence setting which determines whether or not a record matches. This helps reduce the number of false positive matches.

• Data Mapping screen -- highlight 4 (harcoded) matching fields
  (until we have feature allowing user to select fields to match)
• Data Mapping screen -- better description of valid, semivalid and invalid indicators

Refresh navigation results after mapping, matching, etc. to show updated number of buildings, etc.

It is confusing that every time an email address is added to a new org, the user receives a new invitation each time, even if that email address already has a log-in. It should be that SEED recognizes when an email address is already associated with an account, there is no need to create a new password.

Use needs to be able to see quickly which fields are from the official BEDES list and which are new user defined fields. This applies to many parts of the program -- the type ahead list in mapping, the building Detail view, etc.

For example, in the "type ahead list" in the Mapping screen:
Currently, the type ahead list in the Mapping screen does not have any indication of which fields are BEDES and which are not. Add some sort of indicator (asterisk for example?) in the list to show which are BEDES and which are user defined.

When you have two data sets and some records match and some don't, if you look at the Building Detail view of a record without a match, you only see the master record, you don't see a 2nd column with the name of the file that master record data came from. Seems like you should.

As a parent org, I can see all the sub orgs and their owners, but can’t see the members and viewers -- it would be nice to see those also; Also, add the # of buildings for each sub org and owner / member (which now shows up under organizations). So a complete view of all the parent/sub-org information in one place.

Program should also be able to generate a report (CSV file?) listing all this information.

Currently, the program will import Green Button data, and allow the user to associate a master record with it through hand matching. However, there is no indication in the Building Detail view of that master record that the Green Button data is associated with it. It would be nice to actually see the interval data, but the first step is just to indicate that there is Green Button for that building record. The name of the Green Button data file would be a good start.

Allow user to specify what fields the program should mach on.
Somewhere in the UI, between mapping and matching (before the matching algorithms are triggered), user should be able to specify which fields to match on between the two files being matched. Right now the program is matching on 4 fields (Tax Lot ID, PM Property ID, Custom ID and Address Line 1). Depending on the data, these may not be the right fields to match on, or the user may NOT want the program to match on some of the fields because it may result in False Positive matches. The user knows the data and should be able to specify which fields the program should match on.
(LBNL Ticket # 246 & # 18)

It is possible to have multiple data files associated with a canonical building record (such as multiple years of Portfolio Manager data or any other data where there are multiple data files). However, it is not possible to export that data; it is only possible to export the canonical building record, which only contains one instance of the Portfolio Manager or other multiple data (the first one that was loaded). Need a way to display, filter, sort, and export, the data in the multiple files associated with a record. This may involve significant database and GUI design/redesign.

Setting columns to view in Matching
In the View Matching screen, when you click on Edit Columns, you see a three column list of the fields that can be displayed, with the ones checked that are currently being displayed. This is quite cumbersome to uncheck unwanted fields and check desired fields.

It should be the same type of list that is shown in the Building List view, where the list is one column, alphabetized, with a "global" checkbox at the top that allows the user to toggle on and off the checkboxes for all the fields. This allows the user to "uncheck" all the fields, and then easily select only the fields that they want to display.

Edit column in Mapping alpha sort is case sensitive (shouldn’t be)

Data packaging feature that allows the user to export all their data (including extra data, projects, user accounts and permissions) in a packaged file format. This will allow users to move data between different SEED Platform instances. It may not be a trivial exercise, as the database structures may not be the same between the two instances.
(LBNL Ticket # 104)

Right now the Green Button interval data is not displayed. Come up with a reasonable view of the data (might be a table, might be a graph?)

Need to implement an error message class or something that allows for a standard set of error messages to be displayed under certain circumstances. Right now if there is an error, the program just displays "Error on page" and the user has no idea what caused the error.

[See LBL # 211 (error checking during Building Detail Editing), # 100 (Parent/Org error message)]

Not all fields are BEDES compliant
There is a set of "hard coded" fields in SEED (found in seed/mappings/seed_mappings.py) that are not BEDES compliance. These need to be changed to be BEDES compliant and this will presumably mean a data migration from old datasets to new datasets will be required. This is not a trivial change but should be done relatively soon.

In the Building List Detail screen, it would be nice to have the standard set of record navigation tools, such as first record, next record, previous record and last record.

Put cursor focus on the first input box any time a new screen is displayed, such as the email address in the log in page. etc.

When a list of records is displayed, there are Previous and Next buttons for the next “set” of data. Add First Record and Last Record buttons

The background screen when loading a second data file shows the “save Mappings” screen from the previous file load/mapping/matching step. This can be a bit confusing

Program should keep the settings when you move between the List and Detail view of Buildings. For example when you set these values in the List view (such as filtering or sorting), then go to the Detail view for one of the records, and then come back to the List view, that the filtering, sorting, list settings and displayed buildings per page should be the same as they were before you went to the Detail view.

Currently, what does persist when you come back to the List View is:

• Displayed buildings per page count
• Custom Column View -- the List Settings persist

What does NOT persist when you come back to the List View is (and what would be nice to have persist):

• Sort -- the list goes back to the default sort, which I am not sure what it is
• Filter -- the list goes back to the default unfiltered view

(LBNL Ticket # 272)

It seems that right now the program is matching by combining into one string IDs and Addresses, which results in false positives (and possibly false negatives). A better way would be to match IDs, then address, then whatever else. There could also be a match index for each matching, and the final result could be weighted by those. Related to Issue # 6.

When pages are refreshed, keep the left navigation settings, ie, either expanded or compacted. Right now program always reverts to compacted

Need a progress bar between data load and mapping

Depending on the data being imported, the program can take several seconds after loading the data and showing the data mapping screen. Either keep the data loading progress bar up during this time, or add a progress bar while the program is building the mapping display. Right now, the user doesn't quite know what is going on during this pause.

Released code is tagged in Github as 1.1. Need to add this version # (and possibly a sub-version #) to the About screen.

Add ability to invite members and viewers from a list, rather than one by one.

Users wants some indicator in Buildings List and Building Detail page for which data is still not matched to PM (or other) data. Should go the other way too. Automated assignment of labels indicating that PM records have not been matched.

Need a complete Admin function for setting up the fields in the program. There needs to be control of:

• user defined fields (non-BEDES/SEED) including field type, so they aren’t defined at time of data import (could be an option to allow user defined fields at time of mapping:
• what fields show up in mapping
• what fields show up in the Edit Column and Building List views
• order of fields in the Edit Column and Building List views
• what fields show up in each of the Building Detail sections (Building Info, Contacts, Area, Energy, if these get added back into the program).
• Field type settings for user defined fields

This admin function could also control what fields were actually stored in the database (? Not sure about this -- we want databases to be compatible across different orgs??).

Might also include the ability to clear out the user defined fields and start over with just the BEDES list (see Issue # 16)

The Sphinx generated Readthedocs is missing documentation for projects and label API endpoints. Looking at the code 'projects.py' has the docstrings for the missing API endpoints. Modify Sphinx to have the right pointers for API documentation.

Show Upload Errors
Program should display an error message when there is an uploading problem. For example, the server reports an error that a field has > 256 characters, and the upload process will hang. But the user does not know that there is a problem because the Progress Bar is still displayed and no error message is displayed.

Implement ability to switch between IP and SI units, including Preferences setting for this.

Example is when you want to use your own term "Property ID", the program automatically picks "PM Property ID" and doesn't let you use your own. This may be related to the hardcoded fields issue. Or if there is another field that has elements that you have typed in, the program will choose the field in the list rather than keeping the field that you typed in.

In BE.settings.main, the APP_NAMESPACE (derived from the STACK_NAME) environment variable is used to set the default S3 variables AWS_STORAGE_BUCKETNAME and AWS_UPLOAD_BUCKET_NAME. The local_untracked module only defines AWS_BUCKET_NAME, which means that different S3 buckets end up being used for different purposes.

Also, since S3 names are global, basing bucket names off the stack name will mean that conflicts will occur when multiple SEED installations are deployed by different organizations.

Fix API documentation in the code for readthedocs, or do what Portfolio Manager has, as suggested by Claudine. http://portfoliomanager.energystar.gov/webservices/home/api

Add feature to allow adding labels during record matching, so that questionable matches can be flagged for future followup

I have a Portfolio Manager data file with 192 fields in it, and during the mapping phase, the program is very slow to generate the list when I am typing ahead. To the point of waiting a few seconds for each letter to appear after I have typed it. Need to figure out a way that the program can handle this number of fields when generating the type ahead list.

FILTERING: Can’t specify exact match
When doing the filtering, can't specify an "exact match" -- so there is not a way to see only records that have Use Description set to "Multi-Family", not including "Multi-Family Condominium". Could specify exact matches as surrounded in "quotes" (?) or some other scheme. This might enable blank fields to be filtered by “”

(LBNL Ticket # 124)

Need to document the database schema.

Migrate seed to use the seed-platform versions of superperms