shapiromatron / bmds-server Goto Github PK
View Code? Open in Web Editor NEWA web application for EPA's BMDS software
License: MIT License
A web application for EPA's BMDS software
License: MIT License
Make the input data views on the modal consistent with the word report. There should be two tables; the model options, and the the parameter prior values.
The goal of this PR is to replicate the functionality of the Logic
tab in the BMDS 3.1.2 excel software. In the future, the backend will support accepting data from the frontend with these data.
We'll need to create a new tab in a bmds setting and build a form similar to the Excel spreadsheet. This PR should be able to configure all options as expected, with outputs synced w/ the store.
// constants
const BINS = Object.freeze({
NO_CHANGE: 0,
WARNING: 1,
FAILURE: 2,
}),
RULE_NAMES = Object.freeze({
BMD_MISSING: "bmd_missing",
BMDL_MISSING: "bmdl_missing",
BMDU_MISSING: "bmdu_missing",
AIC_MISSING: "aic_missing",
ROI_MISSING: "roi_missing",
VARIANCE_TYPE: "variance_type",
VARIANCE_FIT: "variance_fit",
GOF: "gof",
GOF_CANCER: "gof_cancer",
BMD_BMDL_RATIO_FAIL: "bmd_bmdl_ratio_fail",
BMD_BMDL_RATIO_WARN: "bmd_bmdl_ratio_warn",
ROI_LARGE: "roi_large",
WARNINGS: "warnings",
HIGH_BMD: "high_bmd",
HIGH_BMDL: "high_bmdl",
LOW_BMD_WARN: "low_bmd_warn",
LOW_BMDL_WARN: "low_bmdl_warn",
LOW_BMD_FAIL: "low_bmd_fail",
LOW_BMDL_FAIL: "low_bmdl_fail",
CONTROL_RESIDUAL_HIGH: "control_residual_high",
CONTROL_STDEV_FIT: "control_stdev_fit",
DOF_ZERO: "dof_zero",
});
// example settings (with defaults but can be configured in store):
const logic = {
recommend_viable: true,
recommend_questionable: false,
sufficiently_close_bmdl: 3,
rules: {
[RULE_NAMES.BMD_MISSING]: {
enabled_continuous: true,
enabled_dichotomous: true,
enabled_nested: true,
threshold: null, // null or float
failure_bin: BINS.FAILURE,
},
[RULE_NAMES.VARIANCE_FIT]: {
enabled_continuous: true,
enabled_dichotomous: false,
enabled_nested: false,
threshold: 0.05, // null or float
failure_bin: BINS.WARNING,
},
// ...
},
};
Investigate why the x and y axes are incorrect on the plotly plot when things are changed. Determine if a fix can be made, and if so, fix it, otherwise document the issue.
Steps to recreate:
Currently the selected dataset on the Data and Output tabs are synced, that is, if a user changes the selected dataset on one tab, it persists in the other. Update so that tab changes are independent.
We'll likely keep the dataStore injection, but the input props will change so that these props which will come from the relevant store are passed in (eg; feel free to adjust):
<DatasetList
selectedModelIndex={outputStore.selectedModelIndex}
onChange={outputStore.handleSelectedDatasetChange} />
Each dataset will have an optional identifier field, which can be a string or a number. This will be carried thorough to the outputs as well
Currently, when hovering over a data point on a plotly plot, the format is 0.5555 +0.6666 / - 0.3333
which is long and hard to read.
Using the hovertemplate attribute might be easiest: https://plotly.com/javascript/hover-text-and-formatting/#hovertemplate
Using the example above, the formatted text should look something like 0.555 (0.333, 0.666)
Please check and post screenshots for both continuous and dichotomous datasets.
This dataset demonstrates some visual behaviors that need to be fixed:
Dose N Mean Std. Dev.
0 9 3302.8 410.9
0.05 10 3342.5 545.57
0.15 10 3342.5 800.35
0.5 10 1625 504.98
Items to change:
Refine how the plotly plots are currently rendered:
margin: {l: 50, r: 5, t: 50, b: 50}
legend: {yanchor: "top", y: 0.99, xanchor: "left", x: 0.01}
At the end of a BMD analysis, generally a user needs a single BMD as the "answer" for this analysis. When using the Bayesian model averaging approach, this selection will be done automatically by weighting all models based on fit and building a BMD estimate from all model weights.
However, in other cases, generally a decision logic tree is applied a model is recommend for selection. A user must manually select a best-fitting model however, and they need the ability to potentially override the selection.
Therefore, we'll need to update the UI to account for this. On the outputs tab, after a model has executed and results are shown:
<tr class="table-success">...</tr>
class in bootstrapCurrently the "logic" tab when in read only mode is still an editable form. Modify to make a read-only view, like was done with other tabs.
Update the model output table for selected datasets. Currently, all model outputs regardless of model type (frequentist restricted, frequentist unrestricted, bayesian) are shown in a single output table and intermingled. Instead, we'll show frequentist models in one table, and bayesian models in a separate table. In additional, a table subheading will be shown for frequentist restricted and frequentist unrestricted models. See screenshots below:
The backend has to be updated with additional metadata to place models in the correct place; this will be done in a subsequent PR. For now, assume all output models are all three types; so the UI will show duplicate rows in different places until the backend has been revised; that's ok for the purposes of this PR...
create a button for loading analysis to load a json file.
hydrate the store with json file settings.
Previously, outputs which were created in BMDS were in a format which was speculative, but more recently, the format has stabilized and should be close to final BMDS#33.
Based on these changes, output values will need to be updated in the user interface of bmds-online.
**Design considerations: ** Generally the data format for continuous and dichotomous datasets are pretty different. Therefore, it is appropriate to separate components for output tables as needed for each table type. If you really want to use the same component, then just bake the constants directly in the file, since they'd only be used in one place. Feel free to discuss if it'd be valuable.
tbl()
methods here, which may be instructive in how to build the html tables since the data structure is identical:
Currently the output selector (the blue "Dataset #1") on the far-left takes up a lot of vertical screen-space, which is needed for some of the output plots and visualizations. Per a suggestion from @bishwobhandari ; explore changing this to a select input or potentially a horizontal pill instead of a vertical pill.
Consistent with #101; update the OptionsForm to use the new approach described in that PR to update how content is saved; changed. You may want to investigate use of _.set()
which allows you to change items in an array., eg _.set("options[2].bmr", event.target.value)
Reporting can sometimes take a long time, especially with the formatting requested.
The download Excel and download Report capabilities may not return a synchronous response anymore as they currently do. Instead, If this is the first request, we will schedule a task to the redis job-queue for a report to be created, and a JSON response of pending will be submitted to indicate the job is ongoing. When the job is complete, it will report a docx/xlsx report. If the job has already completed and the report is cached, it may return on the first response.
Work started on #88; @bishwobhandari can tackle the frontend and myself or @rabstejnek can handle the backend.
Currently, entering a dataset into bmds-online, can be time consuming, especially if the dataset is large.
Therefore, we will add a new button named "Copy from Excel" to the "Data" tab of bmds-online. If this button is clicked, a modal with a single large textarea with a "save" button appears. A user can copy paste data from Excel into the form.
Fortunately, a copy/paste from excel is tab-delimited, eg.,
5 1 100
10 2 200
20 3 300
30 4 4000
We can parse the textarea, and if the number of columns matches the # columns for the dataset type (3 dichotomous, 2 continuous individual, 4 continuous summary), the data is saved, the modal is closed, the form/plot updates so the data are shown as well as the # row/columns.
Validation rules:
Help text:
Copy/paste data from Excel into the box below. Data must be all numeric with no headers or descriptive columns.
At any times, a user can hit save again to validate and save/close if successful or close the form.
Re-enable the "lognormal" setting.
In addition:
Suggestion - create new computed
boolean prop in the optionStore - hasLognormalDistribution
.
There should be no changes for dichotomous datasets, just continuous.
This is higher priority that #128.
When clicking a modal average summary result, the modal table will look different than standard model detail pages.
Please build the following data tables in the screenshot, including the prior column (shown on the summary table) as well:
In addition, the CDF table and plot should be shown, as well as the model dose-response visual being created in #121
DatasetName 0
Dose
Response
, for continuous summary, Mean
, for Dichotomous, Incidence
Currently we've worked on the view where a user is configuring a bmds session and executing it. However there's also a read-only view of a bmds-session where settings cannot be configured or executed and should only be presented instead.
In this case, we'll need to present all form-based components in a read-only table style view. For example, the options or model selections should just be static tables. All buttons for saving, loading, or running, should be hidden, etc.
You can determine which mode we are in by reading the config on the page; if there's no editSettings
component in the config, then we're in the read-only view. https://github.com/shapiromatron/bmds-server/blob/master/bmds_server/jobrunner/views.py#L51-L66
use a _.groupBy
to show all results for a given dose group in a single row, instead of one row for each pair
Update model store:
modelName.split("-")
from modelsStore; pass in appropriate data via the calling methodUpdate to options store:
Updates to main store:
Component updates:
Please work off the next
branch.
frontend/src/components/Main/main.css
, unless they are component specificmb-2
etc where relevant insteadIt's ok if the colors and exact layout do not match our excel prototype. We've migrated to the web and now is a good time to allow for breaking changes.
This is a collection of related updates on the user-interface for the "main" page.
setModelSelection("frequentist_restricted", "Gamma", e.target.checked)
that basically adds or removes the model (2nd argument) from the array (1st argument). That way we dont have to manage any intermediate state. This change should remove a fair amount of code and complexity.bmds_server.jobrunner.models.Job.default_input
method, though if factored correctly it's probably less redundant to do it using the same handlers as the model_type change handlerThe recommended model summary text takes up too much room and makes it hard to view results.
Thus, we will adjust layout using a few approaches:
fa-eye
and fa-eye-slash
as an icongetModelBinLabel
value, with no getRecommenderText
HelpTextPopover
component that shows the
getModelBinLabel
textgetRecommenderText
Currently on the Output
tab, curve-fits are only shown on when hovering over a row in the table. Change this so that all curves are shown by default, and when one is hovered upon the color changes to black and the line width is thicker.
When hovering over a curve, change the line color or width to indicate that its being shown.
Also, it might be good to investigate the React plotly update method instead of redrawing the plot entirely; that would speed up performance and wouldn't look as choppy.
Begin designing the frontend application for the nested dichotomous model. A subsequent request will design the backend to save the data, and more work for execution later.
Here are the items, and the enum values that we'll save as state:
Nested Dichotomous
(ND
)dose, litter_n, incidence, litter_covariate
)The example dataset should be:
Dose Litter Size Incidence Covariate
0 16 1 16
0 9 1 9
0 15 2 15
0 14 3 14
0 13 3 13
0 9 0 9
0 10 2 10
0 14 2 14
0 10 1 10
0 11 2 11
0 14 4 14
25 9 5 9
25 14 6 14
25 9 2 9
25 13 6 13
25 12 3 12
25 10 1 10
25 10 2 10
25 11 4 11
25 14 3 14
50 11 4 11
50 11 5 11
50 14 5 14
50 11 4 11
50 10 5 10
50 11 4 11
50 10 5 10
50 15 6 15
50 7 2 7
Instead of using raw html for select
and input
components, use some predefined components to standardize the user-interface
components/common/SelectInput
; modify as needed to workcomponents/common/FloatInput
; modify as needed to workcomponents/common/IntegerInput
components/common/TextInput
components/common/CheckboxInput
if this make sense (but only if it makes sense; I know the model selection items ones are already heavily componentized so they may be fine as is)components/common/LabelInput
if this make senseMigrate from custom html to these components. It's ok to not migrate over very customized inputs if needed, but we should default to using the components when possible.
Add a new "dirty data" flag, where if any settings have been modified, then the software cannot be executed unless the settings have been saved. For now, it may just be a new dirtyData=false
flag on the main store, and if content in any store changes, set true.
This would occur if an analysis has already been executed, and a user changes input settings. Only when the results are saved and the response is successful is the dirty data flag removed.
A half-baked version is available #110; feel free to use this branch or start a new one. I had an issue when data was hydrated from the server, the dirty flag would be triggered, which isn't what we'd want
I’ll add a new option for “models”, which will be required. To get the tool with outputs in the current format, you would set to “default”, otherwise you can set to an array of objects. Each object will be a model name, and I’ll have to add a way to specify custom model settings for that model. For example, you should be able to do something like this:
{
"models": [
{"name": "Polynomial", "model_settings": {"degree": 2}},
{"name": "Polynomial", "model_settings": {"degree": 3}},
{"name": "Polynomial", "model_settings": {"degree": 4}}
],
}
Or this:
{
"models": "default"
}
Our DataStore.js
file has grown very large and now contains multiple different independent stores. Let's split the current DataStore.js into multiple files and combine them into a single root store, using a pattern similar to this:
https://mobx.js.org/best/store.html#combining-multiple-stores
class RootStore {
constructor() {
this.userStore = new UserStore(this)
this.todoStore = new TodoStore(this)
}
}
class UserStore {
constructor(rootStore) {
this.rootStore = rootStore
}
getTodos(user) {
// access todoStore through the root store
return this.rootStore.todoStore.todos.filter(todo => todo.author === user)
}
}
class TodoStore {
@observable todos = []
constructor(rootStore) {
this.rootStore = rootStore
}
}
We recently merged in the integration test branch after successfully testing both locally and in gtihub actions #71 .
However, the initial integration tests were stubs of tests instead of actual tests. Actual tests should be created instead. The goal of these integration tests are to demonstrate proper rendering of the javascript frontend and successful execution of the models on the backed along with state and flow functioning properly
Points of reference:
Update data entry form using the attached mockup:
fa-trash
from font awesome instead of 'x'On the data store, try to exactly mirror the structure of how the datasets
object is returned from the server. Then it will be easy to pass the data back and forth
The components should roughly be (this is rough; feel free to change)
<DatasetFormContainer>
<DatasetSelector/>
<DatasetForm/>
<DatasetScatterplot/>
<DatasetFormContainer/>
There will also be a selectedDatasetIndex
, and whenever a new dataset is created or a different dataset is selected, this will update, and will change what is displayed on the DatasetForm
and DatasetScatterplot
visual components.
The layout above is rough and minor changes are fine, though keeping the three column layout is preferred if possible.
Update dependencies
in package.json to the latest versions. Do not worry about devDependencies
. Test and make sure all functionality still works.
Make word output more consistent with changes in the output html page:
Note that the effort required for this will likely result in a PR in both this repository and https://github.com/shapiromatron/bmds
Previously, a single dose-response plot (and table) visualized all model outputs for frequentist and bayesian models. This plot was removed in #120, and should be restored with updated functionality.
src/components/common/DoseResponsePlot.js
We'll only show the frequentist table when frequentist models exist, and only show the bayesian table when bayesian models exist.
When save-analysis is performed the data returned is already data.inputs
when fetch-analysis is performed the data returned is data.
when we checking data.inputs in updateModelStateFromApi() , the data object is different each time that is why the is Executing is not True when save analysis is performed (because data object doesn't have inputs object. the returned data are follows for each case 👍
save analysis returned data
analysis_description: ""
analysis_name: ""
bmds_version: "BMDS312"
dataset_type: "D"
datasets: [{…}]
models: {frequentist_restricted: Array(2)}
options: [{…}]
fetchAnalysis returned data
api_url: "/api/v1/job/c35c5ae6-6a5c-433f-8b60-fbd422a3c57e/"
created: "2020-06-09T15:15:17.758652Z"
ended: null
errors: ""
excel_url: "/api/v1/job/c35c5ae6-6a5c-433f-8b60-fbd422a3c57e/excel/"
has_errors: false
id: "c35c5ae6-6a5c-433f-8b60-fbd422a3c57e"
input_url: "/api/v1/job/c35c5ae6-6a5c-433f-8b60-fbd422a3c57e/inputs/"
inputs: {bmds_version: "BMDS312", analysis_name: "", analysis_description: "", dataset_type: "D", models: {…}, …}
inputs_valid: true
is_executing: false
is_finished: false
output_url: "/api/v1/job/c35c5ae6-6a5c-433f-8b60-fbd422a3c57e/outputs/"
outputs: null
preferences: ""
started: null
The Model type
field on the Main
page should filter the list of available data types on the Data
page.
When a user changes the Model type, reset (aka delete) the options, datasets, and model selections which were previously made, since this is consistent with what is saved to the server.
Add simple plot visuals to the site. Write components in a way so that they're well contained, so that we can swap out the plotting library in the future as needed, as requirements may change.
Would recommend using plotly as a starting point as it's a nice high level library that has the functionality that we need, but am open to other suggestions.
A major requirement of the application is the ability to download the results of a bmds assessment in reports. We have some preliminary code that can generate these word and excel reports, but the purpose of this PR is just to stub out how users can access them and not actually connect them.
On the backend:
excel
works and returns a valid report. If it does not, just stub out an example report. https://github.com/shapiromatron/bmds-server/blob/master/bmds_server/jobrunner/api.py#L104.word
that returns reports. For now, then can be just simple stubs of empty reports that will be refined. https://github.com/shapiromatron/bmds-server/blob/master/bmds_server/jobrunner/api.py#L104. It should return a hello world report, generated by python-docx
On the frontend:
Python-docx: https://python-docx.readthedocs.io/en/latest/
The header and footer for this application will be revised away from the default bootstrap4. A subsequent task will make the header and footer styles configurable based on settings configured at the application level so that we can use the EPA branding or more generic branding.
includes/epa-header.html
and include in base.html
.app-header
from the clmapper siteincludes/epa-footer.html
and include in base.html
.main-footer
from the clmapper siteReview all rendered components and remove all unnecessary DOM elements and nesting.
Examples:
Create a plot like this showing bmdl/bmd/bmdu for frequentist models, and one for bayesian models. Each row would be a separate dose-response model (including model average), there would be up to three values for each row. Reorganize the figures so that the would take a full row and each would be col-6. Interactivity is minimal; if easy, add tooltips to show the bmd/bmdl/bmdu values.
https://towardsdatascience.com/lollipop-dumbbell-charts-with-plotly-696039d5f85
Since I'm having troubles running processes with environment variables on Windows, use this library instead to store secrets.
In settings/production.py
, replace all os.environ
with config()
calls, and create a new file to capture.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.