ccao-data / ccao Goto Github PK

New columns have been added to the condo modelling view since the last time the pipeline was run. They need to be added to ccao's var_dict.

Forgot to update this when I updated vars_dict.csv 🤦‍♂️

The vars_dict dataset in this package is used heavily by ccao-data/model-res-avm and ccao-data/model-condo-avm. However, iterating with the model requires us to constantly update this dictionary (and thus the package). Additionally, keeping the dictionary up-to-date is tedious, manual process.

I propose we automate the creation and maintenance of this dictionary and move it out of this package on to S3. We can use Glue and other AWS APIs to construct it on a schedule, then store it in a public bucket.

The modeling pipeline variable needs to be in the package or ingest will throw an error.

Cut a new release of the package containing the following changes:

• Add pre-commit badge (4b9688c)
• Deploy to GitHub pages (c559f46)
• Add core team to CODEOWNERS (#1)
• Add meta_card_protation_rate and loc_tax_municipality_name to vars_dict (#4)
• Update authors and add docs to sidebar (#5)
• Consolidate vars_dict to focus exclusively on renaming/recoding (#6)

Release checklist

• Make sure the changes meet styling and linting requirements:
  • Styling: Run styler::style_pkg() in the console
  • Linting: Run lintr::lint_package() in the console
• Build any documentation updates by running devtools::document() in the console
• Run the build and test processes locally to check for errors:
  • Build: In RStudio, go to the Build tab, then hit Check
  • Test: In RStudio, go to the Build tab, then hit Test OR devtools::test() in the console
• Check the vignettes by building them locally. Run pkgdown::build_site() in the console
• Rebuild the README manually. Knit README.Rmd from within RStudio
• Increment the package version in the DESCRIPTION file appropriately, following the schema laid out in the README
• If the code updates were so substantial that the current database also needs to be updated, follow the database release checklist issue template
• Push the code updates to GitHub. Wait for the resulting CI pipeline to finish
• If there are no pipeline errors, merge the branch to master
• Wait for the merge CI pipeline to finish. If there are no errors, cut a new release from master. Create a new git tag with the version number and title the release with the same version number. Be sure to add a changelog detailing what you updated

I incorrectly assumed the ccao db was in our dag and that is_corner_lot would exist as a column name because of that. Since the ccao db is not in the dag, this column only appears as ccao_is_corner_lot and needs to updated in vars_dict.

This column name was changed and will throw an error in the modeling pipeline as a result.

We need to add:

• meta_sale_count_past_n_years
• char_class
• prox_airport_dnl_total
• prox_nearest_secondary_road_dist_ft
• prox_nearest_university_dist_ft
• prox_nearest_vacant_land_dist_ft
• ccao_is_active_exe_homeowner
• ccao_is_corner_lot
• ccao_n_years_exe_homeowner

The data team has added new employees, the ccao_ids.R should be updated accordingly.

We forgot to add sv_is_outlier to the package after completing the sales validation pipeline.