bedatadriven / activityinfo-r Goto Github PK

Despite setting message=FALSE, ActivityInfo API messages still appear in the knitted RMD document. It also breaks up a single chunk into separate ones for each ActivityInfo API call for some reason, which creates separate 'Share Code' buttons below. I need to set it to include=FALSE for the messages to disappear entirely, but that then means I can't share the code in the document.

names don't match via 'Export via R' and the data.frame result, probably because JSON doesn't like spaces and adds the .?

of course, i can just use the backticks, Cargo del entrevisado and I thiiiink that solves the problem (not sure??) but if I'm just copy-pasting directly from 'Export via R' I will either forget or simply not notice then when I try to 'select' columns I can't find them

We currently lack a test suite for this package A test suite would help prevent regressions caused by changes to the R package, but also ensure that changes to the ActivityInfo server does not break the R package.

We should model the test suite after ActivityInfo's existing Java API Test library that uses the following approach:

1. Every test creates a new user in the pre-preproduction environment with a completely blank slate
2. Each test sets up any required fixtures like a database, forms, etc.
3. Verify results using the temporary database.

This means that running tests require access to ActivityInfo's pre-production environment; this can be provided to any contributor who requires it.

Calling data.frame(formSchema) where formSchema is a list of class formSchema fails with error:

Error in as.data.frame.formSchema(x[[i]], optional = TRUE, stringsAsFactors = stringsAsFactors) : unused arguments (optional = TRUE, stringsAsFactors = stringsAsFactors)

This is because the implementation of as.data.frame.formSchema() doesn't have all arguments. ?data.frame says:

‘data.frame’ converts each of its arguments to a data frame by calling ‘as.data.frame(optional = TRUE)’.

Command getResource() doesn't fail gracefully if no credentials file exists (i.e. the command activityInfoLogin() hasn't been executed before). Error shown is:

Error in strsplit(credentials, ":") : non-character argument

Desired format:

FieldCode	FieldType	FieldName	FieldDescription	Required	Key	Units	Formula	Referenced Form	Choice
	section header	OUTCOME 1 - Child Development		FALSE	FALSE				
	month	Month		TRUE	TRUE				
	multiple select	OUTPUT 1.1: Learning opportunities from early childhood to end of the primary school cycle 2019/2020		FALSE	FALSE				1.1.a
	multiple select	OUTPUT 1.1: Learning opportunities from early childhood to end of the primary school cycle 2019/2020		FALSE	FALSE				1.1.b
	multiple select	OUTPUT 1.1: Learning opportunities from early childhood to end of the primary school cycle 2019/2020		FALSE	FALSE				1.1.c
	multiple select	OUTPUT 1.1: Learning opportunities from early childhood to end of the primary school cycle 2019/2020		FALSE	FALSE				1.1.e
	multiple select	1.1.a		FALSE	FALSE				"# of girls and boys (3-5), including CwDs, provided with support to access and enrolled in ECE schools 2019/2020"
	quantity	"1.1.a_SYR_Female: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_SYR_Male: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_LEB_Male: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_LEB_Female: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_PRL_Female: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_PRL_Male: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_PRS_Female: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_PRS_Male: # of girls and boys (3-5), including CwDs, provided with support to access and enrol in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_OTH_Female: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_OTH_Male: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.a_Motor/Mobility: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.a_Visual (Seeing): # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.a_Audio (Hearing): # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.a_Speaking: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.a_Intellectual: # of girls and boys (3-5), including CwDs, provided with support to access and enroll in ECE schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	multiple select	1.1.b		FALSE	FALSE				"# of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"
	quantity	"1.1.b_PRS_Female: # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.b_PRS_Male: # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.b_PRL_Female: # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.b_PRL_Male: # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.b_Motor/Mobility: # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.b_Audio (Hearing): # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.b_Intellectual: # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.b_Speaking: # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.b_Visual (Seeing): # of girls and boys (6-15), including CwDs, supported to access formal education in UNRWA schools 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	multiple select	1.1.c		FALSE	FALSE				"# of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"
	quantity	"1.1.c_SYR_Female: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_SYR_Male: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_LEB_Female: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_LEB_Male: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_PRL_Female: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_PRL_Male: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_PRS_Female: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_PRS_Male: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_OTH_Female: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_OTH_Male: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"		FALSE	FALSE	Children			
	quantity	"1.1.c_Motor/Mobility: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.c_Audio (Hearing): # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.c_Intellectual: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.c_Speaking: # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	quantity	"1.1.c_Visual (seeing): # of boys and girls, including CwDs, provided with Learning/Retention/Homework Support 2019/2020"	Disability Indicator	FALSE	FALSE	Children			
	multiple select	1.1.e		FALSE	FALSE				"# of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"
	quantity	"1.1.e_SYR_Female: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_SYR_Male: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_LEB_Female: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_LEB_Male: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_PRL_Female: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_PRL_Male: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_PRS_Female: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_PRS_Male: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_OTH_Female:# of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	quantity	"1.1.e_OTH_Male: # of ECD, primary and secondary facilitators/teachers who received training with funding provided by UNICEF"		FALSE	FALSE	Children			
	multiple select	OUTPUT 1.2: Learning and skills building opportunities for adolescents and young people		TRUE	FALSE				1.2.a
	multiple select	1.2.a		FALSE	FALSE				"# of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"
	quantity	"1.2.a_SYR_Male: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_SYR_Female: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_LEB_Male: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_LEB_Female: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_PRL_Female: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_PRL_Male: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_PRS_Female: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_PRS_Male: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_OTH_Female: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_OTH_Male: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"		FALSE	FALSE	Adolescent			
	quantity	"1.2.a_Motor/Mobility: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"	Disability Indicator	FALSE	FALSE	Adolescent			
	quantity	"1.2.a_Visual (seeing): # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"	Disability Indicator	FALSE	FALSE	Adolescent			
	quantity	"1.2.a_Audio (Hearing): # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"	Disability Indicator	FALSE	FALSE	Adolescent			
	quantity	"1.2.a_Speaking: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"	Disability Indicator	FALSE	FALSE	Adolescent			
	quantity	"1.2.a_Intellectual: # of adolescent boys and girls trained on life skills, conflict resolution and healthy life styles"	Disability Indicator	FALSE	FALSE	Adolescent			

Related to #28

A successful upload of add/updateRecord() returns a NULL when trying to save the output as an object. Below is the output from when I wrap purrr::safely() to updateRecord() and the update succeeds without issue:

While there is the Sending POST request to … resources/update and the OK (HTTP 200) messages that appear in the console, these messages aren't returned and can't be saved as objects. I have to go out of my way to use capture.output() to grab these messages but the outcome is less than ideal as the strings aren't formatted properly.

Current message that shows up in the console: Sending POST request to … resources/update is unhelpful to user as they don't really give meaningful info. Could be replaced with something like:

Updating/Adding record ID: __(id)__ in Form: __(id or label)__

Basically anything that can tell me what exactly is being acted on in the API/database. Also compared to other functions, when add/updateRecord() is successful it doesn't actually return anything. I would rather have it return some 'text' whether it's the message example provided above or something similar:

Record ID: __(id)__ in Form: __(id or label)__ OK! (HTTP 200)

Good example of something similar that already exists is the message one gets when running getFormSchema():

formsch <- activityinfo::getFormSchema(formId = "cb62ijfl8k2ymxx3")
Sending GET request to https://www.activityinfo.org/resources/form/cb62ijfl8k2ymxx3/schema
OK (HTTP 200).

Of course, when getFormSchema() is successful, it returns the actual schema object, so I don't need the message returned as an object with the result. This was just to show what I would like add/updateRecord()'s messages to look more like.

The good thing is that the form/record IDs/labels are all created as R objects inside the add/updateRecord() functions themselves OR they are directly passed into the functions as arguments so it wouldn't be hard to create the proper message texts as the info already exists inside the function itself.

based off of the work done in #44 #32 >> next step would be to let users have a data.frame, imported CSV/Excel file in R be transformed into an activityinfo form by using the various form schema helper functions

the difficult part would be stuff like single-select/multiple select since those would all be simple 'text' fields in a data.frame when checked with str() or dplyr::glimpse(). so i imagine for special fields, the user would need to specify certain columns/fields in their data.frame to be a certain field type so that the constructor function will handle it properly when creating the form schema based off the data.frame

This function cannot be found in the R package source code :
getFormRecordTable : https://www.activityinfo.org/support/docs/R/getFormRecordTable.html

It would be good to understand if perhaps this is an older feature that is no removed but that the markdown files are not getting cleaned up on the website.

Is there something in the API that would let you know if a formula in a calculated field has an error?

In the ActivityInfo interface, you can clearly see that when you are viewing a form but I don't see anyway that from a bird's-eye-view, lets you see across all of the tables within a single database where there is some error in calculated fields (or any other errors within tables in general, not just formulas in calculated fields). Unless I missed something?

Maybe this isn't possible via ActivityInfo's API but it would be interesting because for example I can create a script that can check for these errors and if there are, send an email to the project MIS person and/or me to point out which fields in which table are "erroring" and to prompt them to take a look.

This was brought up in the most recent ActivityInfo with R webinar, some helper function for users to set up the structure of the ActivityInfo form schema when using the addForm() function, kind of like the permissions() helper function we have for updateUserRole()? This same helper function can be applied to updateFormSchema().

Can also extend this for setting up the structure of Database schema as well and then updating the database schema/tree? Somewhat related, I also remember seeing a createDatabase() function in an older webinar from 2018, but its not in the current R package anymore.

We currently have an activityInfoLogin() function which accepts a username and password to authenticate to the API. However, we are deprecating this approach and now offer personal api tokens as a more secure alternative.

We should:

• Add a new activityInfoToken() method to accept and store the user's api token (User name not required)
• Add a warning message to activityInfoLogin() encouraging users to switch
• Update documentation to highlight activityInfoToken() as the preferred manner of authenticating

queryAuditLog(
  databaseId = "c6swoa5kymocnz02",
  before = end_date,
  after = start_date,
  resourceId = NULL,
  typeFilter = "RECORD",
  limit = 10000 ## ... or more!
)

Sending POST request to https://www.activityinfo.org/resources/databases/c6swoa5kymocnz02/audit
Reading username:password from C:/Users/ryo90/Documents/.activityinfo.credentials...
OK (HTTP 200).
Received 83 events...
Sending POST request to https://www.activityinfo.org/resources/databases/c6swoa5kymocnz02/audit
OK (HTTP 200).
Received 110 events...
Sending POST request to https://www.activityinfo.org/resources/databases/c6swoa5kymocnz02/audit
OK (HTTP 200).
Received 56 events...
Sending POST request to https://www.activityinfo.org/resources/databases/c6swoa5kymocnz02/audit
OK (HTTP 200).
Received 63 events...
Sending POST request to https://www.activityinfo.org/resources/databases/c6swoa5kymocnz02/audit
OK (HTTP 200).
Received 81 events...
Sending POST request to https://www.activityinfo.org/resources/databases/c6swoa5kymocnz02/audit
OK (HTTP 200).
Received 83 events...
Sending POST request to https://www.activityinfo.org/resources/databases/c6swoa5kymocnz02/audit
OK (HTTP 200).
Received 74 events...

... etc. esp. when trying to grab 10,000s of audit log records so it can get rather annoying. I don't necessary mind it giving updates but maybe less verbosely (we don't need to see Sending POST request to --url-- every single new request for example)

Running activityinfo::queryTable() gives the following error:

Request for https://v4.activityinfo.org/resources/form/[FORMID HERE]/query/columns? failed with status code 404 Client error: (404) Not Found

This endpoint is called here.

According to our (internal) API reference documentation, the /resources/form/{formid}/query/columns endpoint isn't public. The same holds for the /resources/query/columns endpoint used here.

I get requests to grab table from multiple tables and/or multiple project databases at once. Basically stuff that would be really tedious to do manually (dragging-dropping all the specific columns in the UI, switching between tables and projects, grabbing the correct reference fields, etc.).

So I have my colleagues send me a CSV/Excel of what projects, tables, and fields are needed as columns. See example below (I left-joined together the database ID and form IDs with R separately):

So basically I want to programmatically iterate over each row so that I'm grabbing the specific fields for a specific table in a specific project, then the next, then the next, etc.

So I created a function to grab the columns I need, transform it into the 'named list' structure needed for the columns argument in queryTable():

but I get this back:

Only the Survey.Date field is returned with proper data because that's the only field that is native to the form I'm querying.

So looking at this, it means that I have to deliberately go search inside each form-schema and find the correct field ID which adds another layer of complexity and makes something like this where I'm simply trying to grab data from a given set of "field names" and "table IDs" a lot harder than it should be.

What I'm given:

What I actually need, which would require even more steps instead of being able to simply match by field name.

The User Interface 'export via R' option gives me the proper field codes as shown above but I can't extract that via R.

In reality, I can't even find the above "cb1ee1b8.c288fa67" in this form's schema as it only shows me "cb1ee1b8" for the fieldId. Which means I'll have to again, jump another level to search the parent form and look up the field using only the field label and grab the field ID of the referenced field.

I shouldn't need any more than this the first screenshot in this email to grab the fields I need from multiple tables in multiple projects but as I explained above its not enough and that's a huge bottleneck for me. I can't ask my colleague to provide me with the reference form IDs and reference field ID because at that point they might as well just manually download all of the forms themselves using the user interface.

So what should be a simple loop over queryTable()​ per 'table_id' selecting columns in 'Field.Names' turns into a complicated process of jumping through 2 different form schemas (the child form schema and then the parent form schema, matching the field by label) then selecting using that (or so I imagine the process would go).

Is there no way to make this process easier?

Or, as I imagine this won't be a simple fix, if we could instead fix the issue with large forms with many reference fields getting status 500 errors when trying to extract them from the API with queryTable(), then I can just grab everything (queryTable() with no 'columns' argument) and dplyr::select() afterwards instead.

Preserve the data inside fields that have changed types (within type changes that make sense, of course).

## changing field type from form-schema
## ... does not preserve data in changed fields??

## grab form schema from existing table: Recover_Record_EX1 in FIFES
fifstab <- queryTable(form = "cb62ijfl8k2ymxx3")

dplyr::glimpse(fifstab)
## NUMPAR is a 'quantity field'

formsch <- activityinfo::getFormSchema(formId = "cb62ijfl8k2ymxx3")

## change NUMPAR field type from 'quantity' to 'FREE_TEXT'
formsch$elements[[1]] <- purrr::list_modify(formsch$elements[[1]], type = "FREE_TEXT")

## Revert formschema classes to regular "list" so that JSON // ActivityInfo can parse it properly
## formSchema S3 class >>> list
class(formsch) <- "list" 

## formField S3 class >>> list       class(formsch$elements[[1]]) 
formsch$elements <- lapply(formsch$elements, function(e) {
  class(e) <- "list"
  e
})

## Update schema with changes back to database
updateFormSchema(schema = formsch)

Currently doesn't work, data is wiped away when underlying field type changes. This was actually a Feature request brought up by project staff originally but since I knew about the updateFormSchema() function I had told them they could just use the R package. But unfortunately with the above problem, it's a harder sell as project staff don't want to lose data that already exists in the fields.

Of course, values getting wiped makes sense if I'm changing like DATE to numeric or Attachment to text or some other weird conversion but for something like text to quantity (and vice-versa), which is a conversion that's going to happen a lot (whether due to a mistake or otherwise), it would be nice if we could preserve the underlying values somehow?

Split from #32 (comment)

getFormSchema() has a method that allows you to transform it into a data.frame but getDatabaseSchema() currently does not.

it'll be nice to get the database schema into a data.frame format as well so that updates to the database schema can also be handled via data.frames , similar to #33

afda_schema <- getDatabaseSchema("c5h9mudkz3sqce52")

## Tables come from the "resources" list
## turn below code into a function

## Get table labels
tab_label <- afda_schema %>% 
  purrr::pluck("resources") %>% 
  purrr::map(pluck("label")) %>% unlist()

## Get table IDs
tab_id <- afda_schema %>% 
  purrr::pluck("resources") %>% 
  purrr::map(pluck("id")) %>% 
  unlist()

## put 'em together to create a nice look-up
data.frame(
  Table = tab_label,
  table_id = tab_id
)
``

fairly simple, see example below:

rechist <- getRecordHistory(formId = "c8301e4022c", recordId = "cman42dl4qztru2d")

rechist_df <- tidyr::as_tibble(rechist) %>% 
  tidyr::unnest_wider("entries") %>% 
  tidyr::unnest_wider("user")

would like to expand out the 'values' list-field into their own columns but need to account for the fact that the first row when the record is added has a NULL in it compared to the other values for that column

I would like to propose we comment out or remove this line of code to prevent credentials path location leaking out unnecessarily in logs potentially generated in Github actions and such. While presumably anyone who has file access should have access, I'm not sure announcing the location every time is useful. Better to have this documented another way. Also I imagine that many users that are saving credentials to file instead of as an environment variable or secret are potentially less aware of security risks.

When working inside R, having to also remember the ActivityInfo formula syntax is an extra layer of cognitive load that can be confusing when I'm also working with R formula syntax within other lines of code in the same script.

Nic mentioned checking out how dbplyr does its translations from SQL <> R

might be other interesting packages to draw inspiration from

currently the process of:

• having an Excel/CSV file, uploading the schema (or more time-consumingly building the schema in the user interface), and then finally uploading the data is a bit cumbersome

alongside the changes to improve how we manipulate the form/database schemas in issue #32 and #31 we should think of ways to streamline this process

Create {pkgdown} website for documentation and vignettes (currently lying dormant in inst/examples folder in the Github repository)?

Some of functions also require preparation in the ActivityInfo UI beforehand (setting up forms, roles, formulas, etc.) so it would be nice if documentation could also include stuff like that.

Ex. The various permissions/updateRole functions took a long time for Robert and I to properly grok as the set-up includes lots of actions needed in ActivityInfo UI in tandem with what's needed in the R code to run, which is a lot of lists-within-lists for filters, parameters, etc. This is a topic that would be perfect for a vignette hosted on a pkgdown website with screenshots of the actual tables and database settings in the ActivityInfo UI alongside the R code.

For example:

activityinfo::getResource("reports")

Results in the error:

Error in sys.call(which = -4) : not that many frames on the stack
6. sys.call(which = -4) at rest.R#47
5. structure(list(message = message, call = call), class = c(http_class, type, "condition"))
4. http_condition(result, type = type, task = task, call = call) at rest.R#47
3. activityInfoAPICondition(result, task = task) at rest.R#75
2. checkForError(result, task = task, requireStatus = requireStatus) at rest.R#105
1. activityinfo:::getResource("reports")

For at least some forms, the call to rbind in the as.data.frame.formSchema method fails with an error:

numbers of columns of arguments do not match

See this commit for the offending changes: bd0fa80

The user interface uses a specific set of heuristics to produce a flat table from a form with references. In many cases, those using R want to refer to the same set of columns and column names.

We need a way to query this form specifically, perhaps a separate function.

Hi all,

For the database introspection tool I would like to check for key fields in each table.

It appears that here: https://github.com/bedatadriven/activityinfo-R/blob/master/R/forms.R#L188 the method for coercing the form schema to a data.frame removes the column.

Could we please remove the line so we can keep it in?

Thanks!

Ryo

EDIT: upon discussion with Alex + Nic, this will be more of a focus in January esp. as this is something ActivityInfo needs to do on their back-end of the API rather than any implementation on the R side of things (at least until the data is exposed to the API)

The (S3) methods in this package result in a WARNING when running R CMD check because the method signature doesn't match the signature of the generic function. The Writing R Extensions manual lists three rules for S3 methods:

1. A method must have all the arguments of the generic, including … if the generic does.
2. A method must have arguments in exactly the same order as the generic.
3. If the generic specifies defaults, all methods should use the same defaults.

This should be fixed.

A call like

activityinfo:::getResource(path = "form/{formId}/query/rows", queryParams = list("[Record identifier]" = "_id"))

where {formId} is a valid form identifier, gives a 400 Bad Request error, because the getResource() function doesn't encode the URL. The brackets in this example are the culprit.

To fix, getResource() should not create the full URL itself using paste() but rather use the query argument to httr::GET().

Split from issue #31

Function should take:

• Original field id
• New field schema
• Conversion function

Hi all,

I seem to be having a problem when running queryTable() for this form/table.

This is directly copy-pasted from Export >> Export via API >> Query using R in the ActivityInfo interface on the form linked above.

PrList <- activityinfo::queryTable(
  "c6ctrxakunxk7kv2",
  "Name" = "czh8vcgkunxkpin3",
  "Country Name" = "c5qq4h8kunxl5h34.c5rpu3wkjslzmo48",
  "From" = "c9e6ri0kunxlb5a5",
  "To" = "cahg29ikunxleji6",
  "Code" = "ca02a7ykunxlijx7",
  "ActivityInfo Database ID" = "c7x5mnvkunxm07a8",
  "Indicator Report ID" = "c6ctrxakunxk7kv2.cbr6qxtkunxm7jr9",
  "Hierarchy" = "c6ctrxakunxk7kv2.cla4lgskuwp2us9d")

The error:

OK (HTTP 200).Error in `[[<-.data.frame`(`*tmp*`, cn, value = NA) : 
  replacement has 1 row, data has 0

Here is my session info:

R version 4.0.5 (2021-03-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19043)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 
[2] LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    
system code page: 65001

attached base packages:
[1] stats     graphics  grDevices utils     datasets 
[6] methods   base     

other attached packages:
[1] activityinfo_4.17 centralperk_0.1.0

loaded via a namespace (and not attached):
 [1] digest_0.6.27     R6_2.5.1          jsonlite_1.7.2   
 [4] magrittr_2.0.1    evaluate_0.14     httr_1.4.2       
 [7] rlang_0.4.11      curl_4.3.1        rmarkdown_2.8    
[10] rjson_0.2.20      tools_4.0.5       purrr_0.3.4      
[13] xfun_0.23         yaml_2.2.1        compiler_4.0.5   
[16] htmltools_0.5.1.1 knitr_1.33  

The name and the category are included in the result, but not the identifier.

trying to upload table with ~19000 rows and ~200 columns but it looks like i'm hitting the size limit as I get a 413 API error

Similar to #28 . For a successful query, the returned object is the actual data.frame object so the status message being returned is not a concern. So the issue is more about having a better message show up for the user in the console and for me a better message that will show up in my log files.

Something like Getting Form: '__formId__/formLabel__' would be much better than the cryptic Sending POST request to query/columns which doesn't tell me anything about what I'm trying to download from the database using the queryTable() function.

It should be possible to query a date range of a database's audit log as a data.frame.

/cc @Ryo-N7

At the moment, there are few functions that are not behaving the same as others. When there is an error, they do not return an error but rather a specially formatted object or a NULL. This makes it difficult to have a consistent and predictable pattern for each function call in the package and may also confuse users. I'm opening this issue to be able to document these in one place and so we can discuss.

As I am writing tests for the package functions, I've noted two so far:

• getDatabaseUsers() returns NULL if there is no user (http status == 404). However, one should expect that if the user is being fetched by ID then you know the ID and if it does not work, you would expect an error rather than a NULL value like with the other package functions.
• addDatabaseUser() returns a custom object that is contains both a boolean for whether the user was added returnvalue$added = TRUE and returnvalue$user as a list if the user was added. If there is an http status == 400, it gives returns returnvalue$added = FALSE and returnvalue$error with the response object and finally with any other http status it throws an error.
• most other functions will just throw an error if there is not a 200 status and return the API object otherwise.

Ideas:

1. make the API objects more user friendly and a pleasure to code with so we do not need custom return values
2. make helper functions to work with the API objects
3. make a vignette on dealing with errors in a good way (logging/logic/etc)
4. create more consistency and softly / slowly deprecate the return values that are more inconsistent

updateRecords() and other functions emit too much debugging. This risks to drown out logging. @Ryo-N7

Following PR #27, the request is to offer a way to avoid the request to save the file while in interactive mode.

When I try to install GitHub package with the command:

install_github( "bedatadriven/activityinfo-R", ref = "release")",

the installation fails with following message:

Downloading GitHub repo bedatadriven/activityinfo-R@release
from URL https://api.github.com/repos/bedatadriven/activityinfo-R/zipball/release
Installing activityinfo
"C:/PROGRA1/R/R-321.3/bin/x64/R" --no-site-file --no-environ --no-save
--no-restore --quiet CMD INSTALL
"C:/Users/Lara/AppData/Local/Temp/RtmpARBQ3n/devtools5301bc13101/bedatadriven-activityinfo-R-a7898de"
--library="C:/OneDrive/المستندات/R/win-library/3.2" --install-tests

• installing source package 'activityinfo' ...
  Warning in file(file, ifelse(append, "a", "w")) :
  cannot open file 'C:/OneDrive/GacSJdOGJ/R/win-library/3.2/activityinfo/DESCRIPTION': No such file or directory
  Error in file(file, ifelse(append, "a", "w")) :
  cannot open the connection
  ERROR: installing package DESCRIPTION failed for package 'activityinfo'
• removing 'C:/OneDrive/المستندات/R/win-library/3.2/activityinfo'
  Error: Command failed (1)

The following code inside the conversion from an object of class formSchema to a data frame

Error in as.data.frame.default(x[[i]], optional = TRUE, stringsAsFactors = stringsAsFactors) : cannot coerce class ‘"formField"’ to a data.frame

Split from #44 (comment)

The R package should expose a function for fetching metadata on the set of forms that belong to a database, or to a billing account. The results should include a list of forms, each with the metadata currently stored internally:

• The form's label
• The form's database
• The number of records in the form
• The time of the last update to the form
• The form's version number
• The form's schema version number
• The visibility of the form (private, reference or public)

Additional permissions metadata that would be useful:

• The number of users with read access
• The number of users with write access

Update form schema in data.frame format and pass into the updateFormSchema() to update it. Not as much a priority as previous points as people can just get better at manipulating list objects but could be a useful functionality for all users not just AV.