ccs-amsterdam / amcat4r Goto Github PK

Might be obvious from a database perspective, but took me a loong time to figure out: the axes argument in query_aggregate() is apparently limited to "date" and "keyword" field types, if I'm not mistaken. Relatedly, the set_fields() function seems to work only on empty indices (i.e. before uploading documents).

Both issues should be flagged in the documentation and/or backed by more informative error messages than the current 'internal server error' (HTTP 500) to save future users some time ...

Thank you!

# No field specification 
my_data <- data.frame(date = rep(as.Date("2020-01-01"), 6),
                      title = paste0("Title ", seq(1:6)),
                      text = paste0("bla bla blub ", seq(1:6)),
                      aggregator = c("A", "A", "A", "B", "B", NA))
create_index("my_index", description = "My Index")
upload_documents("my_index", my_data)
get__fields("my_index")
query_aggregate(index = "my_index",  
                queries = NULL,
                axes = list(list(name="aggregator",
                                 field="aggregator")))  # HTTP 500 error
set_fields(index = "my_index", list(aggregator = "keyword")) # HTTP 500 error

# With field specification 
my_data <- data.frame(date = rep(as.Date("2020-01-01"), 6),
                      title = paste0("Title ", seq(1:6)),
                      text = paste0("bla bla blub ", seq(1:6)),
                      aggregator = c("A", "A", "A", "B", "B", NA))
create_index("my_index", description = "My Index")
set_fields(index = "my_index", list(aggregator = "keyword")) # Before upload!
get__fields("my_index")
upload_documents("my_index", my_data)
query_aggregate(index = "my_index",  
                queries = NULL,
                axes = list(list(name="aggregator",
                                 field="aggregator")))  # Ah!

While going through the functions, I noticed list_indexes() and create_index() did not work. I found that the issue was that they were sending requests to http://localhost/api/index which produced a 405 (Method Not Allowed) error. I fixed list_indexes() in 15d41eb and will fix create_index() in a minute. I wonder though:

• is this because of my setup (docker)
• are there more functions which do that
• Would it be more robust to fix the behaviour of request(), which removes trailing / from the url

@kasperwelbers has done most of the work and documented it. It should be implemented in amcat4r:

• write function to wrap getting the middlecat config and getting the token
• store the token on disk so it persists over sessions. httr2 has this implemented and stores encrypted token in rappdirs::user_cache_dir("httr2") (they can be decrypted with httr2 by any user who has access to the file). The httr2 readme suggests to ask the user if they are happy with the risk in storing the token on disk. If they opt out, the token will only be stored in memory.
• Another function should wrap the token refresh flow. It should probably be triggered whenever the user makes a request to amcat, check if the token is still valid, and refresh if necessary.
• It probably makes sense to keep the old login() function for now, but show a deprecation message informing the user that the old amcat authentication is slowly phased out.
• Write tests for the new functions

We do have a page parameter in the API, but never implemented it here or in amcat4py.

I started to write some tests for middlecat in #4. It would be nice though to do more testing. There are two approaches which I'm considering:

1. Mocking a server response with httr2::response + httr2::with_mock like in the test for getting the config. This is quite cumbersome though as it requires building half the api again just for testing. The vcr package, which solves this problem for httr does not support httr2 and I'm also unsure if it would solve our purpose since we process server responses right away.
2. We can just spin up docker images with the actual apis and query them. This probably means that the tests will run for quite a while, but it seems generally doable and gh actions does some caching there, it seems.