PyCap/Requests doesn't seem to like the ssl certificate on my server, but it looks like Requests can be set to ignore ssl verification using the "verify=False" *kwarg.

For example:
In [32]: project = Project(api_url,api_key) SSLError Traceback (most recent call last) ... --> 106 r = post(self.url, data=self.payload, **kwargs) ... SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
But if "verify=False" directly from Requests:
In [7]: requests.post(api_url, verify=False) Out[7]: <Response [501]>
Any thoughts on the best way to ignore ssl verification? It looks like "RCRequest.execute()" can take Requests **kwargs, but not directly from the Project object.

The REDCap API seems to have a type=eav parameter.

I can't figure out how to get at it from the current PyCap API.

I'm inclined to add support for it in import_records()

Something like…

from my_module import awesome_function
project = Project(URL, TOKEN)
df = project.export_records(format='df')
df['computed_field'] = df.apply(awesome_function, axis=1)
project.import_records(df)

This assumes computed_field is a valid field in the project.

@fonnesbeck, @nicholsn: would you use it?

It may be useful to retrieve the version of REDCap associated with a project to drive the way data is retrieved via PyCap. With the move to REDCap 6.x record IDs are not automatically exported for all forms. Specifically, if any forms are named, the API will not return the study ID unless specifically requested.

This would be useful in controlling the exposure of other API parameters that are only available in later versions.

I am trying to connect to a project on Vanderbilt's Redcap server, but am unable to initialize a project. It looks like filter_metadata is expecting self.metadata to be iterable when it is not:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-c75a1b7f0fbb> in <module>()
      3 api_key = <my api key here>
      4 
----> 5 jordan_project = Project(api_url, api_key)

/Library/Python/2.7/site-packages/redcap/project.pyc in __init__(self, url, token, name)
     23 
     24         self.metadata = self.__md()
---> 25         self.field_names = self.filter_metadata('field_name')
     26         # we'll use the first field as the default id for each row
     27         self.def_field = self.field_names[0]

/Library/Python/2.7/site-packages/redcap/project.pyc in filter_metadata(self, key)
     68             A known key in the metadata structure
     69         """
---> 70         filtered = [field[key] for field in self.metadata if key in field]
     71         if len(filtered) == 0:
     72             raise KeyError("Key not found in metadata")

TypeError: 'instancemethod' object is not iterable

Running version 0.6.1 on Python 2.7.2 under OSX 10.8.2.

I am exporting data from one of our REDCap servers. On the statement: p = Project(URL, API_KEY), I get a message "key not found in metadata". This is only happening on one server - it works on my other REDCap instances. I also used the PHP version of the API published by Vandy and it worked.

Hi Scott et al,

Thanks for your great work on PyCap.

I am using it to pull subjects from a couple different trial arms into an external DB, and it is working flawlessly and saving us tons of labor hours. However, now I need my automated surveys within Redcap to send out 90 days after one specific form has been completed. I have the calculation inside my RedCap 'Automated Invitations', but it only tests the automation logic on a record save action. I don't want to mess with importing recs into RedCap or anything like that, I just want to save all the records at once, as they are in place, everyday (cron job of course), in order trigger the form to be sent for appropriate records. I just know I must be missing something simple (probably really simple), but I just don't see it in the PyCap API or documentation...but I would have to imagine that this is a much desired functionality since clearly it's a design flaw in RedCap itself.

Your help is very much appreciated. If you point me in the right direction, I will gladly contribute to documentation to make it better.

Best,

~mc

@aaron0browne I recently gave PyCap a DOI so it can be cited. Are you ok with your affiliation I put on the README & intro page of the docs?

Sorry @sburns if this is abusing the purpose of GitHub issues, but I couldn't find your email online. I was curious if you were going to Park City UT in a few days? I'm giving a presentation on Monday, and I reference PyCap a few times.

In the Read the Docs documentation, a function is provided which performs chunked records export (see http://pycap.readthedocs.io/en/latest/deep.html#exporting-data). This functionality is very important at least for our project.

For longitudinal projects with many events, the chucked_export() function will pull records in duplicate if a certain record ID lies in rows (multiple rows for multiple events) which cross the edge of a chunk.

E.g. chunk_size = 500
500th row: "record_id_234" <- row for "event 1"
501st row: "record_id_234" <- row for "event 2"

In this case, all rows for "record_id_234" will be pulled in the first and second chunks, duplicating the rows.

If you do a manual REDCap export, you have the choice of "labels" or "raw", which affects the column headings, e.g. a choice between "More than one race (choice='White')" and "race_multi___1", for a column for a dropdown-type field whose name is "race_multi" and whose label is "More than one race".

The PyCap documentation claims:

# Same data, but keys will the field labels
data = project.export_records(raw_or_label='label')

However, in the code, it suggests that this option actually controls the values, not the keys, and I saw no difference in the field keys.

Also, the code made it sound like the event_name option might control the use of field labels as keys, but I didn't see that it affected the returned results at all. (My data is in a "classic" repository database project, BTW).

I was hoping to reproduce the effects of a manual export with labels, but it seems that will take a little work (to apply labels, order the fields, etc).

Could someone please help me out? I am having issues importing a file for my study. I must be missing something.

Let's face it, the folks at requests are much more forward thinking than I.

Hi all, very sorry for posting this here, but I'm having a hard time finding a RedCAP forum accessible to end users. My question is whether it's possible to grant and change permissions for a single person on multiple projects. I have a couple hundred projects, and need to grant one person access to all of them. Is there an easy way to do this, and if not, could it be scripted?

Thanks!

@sburns, Consider if there's something you want to add to this thread
https://groups.google.com/forum/#!topic/project-redcap/p-9v51TbK6o

my call to Project() used to work, but we recently upgraded to REDCap 6.4.4 and now I get:

  File ".../python2.6/site-packages/redcap/project.py", line 52, in __init__
    arm_nums = tuple([a['arm_num'] for a in arm_data])
TypeError: string indices must be integers

a debug session shows that for class projects, the REDCap API succeeds where the code expects an error; then the code goes to get the arm info and the REDCap API says "no, silly!"

.../python2.6/site-packages/redcap/project.py(46)__init__()
-> if 'error' in ev_data:
(Pdb) p ev_data
[{u'day_offset': u'0', u'event_name': u'Event 1', u'arm_num': u'1', u'offset_min': u'0', u'unique_event_name': u'event_1_arm_1', u'offset_max': u'0'}]
(Pdb) n
> .../python2.6/site-packages/redcap/project.py(51)__init__()
-> events = ev_data
(Pdb) n
> .../python2.6/site-packages/redcap/project.py(52)__init__()
-> arm_data = self._call_api(self.__basepl('arm'), 'exp_arm')[0]
(Pdb) n
> .../python2.6/site-packages/redcap/project.py(53)__init__()
-> arm_nums = tuple([a['arm_num'] for a in arm_data])
(Pdb) p arm_data
{u'error': u'You cannot export arms for classic projects'}

obligatory version info:

> pip freeze | grep PyCap
PyCap==1.0

We've got a project with >10K records. When we try to export just the record_id field from this project, we get:

REDCap ran out of server memory. The request cannot be processed. Please try importing/exporting a smaller amount of data.

I see a filter parameter when exporting records, but I'm struggling to figure out how to use that for pagination in a general-purpose way.

Suggestions?

Hey everyone, after having a great discussion with @pbchase, I've decided to move PyCap to a new organization. I'd like it to be a place where people can put interesting projects based against REDCap. I actually don't want it to be python focused ( 😄 @wibeasley) but rather language agnostic.

Obviously some projects have certain ownership that must be maintained, either at the laboratory or university level. I'm not asking for private code but rather fun projects that have either scratched an itch or you depend on day in and day out that should be shepherded by a group of like-minded individuals.

I just need a name! Thoughts?

I will definitely put everyone who has contributed either in code or otherwise.

@dckc @aaron0browne @tjrivera @nicholsn @bud42 @byvernault

To do list

• New ReadTheDocs project
• Update TravisCI (since github forwards org changes maybe that'll just work?)

This function just doesn't work.

Example working code with requests:

from requests import post
payload = dict(token=token, content='file', action='import', record=record, field='report_file', returnFormat='json')
filed = {'file': ('test.html', io.open('test.html', mode='r', encoding='utf-8'))}
r = post(url, data=payload, files=filed)

It doesn't help when there's no response from REDCap, but its my issue because this code works and project.import_file doesn't.

When following the tutorial for updating a single field there are two problems.

1. the "redcap_event_name" value is set to the readable string rather than the unique id.
2. the response for a single field includes a "redcap_survey_identifier" and "redcap_survey_timestamp" in the response, but they are blank.

data = project.export_records(fields=['hamd_18'])
print data
[{u'hamd_18': u'1', 
u'participant_id': u'1', 
u'redcap_event_name': u'Visit 1', 
u'redcap_survey_identifier': u'', 
u'redcap_survey_timestamp': u''}]

data[0]['hamd_18'] = 2
response = project.import_records(data)
print response
{u'error': u'The following fields were not found in the project',
 u'fields': [u'redcap_survey_identifier', u'redcap_survey_timestamp']}

If I remove "redcap_survey_identifier" and "redcap_survey_timestamp" from the dict, I then get this error:

{u'error': u'One or more of the event names you entered was not valid'}

Finally, after I set the "redcap_event_name" to the unique id, I am able to complete the update.

data[0]['redcap_event_name'] = 'visit_1_arm_1'
response = project.import_records(data)
print response
{u'count': 1}

Is it possible to have the initial response populated only with the required (and correct id) information needed for an update?

Hello everybody,

I'm importing data in our REDCap server.
For the third time, my ip address was banned from REDCap.
However, I'm importing just a few data (like 20 records).
Did someone had the same issue ?
Thanks

I used import_records without concern for what it returned; something like:

project.import_records(data[offset:offset + chunk_size])

The result in REDCap wasn't what I expected. So I started looking at the return value, and where I expected a count, I found

{
  u'error': u'The following fields were not found in the project',
  u'fields': [u'malignant_n_36_code', ...]
}

I think import_records should raise an exception in this case.

It's nice to automatically have the metadata loaded for a project during instantiation, especially when working with projects interactively.

However, it's slow and a wasteful API call when calling from libraries/applications that don't make use of the metadata structures.

This was a stupid mistake early on. The obj format value in export_* functions should mirror the REDCap API parameters which in this case is json.

This is backwards incompatible and should only ship in 1.0

Are there any plans to upgrade PyCap to include delete for records? If not, I will take a stab at it. It shouldn't take long.

Cheers from Steve and the VUIIS folks

Exported data from a longitudinal project has non-unique values in the primary field column. Specifically, there will be as many rows for a record as events have data for that record. pandas.read_csv doesn't break on these projects currently because it doesn't enforce uniqueness for the index, but a truly unique index is [project.def_field, 'redcap_event_name'].

Thus, this feature will return a hierarchically-indexed df when exporting from a longitudinal project, which I think is semantically correct.

Do you agree @nicholsn?

Prior to 6.x setting the raw_or_label parameter to true would export checkbox field's labels as their value.

In 6.x setting raw_or_label to true simply changes the raw export value of checkbox fields from "0" to "Unchecked" and "1" to "Checked".

To export records with checkbox values labeled as before one must pass exportCheckboxLabel and set the value to true. I'd like to propose potentially adding checkbox_labels=False as a keyword arg to Project.export_records to support this functionality. I'd be happy to start work if you think it would be a welcome addition.

In the past I was able to access the form status from a full redcap export. now that field is only available if i only pull that form specifically.

for example if i have a form called test

export_records(forms=['test'])
will result in rows with the variable 'test_complete' in them
however
export_records()
will not.

Is this intended behaviour?

Scott, in Issue #54, you lament there is no API version. What of the Export REDCap Version API? Would this be helpful if we implemented it? It's not an API version number, but it might help PyCap determine the appropriate response.

Export REDCap Version

Description

This method returns the current REDCap version number as plain text (e.g., 4.13.18, 5.12.2, 6.0.0).

URL

https://host.domain/redcap/api/

Supported Request Methods

POST

Parameters (case sensitive)

Required

• token
  the API token specific to your REDCap project and username (each token is unique to each user for each project) - See the section above for obtaining a token for a given project
• content
  version

Returns:

the current REDCap version number (three numbers delimited with two periods) as plain text - e.g., 4.13.18, 5.12.2, 6.0.0

A recent issue has arisen when using PyCap to export records from REDCap in which at least one field becomes mangled from its original form in the database. For example, the Study_ID field in the screen shot below in REDCap is a series of 4 digits separated by dashes:

However, when I export these into Python via export_records, they are somehow encoded into an alphanumeric sequence:

redcap.__version__ reports 1.0 as the PyCap version. Running in Python 2.7.6 on OS X 10.9.4.

New Things

Hello all,

So I've started going through the PyCap code and comparing it to the changes that have been made to the REDCap API post version 6.0.0. While most things shouldn't be too difficult to add in, there were two major changes that might call for restructuring the Project class.

1. In version 6.5.0 a "Export Project Information" API method was added that would make initializing a project object cleaner and more descriptive, it returns a large field of values with only the standard [token, content, format] input required.

   • This method returns the following:

   project_id, project_title, creation_time, production_time, in_production, project_language, purpose, purpose_other, project_notes, custom_record_label, secondary_unique_field, is_longitudinal, surveys_enabled, scheduling_enabled, record_autonumbering_enabled, randomization_enabled, ddp_enabled, project_irb_number, project_grant_number, project_pi_firstname, project_pi_lastname, display_today_now_button
   

   • This would deprecate a lot of the manual work used in PyCap to find things like is_longitudinal

2. In version 6.11.0 a "Create Project" API method was added that would allow for creating new projects directly in PyCap as long as the user has a 64-character supertoken.

   • All input parameters:

   project_title, purpose, purpose_other, project_notes, is_longitudinal, surveys_enabled, record_autonumbering_enabled
   

   • Returns a 32-character API token for continued access to project

For Consideration

Problems

Here are some of the potential problems in adding the above methods.

• Both methods listed above used the same content = 'project' parameter when talking to the REDCap API, the distinguishing factor being the length of the API token.
• If changes were made to the Project class, there are 3 major version changes to account for and handle.
  • Post version 6.0.0 when Export REDCap Version was added
  • Post version 6.5.0 when Export Project Information was added
  • Post version 6.11.0 when Create Project was added
• Beyond this a number of new API methods and parameters were added throughout REDCap 6.X. which might encourage tackling things in a more adaptive, version-specific manner.

Options

I will admit to not having prior experience working with REDCap, so it's quite likely I've overlooked some things, but here are the potential options as I see them. From there, I'll leave it to the community to decide what path makes the most sense when it comes to tackling these issues.

• Add Export Project Information and Create Project as fully separate methods under the Project object and handle version issues individually.
• Rewrite the Project object's initialization function such that it starts with a bit of logic to the extent of:

try:
  version_num = redcap.Get_Version( ) # attempt to get redcap version
except:
  log.warning("REDCAP < 6.0.0 not fully supported, use at own risk")
  version_num = 0.0.0
if version_num >= 6.11.0 and len(api_token) == 64:
  redcap.Create_Project( ) # Create new project in REDCap
else if version_num >= 6.5.0:
  redcap.Export_Project_Information( ) # initialize using modern api methods
else:
  # initialize using current methods

• Beyond that, I think there may be call to introduce a more complicated version checking structure to check for existence of API methods and the parameters available at each version. I haven't fleshed out quite how that would work yet, but I imagine you'd build up a dictionary of parameters for for each method with the version # as the key, which once parsed through would give you the available parameters at version x.

A lot of this boils down to how much the PyCap's base can afford to be changed, how much code is dependent on it in it's current form, and what versions of REDCap people are actually working with now, i.e. do we need to support REDCap before version 6.0.0?

Thoughts, comments, suggestions?

-Dax

Redcap has added new methods, we need to keep up (I've included screenshots of the documentation for each)

• Export PDF file of Data Collection Instruments
  
• Export a Survey Link for a Participant
  
• Export a Survey Queue Link for a Participant
  
• Export a Survey Return Code for a Participant
  
• Export a Survey Participant List
  
• Export List of Export Field Names
  

Projects should be able to be initialized without making calls out to the REDCap API.

There are situations where I'd like to make programmatic calls to a project ad-hoc but not have to pull metadata for the project on every initialization. A proposed solution would be to provide an additional keyword argument on the Project object lazy which would default to False (to preserve current behavior). Setting lazy to True would result in the Project being initialized without the metadata being pulled. Users can still export/import records and metadata, it would just be done on demand.

Pull request to follow.

From @aaron0browne

PS Incidentally, I also realized by chance that if you pass an incorrect url while instantiating a new Project, the first error that gets thrown is a KeyError...which is not particularly informative. Thought you might want to know my experience with this particular edge case.

Basically, running the export metadata API call in instantiation with a bad URL should throw something else than the KeyError that happens when making Project.field_names

Judging by the validate method on the RCRequest object, it seems like there is support for getting the formEventMapping content from the REDCap API, but I can't figure out how to access this functionality from a Project object.

Am I missing something?

Hi there,

First of all, thank you for creating PyCap.

I'm helping set up a REDCap project where we want to run a nightly batch job that processes all the recently completed records by running them through some calculations, creating a PDF with the results and uploading them to REDCap. And that works fine!

However, if I try to export a file, I get an empty string back. For example:

fobj = open('report.pdf', 'rb')
response = project.import_file(record='1', field='po_report', fname='report.pdf', fobj=fobj)
file_content, headers = project.export_file(record='1', field='po_report')
headers
# returns {'name': 'report.pdf'}
file_content
# returns ''
len(file_content)
# returns 0

This is with REDCap 5.5.8, PyCap 1.0, Python 2.7.6 and Ubuntu 14.04

Note: I just tested it using curl, and this works fine:

curl -F token=${TOKEN}  -F content=file -F action=export -F record=1 -F field=po_report  ${SERVICE} > report_ex.pdf

Oy. Lesson learned, need to slow down when 1) getting back into old project, 2) at night when brain is a bit tired.

I've verified that my key works with the following code

from requests import post
# Two constants we'll use throughout
TOKEN = 'SECRET_KEY'
URL = 'https://redcap.vanderbilt.edu/api/'

payload = {'token': TOKEN, 'format': 'json', 'content': 'metadata'}

response = post(URL, data=payload)
print response.status_code
200

However, I'm receiving a KeyError when trying to instantiate the Project object. Can you confirm if PyCap has been tested for REDCap version >= 7.4.9?

>>> from redcap import Project
>>> url = 'https://redcapproduction.example.com/api/'
>>> key = 'API_KEY'
>>> p = Project(url, key)
/miniconda3/envs/redcap-env/lib/python3.6/site-packages/redcap/project.py:107: UserWarning: Version information not available for this REDCap instance
  warnings.warn('Version information not available for this REDCap instance')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/miniconda3/envs/redcap-env/lib/python3.6/site-packages/redcap/project.py", line 53, in __init__
    self.configure()
  File "/miniconda3/envs/redcap-env/lib/python3.6/site-packages/redcap/project.py", line 64, in configure
    self.field_names = self.filter_metadata('field_name')
  File "/miniconda3/envs/redcap-env/lib/python3.6/site-packages/redcap/project.py", line 142, in filter_metadata
    raise KeyError("Key not found in metadata")
KeyError: 'Key not found in metadata'

"Note, no matter which fields or forms are requested, the project.def_field key will always be in the returned dicts."

This no longer appears to be true in the latest version of Redcap

Unique identifiers declared by project.def_field are no longer present in the recordset for project subforms that do not explicitly contain the 'defining field'

e.g., how Continuum builds blaze

I cannot make the file_import to work, working with a server running REDCap 7.4.23.
I followed the example from http://pycap.readthedocs.io/en/latest/deep.html#working-with-files, but it fails with HTTPError: 400 Client Error: Bad Request for url.
Interestingly, it fails also from REDCap's API playground, but I managed to import the file using curl from the command line (in linux).

I have just started with REDCap, so I do not know how this worked on earlier versions

I have also noticed that the REDCap documentation for the import API does not have a parameter for file name (and, indeed, it gets ignored when using curl), but has an additional parameter repeat_instance, needed to upload a file to repeating instruments.

I got {} back from project.import_records(...). I went in with the debugger and found that the HTTP response was a 500 error: "ERROR: REDCap ran out of memory!"

It looks like raise_for_status ignores errors in case self.type is 'imp_record'. I don't see why.

I found import_records() have eav type supported.
It seems like export_records() doesn't have eav supported. Please correct me if I'm wrong.
I'm wondering if I can add support to this,

Hi,
Thanks for this work. Is there any examples for using the filter() function?

I can't determine what the expected object query is. Also do you know of any more documentation about the filterlogic strings expected by redcap are?

Thanks.

With REDCap 5.x and higher, data access group and survey timestamps are no longer exported by default. Instead, these require additional request header fields like so (from the php API examples on the RC Wiki):

'exportSurveyFields'=>'false'
'exportDataAccessGroups'=>'false'

PyCap cannot currently set these, so the fields in question are not currently exportable.

Thanks!

Torsten

Theoretically, you can quickly backup an entire redcap project through the API. You can download:

• The data dictionary (project.export_metadata)
• The data (project.export_records)
• The users (project.export_users)
• Form-Event mappings (project.export_fem)
• All files (export all records, filter metadata for file fields, loop over subjects and fields and run project.export_file)

It could take a bit of time, but project.backup('/path/to/backup/directory') could be pretty useful. Put it in cron (or some other scheduler) and you'll always have an up-to-date backup of your project.