vuiis / dax Goto Github PK

There are missing imports. Just posting this so no one installs it and wonders why it doesn't work. I've committed to master a fix

Two things I'm getting at here.

1: Could we start to incorporate the logger level in the project settings file. This way we could control logging on a project-by-project basis to expedite debugging.

2: Could we configure the logging via some sort of config file (outside of dax_settings) so that we can change globally on the fly? This way we don't have to reinstall and we can get depper (or less) logging quickly

Looks like this was broken in recent changes.

Hi,
the utility XnatCheckLogin use the XNAT_HOST variable without trimming the slash "/" for the address.

Let's see this example:

INFO: Trying to open a connection with XNAT with your login...
 --> error: Wrong login.
 *Try typing them again:
  --> XNAT_HOST: http://127.0.0.1:9090/
  --> XNAT_USER: admin  
  --> XNAT_PASS:
  --> error: Wrong login.

A 'nc' session opened to the 9090 door receveid:

$ nc -l 9090
GET //data/JSESSION HTTP/1.1
Host: 127.0.0.1:9090
Authorization: Basic YWRtaW46YWRtaW4=

So the GET // are an error to fix.
Regards

We found an interesting issue with using the current SCREEN=$$$$ in the run_spider and template files. On our SGE grid we were getting very high PIDs. These resulted in a PID of > 10 digits. These silently error out in xvfb-run. The proposed solution (that has been working for several days) is to do this

SCREEN=$$$$
SCREEN=${SCREEN:0:8}

(Thanks Andrew for helping out with this).

Are there any objections to a direct commit to master for this since it currently has the possibility to torch silently?

Steve

Traceback (most recent call last):
File "/home/masispider/masimatlab/trunk/xnatspiders/spiders/Spider_dtiQA_v3_1_0.py", line 383, in
arguments=init_dtiQA(args)
File "/home/masispider/masimatlab/trunk/xnatspiders/spiders/Spider_dtiQA_v3_1_0.py", line 49, in init_dtiQA
arguments['jobdir'] = XnatUtils.makedir(os.path.abspath(options.tempdir))
File "/home/masispider/.local/lib/python2.7/site-packages/dax-0.4.0-py2.7.egg/dax/XnatUtils.py", line 1533, in makedir
os.mkdir(directory)
OSError: [Errno 17] File exists: '/tmp/XXXXXX-x-XXXXXXXX-x-XXXXXXX-x-DTI2-x-dtiQA_v3'

dax_build is the culprit here with lots of unhelpful database errors causing stuff to die consistently. Are there any objections to surround this process in a try/catch to catch all of pyxnat.core.errors?

We would like to add a new feature but now that Benj is travelling the world it would be great if we could get a bit of guidance from you.

Here is a typical example. Let's say we have a scan "1" and an assessor that runs on this scan "proj-x-sub-x-sess-x-1-x-proc1". We now have another assessor that takes one output from the first assessor as an input. It leads to "proj-x-sub-x-sess-x-proc2". We start here having discrepancy as one assessor is scan based and the other is session based.

The real problem for us starts when we have more than one input scan going through the first process. If I have two scans, "1" and "2", they both have their own assessor, which is the behavior we want. The second process now grabs data from both proc1 assessors and combine them into a single assessor. Ideally, we would like to have:

scans:
- 1
- 2
assessors:
- proj-x-sub-x-sess-x-1-x-proc1
- proj-x-sub-x-sess-x-2-x-proc1
- proj-x-sub-x-sess-x-1-x-proc2 or  proj-x-sub-x-sess-x-1-x-proc1-x-proc2
- proj-x-sub-x-sess-x-2-x-proc2 or  proj-x-sub-x-sess-x-2-x-proc1-x-proc2

Does it makes sense to you? Is there already something in place to replicate this behavior and did we miss it? If not, could you give us a few pointers on where to start, especially to @bariskanber, who is going to have a look at this one.

In case, it is of any use and we should have written it differently, below is the yalm processor associated with the second assessor:

---
inputs:
  default:
    spider_path: /home/dax/Xnat-management/ucl_processing/pipelines/BrainTivFromGIF/v1.0.0/Spider_BrainTivFromGIF_v1_0_0.py
    working_dir: /scratch0/dax/
    nipype_exe: perform_brain_tiv_from_gif.py
    env_source: /share/apps/cmic/NiftyPipe/v2.0/setup_v2.0.sh
    omp: 1
  xnat:
    assessors:
      - assessor1:
        proctypes: GIF_Parcellation_v3
        needs_qc: False
        resources:
          - resource: SEG
            varname: seg
command: python {spider_path} --exe {nipype_exe} --seg {seg}
attrs:
  suffix:
  xsitype: proc:genProcData
  walltime: 01:00:00
  memory: 4096
  ppn: 1
  type: session
  

It would nice to be able to filter the assessors downloaded based on scan type. Maybe there's a way to accomplish this already? Someone just asked me if it was possible and I didn't have an answer.

We have a lot of new features. Let's close as much as we can and release?

OK, I'm bringing this one back. I am currently attempting to use Xnatmirror to copy projects from my 1.6.3 production, over to a 1.6.5 VM, but I get a refused connection error telling me to check credentials. The credentials are definitely correct (I have tried several times - and in any case the credentials are written in plaintext) and I have verified that the user credentials are the owner of the test project.

One thing of note here is that the credentials are authenticated via LDAP, which I suspect might be causing the issue. Thoughts?

I think we should add an --ignore option to Xnatinfo. Some projects have tens of thousands of scans and the massive dump of unusable is generally not helpful in these cases (especially when there are hundres or thousands of different scan variants as is the case with hospital data).

Sphinx is failing as a result of the ~/.dax_settings.ini file not existing. Should we modify the setup to have a "basic" config file?

get_good_cassr returns assessors with NEED_INPUTS. I don't think this should be the case? Thoughts?

I believe Ben was working on a script that checked the flag files for ones that were active for a long period of time. Was this ever completed? It would be nice for situations when everything stops like this weekend.

http://dax.readthedocs.io/en/spider_release/dax.html#dax-spiders-spider-class

What if we implement this? https://pypi.python.org/pypi/coloredlogs

Xnatcheck -p ABIDE --filters proctype=fMRI_Connectivity_PreProcess_Scan_v1 1> 1.txt 2> 2.txt

For Xnatinfo, and more generally XnatUtils methods, all returns are 0 for list_project_* methods (scans, subjects, assessors etc). This recently was noticed with the addition of NIRS data as the "HD_Session" type in XNAT.

if not assessor.exists():

        if self.atype == DEFAULT_DATATYPE.lower():

            kwargs = {'%s/fsversion' % DEFAULT_DATATYPE.lower(): '0'}

            assessor.create(assessors=DEFAULT_DATATYPE.lower(), **kwargs)

        else:

            assessor.create(assessors=self.atype)

Shouldn't that be DEFAULT_FS_DATATYPE? I'm trying a patch now

Hi all,

I would like to make two modifications in the job template files and related code:

• Similarly to export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS={job_ppn}, I would like to use the same variable to set OMP_NUM_THREADS. It basically does the same but is not ITK specific.
• Adding an attribute to specify an environment file to source prior to launching the job_cmds.

The regex isn't working appropriately. T1WMT is getting matched to T1 when it shouldn't resulting in Multi_Atlas running on MT data and not just T1 and MPRAGE data. I'll work on a fix. We need to update this one sooner rather than later...

We're trying to split the redcap output into multiple databases. It would be handy to be able to have multiple instances of the send to redcap module kind of like how you can separate processors by semi colons.

Hey all - I'm trying to follow the Dax install direcitons, but I'm running into an error that dax_setup has not been run. When I try to run it, though, there a number of questions that I don't know answers to, including some questions about xsitypes and cluster settings.

I'm just trying to use the freesurfer editing scripts (fsdownload, fsview and fsupload) to edit some recons. Any tips for getting past this step?

http://dax.readthedocs.io/en/latest/dax_installation.html#install-dax

right now ~/RESULTS_XNAT_SPIDER gets generated but on some OSs it doesn't like the ~ so we should os.path.expanduser like home

Dax fails to install on some older cent boxes. I'll post info from uname once I get back I'm the lab. I'll try to get some sort of wrapper for a lot of the missing dependencies working as soon as pragmatic. We've seen similar sorts of fun recently with other projects on cent boxes

Hi guys,

I think we should probably have an option fs_datatype_on in dax_settings. Some other universities might want to just use proc:genProcData and not fs:fsData.

Also, I am adding some lines of code to check that the datatypes are installed on Xnat before DAX do anything. It will raise an error. For listing assessors, it will just return an empty list and print a warning to the terminal.

I am opening this issue just to remember to do that.
It's going to be in my branch on new options for Xnat tools.

Cheers,

Ben

Hi,

I just found out that Xnatmirror will raise an error if the subject is shared to a project probably because the shared project doesn't exist on the other XNAT instance. See below in the subject.xml.

<xnat:sharing>
<xnat:share label="293IV5" project="DIAN_SP" subject_ID="CMIC_XNAT_S02388">

</xnat:share>
</xnat:sharing>

I will do some test to see if removing the share tags allows us to still mirror those subjects.

Kind Regards,

I followed the instructions on the wiki for adding the freesurfer and proc modules, but I get a build error:

    [exec]     [javac] Compiling 1418 source files to /xnat/deployments/xnat/target/classes
    [exec] /xnat/deployments/xnat/target/src/org/nrg/xdat/om/base/BaseFsAparcregionanalysisRegion.java:18: cannot find symbol
    [exec] symbol: class AutoFsAparcregionanalysisRegion
    [exec] public abstract class BaseFsAparcregionanalysisRegion extends AutoFsAparcregionanalysisRegion {
    [exec]                                                               ^
    [exec] Note: Some input files use or override a deprecated API.
    [exec] Note: Recompile with -Xlint:deprecation for details.
    [exec] Note: Some input files use unchecked or unsafe operations.
    [exec] Note: Recompile with -Xlint:unchecked for details.
    [exec] 1 error
    [exec] 
    [exec] BUILD FAILED
    [exec] File...... /xnat/plugin-resources/cache/maven-java-plugin-1.5/plugin.jelly
    [exec] Element... ant:javac
    [exec] Line...... 63
    [exec] Column.... 48
    [exec] Compile failed; see the compiler error output for details.
    [exec] Total time: 8 seconds
    [exec] Finished at: Thu Sep 22 10:03:21 CDT 2016
    [exec] 
    [exec] [ERROR] Result: 70
BUILD SUCCESSFUL
Total time: 45 seconds
Finished at: Thu Sep 22 10:03:21 CDT 2016

I tried a similar install on my VM and it worked, so something on the production XNAT is different, but unfortunately the guy who configured it is gone. I would appreciate any advice on how to proceed from here... I backed up the SQL database manually before the attempt, and the images themselves are backed up nightly.

I really like the functionality of the AssessorHandler. I feel like it makes my code significantly more readable. I was wondering if we have the inverse function in the works? I.e. I have a project, subjects, session, and scan, make me the string or the same class basically.

Not necessarily an issue, but kind of. I get a lot of requests why the status is NEED_INPUTS. What are everyone's thoughts on setting this in the QC status (like we do for NO_DATA) rather than grepping through the build log? This should alleviate unnecessary emails (I think; assuming we set appropriate and intelligent debug info like "No DTI scan").

I recently installed dax from the HEAD and it does not look like the version of requests works. I get the error message: pkg_resources.UnknownExtra: requests 2.2.1 has no such extra feature 'security'

Due to something brilliant that I did recently we realized that we need to reverse-sort by date the assessors in the upload directory. This way when the pipe breaks, we can pick up with the OLDEST assessor first so we don't run into a recurring non-upload situation (which I caused :]). I'll handle this one

we should check for really old lock files and notify the user to consider deleting them.

Hi,

We talked about this (Steve, Brian and I) and we agreed that we need to have something to test the processors/modules/settings/spiders etc...

This issue is just to remember that we need to work on that.

Two points:

1. We need to implement an options to limit the number of assessor when testing a processor on a new project/dataset, something like --limit 10 for processors where dax_build generates 10 assessors that will run and if we like the outputs, we can remove the limit and it will run on everything.

2. Create a way to test the code in different files (processors/modules/spiders/settings).

   1. spiders: probably straightforward, add a local variable that will not run the finish() function or something like --skip_finish.
   2. processors: probably something like an executable where you give a processor file and a project ID from XNAT. From there, it will run dax_build on a project but for just few sessions (2-5), run dax_launch and print the .pbs file to see that everything is fine.
      E.G: dax_test -p TEST --processor /Users/byvernault/Processor_GIF_Parcellation.py
   3. modules run dax_build on a project ID from XNAT but for just few sessions (2-5)
      E.G: dax_test -p TEST --module /Users/byvernault/Module_dcm2nii.py
   4. settings: display all informations from the settings that will be used by dax in a way that help the user to check that everything if fine.
      E.G: dax_test --settings /Users/byvernault/Settings_VUIIS.py

What do you think?

Cheers,

Ben

I am using Xnatmirror to migrate to a newer version of XNAT. However the current release of dax throws an error when I try to use it. (I actually tried dax 4 5 6 and 7)

I got around it by reverting to an earlier version and commenting out the block that calls XnatUtils.has_dax_datatypes. Also left SKIPPING_PROC_DATA = True uncommented.

So the files appear to be copying now, but I just wanted to let y'all know it doesn't work "out of the box". The more recent versions gave me some kind of an error with Tkinter, and another version also threw errors that looked like it was assuming I was on a cluster (it was looking for some head node configuration or something)

Also.... the Xnatmirror source code under the "current" github master does not even make it to that code block because there is something wrong with the argument parser. Doing a pip --upgrade gives us a version where that works but iirc throws the cluster errors

Hi guys,

I was thinking about doing a logo for the package since we start using it in different sites and I did several presentation here on it.

I created a simple logo so far. We can do a contest and then choose the best logo. You can improve the one I am posting if you want. :)

Cheers,

Ben

Am I doing something wrong?

When I run dax_launch ~/settings.py --sessions XXXXX it doesn't find the session if there are multiple projects in the settings file. dax_build works as expected (iterates through projects until it finds the session).

From Baxter:

DAX bug report: Xnatreport does not double-quote comma-containing fields in the produced CSV file. This matters for the scan's Note field where commas may be present.

SpiderProcessHandler attempts to access XNAT. If it cannot the class throws an exception and the job gets marked as failed in its output status file and thus this is the status when the data get uploaded to xnat even though in all likelihood the process did not fail.

Hello,

I have been talking with @bud42 two weeks ago about DAX and the fact that the project starts to have a lot of files for the documentation that are related to specific spiders (but no code present) as well as a lot of branches for it that are not active.

Could we move all those stuff into a new project for pipelines like VUIIS_pipelines and have the documentations moved to the new project?

I think DAX should not be attached to any specific pipeline but should point out in his wiki/documentation where people can find pipelines developed via DAX. I have one project for UCL pipelines (that I need to update with AutoSpider) in https://github.com/byvernault/ucl_processing . It contains the spiders/modules/processors that I use at UCL. The same thing could be done for Vanderbilt. What do you think?

Kind Regards,

Ben

Not sure if this is the right place, but

I have 75 subjects that have identifying info in a project. I can anonymize them via XNAT now that I have the project anonymization script, but to do so I have to rename each session (and then name it back) causing the project anonymization script to run.

Is there a solution that can anonymize all my scans at once?

We need to clean this piece up. Errors need to be actual errors. For example, just because a download directory doesn't exist, doesn't mean we should print an error. If that really needs to be an error, it should error. However, I think we could just make the directory. This came about as I'm updating all the docstrings for Sphix.

Thoughts?

is_assessor_same_scan_unusable will return 0 for session processors because it always includes scan id. Thus the session processor is never found.

We should add a flag for is_session_processsor=False as the default in the method so no one sees a change.

Would it be possible to incorporate regular expressions into the dax utilities like Xnatdownload? i.e. I want all assessors of the type "Heckers_*" instead of listing out all of the assessors? I can definitely implement this during the daxathon if there isn't a chance before then.

So, almost every error that comes out right now is a DataBase error because the parsing is done incorrectly for Auth errors. There are a whole host of errors that I see with some more severe than others. For example, when a malformed syntax issue is in the body of the return, it often perpetually trashes a project until fixed (usually by logging into the database and doing some manual work). I'd like to make these errors a bit more verbose as to what actually happened so we don't have to always read through the HTML response.

def is_cassessor_usable(cassr):
""" return 0 if assessor not ready or doesn't exist
return -1 if assessor failed
return 1 if ok
"""
assr_info = cassr.info()
return is_bad_qa(assr_info['qcstatus'])

Returns -1 if the assessor status is "bad". In the function get_good_cassr

def get_good_cassr(csess, proctypes):
""" return cassr list from a csess if there is a good assessor """
cassr_list = list()
for cassr in csess.assessors():
if is_cassessor_good_type(cassr, proctypes) and is_cassessor_usable(cassr):
cassr_list.append(cassr)
return cassr_list

-1 returns True in an if statement and thus returns 'bad' and 'fail' assessors as "good" ones in cassr_list

Hi guys,

I realised today something odd but it makes sense why we have this. If a session that has existing assessors is shared to a new project where you run the same spiders, it will creates a new assessors (with the new project name) on the same session.

Example:
If you have a project called TEST where you run fMRIQA_v1 for example via DAX, you will for the session s01_1 under the subject s01: TEST-x-s01-x-s01_1-x-3-x-fMRIQA_v1. Normal.
Now, if you create a new project TEST2 and you share the session s01_1 to this project and run fMRIQA_v1 via DAX as well, you will see a new assessor in your session called:
TEST2-x-s01-x-s01_1-x-3-x-fMRIQA_v1
It makes sense as well.

But since XNAT when sharing a session, just use the same location between project, it means that you will see in both projects all those assessors.

I am trying to think if we could do something to fix this behavior. Should we check if either project_id or project_id_shared assessors existed when looking at should_run()?

Cheers,

This is in dax 0.4.0 release. Just wanted to make sure it's actually fixed in the repo. If not. I'm going to apply a patch because it's broken for a project right now.

elijah@picasso:~$ bash xnatmirror_env.sh
Src XNAT Host: http://10.42.40.32
Src XNAT Proj: test_project
Dest XNAT Host: http://10.80.32.72
Dest XNAT User: root
Dest XNAT Proj: test_project

Time: 2015-11-13 11:26:35.087058 ========================================================
Getting projects...
Traceback (most recent call last):
File "/usr/local/bin/Xnatmirror", line 4, in
import('pkg_resources').run_script('dax==0.3.1', 'Xnatmirror')
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 735, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 1659, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/dax-0.3.1-py2.7.egg/EGG-INFO/scripts/Xnatmirror", line 295, in

File "/usr/lib/python2.7/dist-packages/pyxnat/core/resources.py", line 369, in create
output = self._intf._exec(create_uri, 'PUT')
File "/usr/lib/python2.7/dist-packages/pyxnat/core/interfaces.py", line 380, in _exec
catch_error(content)
File "/usr/lib/python2.7/dist-packages/pyxnat/core/errors.py", line 69, in catch_error
raise DatabaseError(error)
pyxnat.core.errors.DatabaseError: Apache Tomcat/7.0.62