dbic / handbook Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 5.0 4.06 MB

WiP: DBIC Handbook

Home Page: https://dbic-handbook.readthedocs.io

License: Creative Commons Attribution 4.0 International

Makefile 0.33% Python 12.95% HTML 48.15% Shell 38.57%

handbook's People

Stargazers

Watchers

Forkers

feilong xinmingxu andycon terrysackett asmacdo

handbook's Issues

Q: if there is no DataLad -- how to transfer

we had notes in the "cheatsheet" -- need to be brought here

also doublecheck various tools (rsync/scp) on how they deal with symlinks

Q: doubling the space with DICOMs (original and in BIDS)

"isn't that bad?"

enable (after review/setup) circleci

we inherited configuration from BIDS. It includes building and checking urls. But ATM it is not used at all -- we just rely on RTFD to render it.

Document use of ReproStim

ref: https://github.com/ReproNim/reprostim

ATM AFAIK we suspended collection of audio/videos, but it is to be resumed shortly if not already. After it is resumed, handbook should be updated to provide information on where to find the videos etc.

Invited @andycon to the @dbic/staff

Q/Discovery: anyone in DBIC relies on ACL?

The easiest way to resolve all "DataLad difficulties" and some user confusions with ACL, is to not use them. ATM ACL is used for /dartfs* volumes by default (/home and scratch are regular POSIX). Could we migrate entire /dartfs/rc/lab/D/DBIC to be POSIX?

Establish easy to use setup on MR console room iMac

should be easy for people (e.g., MR tech) to

open any file in this repo (not sure about editor. pycharm provides nice support for markdown but there might be better/leaner ones)
edit it (with live preview, like pycharm does)
rerender website (could be setup so it would be done automagically, mkdocs serve with a bookmark to the url presenting rendered version)
commit
submit a PR

Instructions on how to use this setup should go into CONTRIBUTING.md

@andycon - interested to take care about this one?

provide more instructions on dups (especially renaming) handling with datalad

Add instructions for using Jupyter Notebook on Discovery

I am starting work on adding instructions to use ssh tunnel to launch a Jupiter server as a submitted job on Discovery. This allows users to interface with the Jupyter notebook from their browser.

Talk to DBIC labs/users on moving the entire /dartfs/rc/lab/D/DBIC under POSIX

or it could be alternatively only subpart(s) of it. Some labs (I was told) found a peace of mind already with ACLs

git-annex way to freeze/thaw might need fixing

of tremendous interest for @jungheejung and @Michael-Sun.

Awhile back we established setup for how to work with discovery filesystem which is described in https://dbic-handbook.readthedocs.io/en/latest/mri/dataaccess.html#discovery-filesystem on how to use our custom freeze/thaw scripts.

to the date it seems to work fine if we use that a little older git-annex

[d31548v@discovery7 fmriprep]$ which git-annex
/dartfs/rc/lab/D/DBIC/DBIC/archive/git-annex/usr/lib/git-annex.linux/git-annex

[d31548v@discovery7 fmriprep]$ /dartfs/rc/lab/D/DBIC/DBIC/archive/git-annex/usr/lib/git-annex.linux/git-annex version
git-annex version: 10.20220127+git47-g9f9b1488e-1~ndall+1

[d31548v@discovery7 test]$ git init
Initialized empty Git repository in /dartfs-hpc/rc/lab/C/CANlab/labdata/data/spacetop_data/test/.git/
[d31548v@discovery7 test]$ git annex init
init  ok
(recording state in git...)
[d31548v@discovery7 test]$ ls
[d31548v@discovery7 test]$ ls
[d31548v@discovery7 test]$ echo 123 > 123
[d31548v@discovery7 test]$ git annex add 123
add 123 
ok                                
(recording state in git...)
g[d31548v@discovery7 test]$ git commit -m 123 123
[master (root-commit) b0232a6] 123
 1 file changed, 1 insertion(+)
 create mode 120000 123
[d31548v@discovery7 test]$ ls -lta
total 120
drwxrwx---  9 d31548v rc-CANlab-admin 293 Feb 17 13:57 .git
drwxrwx---  3 d31548v rc-CANlab-admin  43 Feb 17 13:57 .
lrwxrwx---  1 d31548v rc-CANlab-admin 178 Feb 17 13:57 123 -> .git/annex/objects/G6/qW/SHA256E-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b/SHA256E-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
drwxrwx--- 10 f00689r rc-CANlab-admin 213 Feb 17 13:57 ..

but recently with @Michael-Sun we ran into situation where it did not work for us, and I filed an issue with @joeyh : https://git-annex.branchable.com/bugs/annex_add_freezes_before_moving__58___stopped_working/

Permission denied under /dartfs-hpc/rc/lab/C/CANlab/labdata/data/tmp

(datalad) [d31548v@discovery7 tmp]$ ( pwd; rm -rf test; set -x ; mkdir test; cd test; echo 123 > 123; nfs4_getfacl 123; /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123 ; nfs4_getfacl 123; mkdir d; mv 123 d/ || echo ERRORED && ls -lR; )
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/tmp
+ mkdir test
+ cd test
+ echo 123
+ nfs4_getfacl 123

# file: 123
A::OWNER@:rwadxtTnNcy
A::GROUP@:rwadxtTnNcCoy
A:g:[email protected]:rxtncy
+ /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123
+ nfs4_getfacl 123

# file: 123
A::OWNER@:rxtncy
A::GROUP@:rxtncCoy
A:g:[email protected]:rxtncy
+ mkdir d
+ mv -i 123 d/
mv: cannot move ‘123’ to ‘d/123’: Permission denied
+ echo ERRORED
ERRORED
+ ls --color=auto -lR
.:
total 80
-r-xr-x--- 1 d31548v rc-CANlab-admin 4 Feb 17 16:19 123
drwxrwx--- 2 d31548v rc-CANlab-admin 0 Feb 17 16:19 d

./d:
total 0
(datalad) [d31548v@discovery7 tmp]$ pwd
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/tmp

No permission denied error under /dartfs-hpc/rc/lab/C/CANlab/labdata/data/spacetop_data/tmp - although looks like with FACLs

(datalad) [d31548v@discovery7 tmp]$ ( pwd; rm -rf test; set -x ; mkdir test; cd test; echo 123 > 123; nfs4_getfacl 123; /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123 ; nfs4_getfacl 123; mkdir d; mv 123 d/ || echo ERRORED && ls -lR; )
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/spacetop_data/tmp
+ mkdir test
+ cd test
+ echo 123
+ nfs4_getfacl 123

# file: 123
A::OWNER@:rwadxtTnNcy
A::GROUP@:rwadxtTnNcCoy
A:g:[email protected]:rwadxtTnNcCoy
+ /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123
+ nfs4_getfacl 123

# file: 123
A::OWNER@:rxtncy
A::GROUP@:rxtncCoy
A:g:[email protected]:rxtncCoy
+ mkdir d
+ mv -i 123 d/
+ ls --color=auto -lR
.:
total 32
drwxrwx--- 2 d31548v rc-CANlab-admin 21 Feb 17 16:20 d

./d:
total 48
-r-xr-x--- 1 d31548v rc-CANlab-admin 4 Feb 17 16:20 123
(datalad) [d31548v@discovery7 tmp]$ pwd
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/spacetop_data/tmp

# seems to be not posix -- allows to write
(datalad) [d31548v@discovery7 tmp]$ touch 1; chmod -w 1; echo 1 >> 1
chmod: 1: new permissions are r-xrwx---, not r-xr-x---
(datalad) [d31548v@discovery7 tmp]$ touch 1; chmod a-w 1; echo 1 >> 1
(datalad) [d31548v@discovery7 tmp]$

/dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives - presumably no ACLs (seems to support that) -- move is working, freeze script does nothing

(datalad) [d31548v@discovery7 derivatives]$ ( pwd; rm -rf test; set -x ; mkdir test; cd test; echo 123 > 123; nfs4_getfacl 123; /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123 ; nfs4_getfacl 123; mkdir d; mv 123 d/ || echo ERRORED && ls -lR; )
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives
+ mkdir test
+ cd test
+ echo 123
+ nfs4_getfacl 123

# file: 123
A::OWNER@:rwatTnNcCy
A::GROUP@:rtncy
A::EVERYONE@:rtncy
+ /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123
+ nfs4_getfacl 123

# file: 123
A::OWNER@:rwatTnNcCy
A::GROUP@:rtncy
A::EVERYONE@:rtncy
+ mkdir d
+ mv -i 123 d/
+ ls --color=auto -lR
.:
total 32
drwxr-xr-x 2 d31548v rc-CANlab-wasabi 21 Feb 17 16:23 d

./d:
total 48
-rw-r--r-- 1 d31548v rc-CANlab-wasabi 4 Feb 17 16:23 123

# test if really posix
(datalad) [d31548v@discovery7 derivatives]$ touch 1; chmod -w 1; echo 1 >> 1
-bash: 1: Permission denied

overall -- I am still a little lost as we seems to have two types of ACLed locations , in one we can achieve "move write protected file" and in another - we can't ... need to figure it all out...

Q: what is the difference between ReproIn and BIDS

"is ReproIn a type of BIDS?"

provide kerberos instructions for passwordless datalad from discovery to rolando

Establish/document DataLad workflow for updating datasets on rolando with curation changes

HeudiConv/ReproIn organized BIDS datasets are not curated. Labs ideally should fetch them using DataLad as described in instructions and curate and enhance them. As long as DataLad (or git/git-annex directly) are used, we can establish a nice and unambigous workflow to propagate those changes back to rolando's centralized collection of datasets. Curation entails

removal or renaming of __dups with potential changes to IntendedFor of fmaps .json files (under git) if populated, and changes to _scans.tsv files (under annex).
addressing TODOs as documented through out the dataset
populate _events.tsv files

To make it work we need to

establish git-annex storage per each dataset git-repo available to both DBIC personnel curating BIDS datasets (well, me ATM), and researchers. We could
- use github for that with private repos, while automatically creating and populating repositories here. LFS for storing those _scans.tsv annexed files. That would also give us issues tracker etc. But even though private, not certain if "kosher". Also mapping rolando users to github might show to be pain
- "mirror" entire hierarchy on rolando where we give write access to researchers to push changes. E.g. could be /inbox/BIDS-curated or alike (e.g. just a nearby bare git-annex repo with needed permissions for each dataset with -curated suffix in the name. The simplest way ATM and could be done on 'case by case' basis). Cons: no issue tracker. Pros: simple/easy
- have gitlab DBIC instance provided somewhere internally. Pros: very featurefull, supports hierarchical organization, Cons: no git-annex support so we would still need the "mirror"
- have gin (https://gin.g-node.org/) DBIC instance. Pros: has all needed features of github and supports git-annex .

@andycon WDYT about gitlab or gin instance for DBIC? In both gitlab and gin cases not sure on how easy to integrate with user accounts/permissions already out there. Needs "research"

Consolidate datalad info and reference it

Datalad usage info is discussed in "Discovery", "Datalad", and "MRI/Data Access"

Consolidate this and link to common docs.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.