dbic / handbook Goto Github PK
View Code? Open in Web Editor NEWWiP: DBIC Handbook
Home Page: https://dbic-handbook.readthedocs.io
License: Creative Commons Attribution 4.0 International
WiP: DBIC Handbook
Home Page: https://dbic-handbook.readthedocs.io
License: Creative Commons Attribution 4.0 International
we had notes in the "cheatsheet" -- need to be brought here
also doublecheck various tools (rsync/scp) on how they deal with symlinks
"isn't that bad?"
we inherited configuration from BIDS. It includes building and checking urls. But ATM it is not used at all -- we just rely on RTFD to render it.
ref: https://github.com/ReproNim/reprostim
ATM AFAIK we suspended collection of audio/videos, but it is to be resumed shortly if not already. After it is resumed, handbook should be updated to provide information on where to find the videos etc.
Invited @andycon to the @dbic/staff
The easiest way to resolve all "DataLad difficulties" and some user confusions with ACL, is to not use them. ATM ACL is used for /dartfs* volumes by default (/home and scratch are regular POSIX). Could we migrate entire /dartfs/rc/lab/D/DBIC
to be POSIX?
should be easy for people (e.g., MR tech) to
mkdocs serve
with a bookmark to the url presenting rendered version)Instructions on how to use this setup should go into CONTRIBUTING.md
@andycon - interested to take care about this one?
I am starting work on adding instructions to use ssh tunnel to launch a Jupiter server as a submitted job on Discovery. This allows users to interface with the Jupyter notebook from their browser.
or it could be alternatively only subpart(s) of it. Some labs (I was told) found a peace of mind already with ACLs
of tremendous interest for @jungheejung and @Michael-Sun.
Awhile back we established setup for how to work with discovery filesystem which is described in https://dbic-handbook.readthedocs.io/en/latest/mri/dataaccess.html#discovery-filesystem on how to use our custom freeze/thaw scripts.
[d31548v@discovery7 fmriprep]$ which git-annex
/dartfs/rc/lab/D/DBIC/DBIC/archive/git-annex/usr/lib/git-annex.linux/git-annex
[d31548v@discovery7 fmriprep]$ /dartfs/rc/lab/D/DBIC/DBIC/archive/git-annex/usr/lib/git-annex.linux/git-annex version
git-annex version: 10.20220127+git47-g9f9b1488e-1~ndall+1
[d31548v@discovery7 test]$ git init
Initialized empty Git repository in /dartfs-hpc/rc/lab/C/CANlab/labdata/data/spacetop_data/test/.git/
[d31548v@discovery7 test]$ git annex init
init ok
(recording state in git...)
[d31548v@discovery7 test]$ ls
[d31548v@discovery7 test]$ ls
[d31548v@discovery7 test]$ echo 123 > 123
[d31548v@discovery7 test]$ git annex add 123
add 123
ok
(recording state in git...)
g[d31548v@discovery7 test]$ git commit -m 123 123
[master (root-commit) b0232a6] 123
1 file changed, 1 insertion(+)
create mode 120000 123
[d31548v@discovery7 test]$ ls -lta
total 120
drwxrwx--- 9 d31548v rc-CANlab-admin 293 Feb 17 13:57 .git
drwxrwx--- 3 d31548v rc-CANlab-admin 43 Feb 17 13:57 .
lrwxrwx--- 1 d31548v rc-CANlab-admin 178 Feb 17 13:57 123 -> .git/annex/objects/G6/qW/SHA256E-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b/SHA256E-s4--181210f8f9c779c26da1d9b2075bde0127302ee0e3fca38c9a83f5b1dd8e5d3b
drwxrwx--- 10 f00689r rc-CANlab-admin 213 Feb 17 13:57 ..
but recently with @Michael-Sun we ran into situation where it did not work for us, and I filed an issue with @joeyh : https://git-annex.branchable.com/bugs/annex_add_freezes_before_moving__58___stopped_working/
(datalad) [d31548v@discovery7 tmp]$ ( pwd; rm -rf test; set -x ; mkdir test; cd test; echo 123 > 123; nfs4_getfacl 123; /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123 ; nfs4_getfacl 123; mkdir d; mv 123 d/ || echo ERRORED && ls -lR; )
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/tmp
+ mkdir test
+ cd test
+ echo 123
+ nfs4_getfacl 123
# file: 123
A::OWNER@:rwadxtTnNcy
A::GROUP@:rwadxtTnNcCoy
A:g:[email protected]:rxtncy
+ /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123
+ nfs4_getfacl 123
# file: 123
A::OWNER@:rxtncy
A::GROUP@:rxtncCoy
A:g:[email protected]:rxtncy
+ mkdir d
+ mv -i 123 d/
mv: cannot move ‘123’ to ‘d/123’: Permission denied
+ echo ERRORED
ERRORED
+ ls --color=auto -lR
.:
total 80
-r-xr-x--- 1 d31548v rc-CANlab-admin 4 Feb 17 16:19 123
drwxrwx--- 2 d31548v rc-CANlab-admin 0 Feb 17 16:19 d
./d:
total 0
(datalad) [d31548v@discovery7 tmp]$ pwd
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/tmp
(datalad) [d31548v@discovery7 tmp]$ ( pwd; rm -rf test; set -x ; mkdir test; cd test; echo 123 > 123; nfs4_getfacl 123; /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123 ; nfs4_getfacl 123; mkdir d; mv 123 d/ || echo ERRORED && ls -lR; )
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/spacetop_data/tmp
+ mkdir test
+ cd test
+ echo 123
+ nfs4_getfacl 123
# file: 123
A::OWNER@:rwadxtTnNcy
A::GROUP@:rwadxtTnNcCoy
A:g:[email protected]:rwadxtTnNcCoy
+ /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123
+ nfs4_getfacl 123
# file: 123
A::OWNER@:rxtncy
A::GROUP@:rxtncCoy
A:g:[email protected]:rxtncCoy
+ mkdir d
+ mv -i 123 d/
+ ls --color=auto -lR
.:
total 32
drwxrwx--- 2 d31548v rc-CANlab-admin 21 Feb 17 16:20 d
./d:
total 48
-r-xr-x--- 1 d31548v rc-CANlab-admin 4 Feb 17 16:20 123
(datalad) [d31548v@discovery7 tmp]$ pwd
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/spacetop_data/tmp
# seems to be not posix -- allows to write
(datalad) [d31548v@discovery7 tmp]$ touch 1; chmod -w 1; echo 1 >> 1
chmod: 1: new permissions are r-xrwx---, not r-xr-x---
(datalad) [d31548v@discovery7 tmp]$ touch 1; chmod a-w 1; echo 1 >> 1
(datalad) [d31548v@discovery7 tmp]$
(datalad) [d31548v@discovery7 derivatives]$ ( pwd; rm -rf test; set -x ; mkdir test; cd test; echo 123 > 123; nfs4_getfacl 123; /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123 ; nfs4_getfacl 123; mkdir d; mv 123 d/ || echo ERRORED && ls -lR; )
/dartfs-hpc/rc/lab/C/CANlab/labdata/data/WASABI/derivatives
+ mkdir test
+ cd test
+ echo 123
+ nfs4_getfacl 123
# file: 123
A::OWNER@:rwatTnNcCy
A::GROUP@:rtncy
A::EVERYONE@:rtncy
+ /dartfs/rc/lab/D/DBIC/DBIC/archive/bin-annex/freeze-content 123
+ nfs4_getfacl 123
# file: 123
A::OWNER@:rwatTnNcCy
A::GROUP@:rtncy
A::EVERYONE@:rtncy
+ mkdir d
+ mv -i 123 d/
+ ls --color=auto -lR
.:
total 32
drwxr-xr-x 2 d31548v rc-CANlab-wasabi 21 Feb 17 16:23 d
./d:
total 48
-rw-r--r-- 1 d31548v rc-CANlab-wasabi 4 Feb 17 16:23 123
# test if really posix
(datalad) [d31548v@discovery7 derivatives]$ touch 1; chmod -w 1; echo 1 >> 1
-bash: 1: Permission denied
overall -- I am still a little lost as we seems to have two types of ACLed locations , in one we can achieve "move write protected file" and in another - we can't ... need to figure it all out...
"is ReproIn a type of BIDS?"
I guess similar to the ones we have in https://dbic-handbook.readthedocs.io/en/latest/discovery.html#password-less-authentication-for-ssh
HeudiConv/ReproIn organized BIDS datasets are not curated. Labs ideally should fetch them using DataLad as described in instructions and curate and enhance them. As long as DataLad (or git/git-annex directly) are used, we can establish a nice and unambigous workflow to propagate those changes back to rolando's centralized collection of datasets. Curation entails
__dup
s with potential changes to IntendedFor of fmaps .json files (under git) if populated, and changes to _scans.tsv
files (under annex).TODO
s as documented through out the datasetTo make it work we need to
-curated
suffix in the name. The simplest way ATM and could be done on 'case by case' basis). Cons: no issue tracker. Pros: simple/easy@andycon WDYT about gitlab or gin instance for DBIC? In both gitlab and gin cases not sure on how easy to integrate with user accounts/permissions already out there. Needs "research"
Datalad usage info is discussed in "Discovery", "Datalad", and "MRI/Data Access"
Consolidate this and link to common docs.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.