alexeyg / cluster_tools Goto Github PK

PREREQUISITES
=============
- add LFC_HOME to your environment,
   (not necessary if only low-level interface and/or executables are used)

   e.g. add to your script something like:
   export LFC_HOME='/grid/lsgrid/marc/'
   This directory will be used to store the data.

- startGridSession lsgrid

CLUSTER STORAGE
===============
Cluster storage performs all operations robustly
- it automatically retries on failure (with waiting periods)
- it replicates accross grid
- it uses hashing to check identity


Python objects
--------------
For sending large python objects, make use of submit / receive functions in cluster_storage.py:

#somewhere on grid, do:
id = submit(your_object)
#or, with fixed id:
id = submit(your_object,"your_id")

#somewhere else on grid, do:
object = receive(id)    

#clean up
destroy(id)


Files 
-----
Same as python objects, but use filenames:

#somewhere on grid, do:
id = submit_file(filename)

#or, with fixed id:
id = submit_file(filename,"your_id")

#somewhere else on grid, do:
filename = receive_file(id)

#clean up
destroy(id)


Low-level
---------
Use ClusterStorageEngine object, available as:  cluster_storage.cs

cs.is_file('/grid/lsgrid/marc/test.dat')
cs.delete_file('/grid/lsgrid/marc/test.dat')
cs.replicate_all('/grid/lsgrid/marc/test.dat')

cs.retrieve_file('/grid/lsgrid/marc/test.dat','local_name.dat')
cs.store_file('local_name.dat','/grid/lsgrid/marc/test.dat')

#do not redownload file to check if store really succeeded:
cs.store_file('local_name.dat','/grid/lsgrid/marc/test.dat', check_equal=False)

cs.list_dir('/grid/lsgrid/marc')

#list of storage servers
cs.get_other_storage_engines()


Executables
-----------
copy all files to bin directory, make them executable if necessary (use chmod), then:

#copy 
gcp /grid/lsgrid/marc/test.dat local_name.dat
gcp local_name.dat /grid/lsgrid/marc/test.dat

#replicate
greplicate /grid/lsgrid/marc/test.dat

#delete
grm /grid/lsgrid/marc/test.dat

#directories
gmkdir /grid/lsgrid/marc/test
grmdir /grid/lsgrid/marc/test


#Note 1:
Grid paths can be indicated by prepending with grid:/
This can be useful with a command such as gcp, to distinguish local and grid paths
(Otherwise, paths starting with /grid/ are assumed to be on the grid). E.g: 

gcp grid://grid/lsgrid/marc/test.dat local_name.dat

#Note 2:
If a grid path does not start with the / symbol, the command will prepend it with 
the contents of LFC_HOME (i.e. your 'home' directory on the grid storage)

#create a dir test under the home dir:
gmkdir test

#copy a file to it
gcp local_name.dat grid:/test/test.dat