wiki:GestionDesDonnees
close Warning: Can't synchronize with repository "(default)" (/SVN/atlas does not appear to be a Subversion repository.). Look in the Trac log for more information.

Version 15 (modified by /C=FR/O=CNRS/OU=UMR5821/CN=Sabine Crepe-Renaudin/emailAddress=crepe@…, 6 years ago) (diff)

--

Data management on IN2P3-LPSC_LOCALGROUPDISK

IN2P3-LPSC_LOCALGROUPDISK is the place where you can store the data you need on the grid. These data can be the one you produce yourself or data produced by any other ATLAS people. It's a "space token" (a kind of virtual space booked on grid disks for a particular ATLAS use) opened to ATLAS LPSC (and Fr) members located on our T2 disks.
Currently (12/2015), 75 To are available on that space. If you consider you will need more space in the future, please contact Sabine.

Important notice: this space is shared and managed by the ATLAS LPSC group that means by you ! It's thus important that you:

  1. check that enough space is available before transferring your data
  2. check that datasets stored there are correctly registered on the grid (which is automatic if you use ATLAS standard tools)
  3. delete regularly the datasets that you don't need anymore

This twiki gives some hints on how to do this. Don't hesitate to contact Sabine if something is missing or not clear, you can also update the documentation by yourself.

What is the space left ?

See LOCALGROUPDISK plot http://dashb-atlas-ddm-acc.cern.ch/dashboard/request.py/ddmaccounting#button=srmview&sites[=IN2P3-LPSC&sitesCat[]=All+Countries&sitesSort=2&sitesCatSort=0&start=&end=&timerange=lastWeek&granularity=Daily&gen_spacetoken=0&gen_streamname=0&gen_project=0&gen_datatype=0&sortby=10&series=All][here]

Note! these plots are updated only once a day and several person from the group may intend to transfer data at the same time. If you see that the available space stats to be small (less than 5 To), please inform the group.

Where to store your data

https://twiki.cern.ch/twiki/bin/view/AtlasComputing/DQ2ClientsHowTo#General_Workflow_for_creating_da

Pour gérer vos données dans IN2P3-LPSC_LOCALGROUPDISK (english version below)

Pour lister les datasets sur notre IN2P3-LPSC_LOCALGROUPDISK :

dq2-list-dataset-site IN2P3-LPSC_LOCALGROUPDISK > & lpsclocal.txt &

Pour connaitre le possesseur d'une réplique d'un dataset :

dq2-get-replica-metadata nomdudataset

Pour lister les datasets qui vous appartiennent :

dq2-list-dataset-site2 -e -o "/O=GRID-FR/C=FR/O=CNRS/OU=LPSC/CN=YOURNAME" IN2P3-LPSC_LOCALGROUPDISK > mesdatasets.txt

et en connaitre la taille

awk -F"," -v a=0,b=0 '{print $6; a=a+$6; print "total = " a; b=a/1000000000; print "total en Giga = " b}' mesdatasets.txt

Pour effacer vos datasets :

dq2-delete-replicas nomdudatasets IN2P3-LPSC_LOCALGROUPDISK


Data management in IN2P3-LPSC_LOCALGROUPDISK

To list the datasets in IN2P3-LPSC_LOCALGROUPDISK:

dq2-list-dataset-site IN2P3-LPSC_LOCALGROUPDISK > & lpsclocal.txt &

To know the owner of a dataset replica:

dq2-get-replica-metadata nomdudataset

To list your datasets replicas:

dq2-list-dataset-site2 -e -o "/O=GRID-FR/C=FR/O=CNRS/OU=LPSC/CN=YOURNAME" IN2P3-LPSC_LOCALGROUPDISK > mesdatasets.txt

To list their size:

awk -F"," -v a=0,b=0 '{print $6; a=a+$6; print "total = " a; b=a/1000000000; print "total en Giga = " b}' mesdatasets.txt

To remove your datasets replicas:

dq2-delete-replicas nomdudatasets IN2P3-LPSC_LOCALGROUPDISK