The gLite WMS and the Data Management System

The gLite WMS and the Data Management System

The gLite WMS and the Data Management System Giuseppe LA ROCCA INFN Catania [email protected] Master Class for Life Science, 4-6 May 2010 Singapore Outline An introduction to the gLite WMS Job Submission via WMS Command line interface Job status The Job Description Language overview JDL attributes

The gLite DMS The Storage Resource Manager (SRM) Grid file referencing schemes LFC File Catalogue Architecture LFC commands File & Replica Management Client Tools Run bioinformatics applications via Grid portal The gLite stack: overview

Overview of the WMS The Workload Management System (WMS) is the gLite 3 component that allows users to submit jobs, and performs all tasks required to execute them, without exposing the user to the complexity of the Grid. Workload Management System (WMS) comprises a set of Grid middleware components responsible for distribution and management of tasks across Grid resources. The Workload Manager (WM) aims to accept and satisfy requests for job management coming from its clients. WM will pass the job to an appropriate CE for execution taking into account requirements and the preferences expressed in the job description. The decision of which resource should be used is the outcome of a matchmaking process. The Logging and Bookkeeping service tracks jobs managed by the WMS. It collects events from many WMS components and

records the status and history of the job. Job Submission via WMS GILDA User Interface create proxy Grid Site Computing Element VO Management Service (DB of VO users) Storage Element

Job Submission via WMS Workload Management Write JDL, Submit job (executable) + small inputs System GILDA User Interface Information System query create proxy publish

state Grid Site Computing Element VO Management Service (DB of VO users) Storage Element Job Submission via WMS Workload Management Write JDL, Submit job (executable) + small inputs

System GILDA User Interface Information System query create proxy Submit job publish state Loggin g

Grid Site Computing Element VO Management Service (DB of VO users) process Logging and bookkeeping Storage Element Job Submission via WMS Workload

Management Write JDL, Submit job (executable) + small inputs System GILDA User Interface Information System query create proxy Retrieve status & (small) output files

Retrieve output Job status Submit job publish state Loggin g Grid Site Computing Element

VO Management Service (DB of VO users) process Logging and bookkeeping Storage Element The Command Line Interface The gLite WMS implements two different services to manage jobs: the Network Server and the WMProxy. The recommended method to manage jobs is through the gLite WMS via WMProxy, because it gives the best performance and allows to use the most advanced functionalities

The WMProxy implements several functionalities, among which: submission of job collections; faster authentication; faster match-making; faster response time for users; higher job throughput. Proxy Delegation To explicitly delegate a user proxy to WMProxy, the command to use is:

glite-wms-job-delegate-proxy -d Example: $ glite-wms-job-delegate-proxy -d mydelegID Connecting to the service ======= glite-wms-job-delegate-proxy Success ======== Your proxy has been successfully delegated to the WMProxy: with the delegation identifier: mydelegID ===================================================== Job Submission Starting from a simple JDL file, we can submit it via WMProxy by doing: $ glite-wms-job-submit d mydelegID test.jdl Connecting to the service ======== glite-wms-job-submit Success ======== The job has been successfully submitted to the WMProxy Your job identifier is: ============================================== Listing CE(s) that matching a job It is possible to see which CEs are eligible to run a job described by a given JDL using: $ glite-wms-job-list-match d mydelegID test.jdl Connecting to the service ==================================================== COMPUTING ELEMENT IDs LIST The following CE(s) matching your job requirements have been found:

*CEId* - - - - ==================================================== Retrieving the status of a job $ glite-wms-job-status ===================================================== BOOKKEEPING INFORMATION: Status info for the Job : Current Status: Done (Success) Exit code: 0 Status Reason: Job terminated successfully

Destination: Submitted: Mon Dec 4 15:05:43 2006 CET ===================================================== The verbosity level controls the amount of information provided. The value of the -v option ranges from 0 to 3. The commands to get the job status can have several jobIDs as arguments, i.e.: glite-wms-job-status ... or, more conveniently, the -i option can be used to Retrieving the output(s) $ glite-wms-job-output Connecting to the service ===================================================== JOB GET OUTPUT OUTCOME Output sandbox files for the job: have been successfully retrieved and stored in the directory: /tmp/doe_yabp72aERhofLA6W2-LrJw ===================================================== The default location for storing the outputs (normally /tmp) is defined in the UI configuration, but it is possible to specify in which directory to save the output using the --dir option. Cancelling a job $ glite-wms-job-cancel Are you sure you want to remove specified job(s) [y/n]y : y Connecting to the service ========== glite-wms-job-cancel Success ============

The cancellation request has been successfully submitted for the following job(s): - ==================================================== If the cancellation is successful, the job will terminate in status CANCELLED Job Submission with CLI GILDA User Interface glite-wms-job-delegate-proxy -d delegID glite-wms-job-list-match d delegID hostname.jdl delegID glite-wms-job-submit -d delegID hostname.jdl JobID glite-wms-job-status JobID

glite-wms-job-output JobID Manage job voms-proxy-init --voms gilda Grid Site Computing Element VO Management Service (DB of VO users) Storage Element process

Possible Job states Job Description Language The Job Description Language (JDL) is a high-level language based on the Classified Advertisement (ClassAd) language, used to describe jobs and aggregates of jobs with arbitrary dependency relations. The JDL is used in WLCG/EGEE to specify the desired job characteristics and constraints, which are taken into account by the WMS to select the best resource to execute the job. A job description is a file (called JDL file) consisting of lines having the format: attribute = expression; Expressions can span several lines, but only the last one must be terminated by a semicolon.

Job Description Language The character cannot be used in the JDL. Comments must be preceded by a sharp character (#) or a double slash (//) at the beginning if each line. Multi-line comments must be enclosed between /* and */ . Attention! The JDL is sensitive to blank characters and tabs. No blank characters or tabs should follow the semicolon at the end of a line. Simple JDL example Executable = "/bin/hostname"; StdOutput = "std.out"; StdError = "std.err"; The Executable attribute specifies the command to be

run by the job. If the command is already present on the WN, it must be expressed as a absolute path; if it has to be copied from the UI, only the file name must be specified, and the path of the command on the UI should be given in the InputSandbox attribute. Executable = ""; InputSandbox = {"/home/larocca/"}; StdOutput = "std.out"; StdError = "std.err"; The Arguments attribute can contain a string value, which is taken as argument list for the executable: Arguments = "fileA 10"; In the Executable and in the Arguments attributes it may be necessary to use special characters, such as

&, \, |, >, <. These characters should be preceded by triple \ in the JDL, or specified inside quoted strings e.g.: Arguments = "-f file1\\\&file2"; The shell environment of the job can be modified using the Environment attribute. Environment = {"CMS_PATH=$HOME/cms"}; If files have to be copied from the UI to the execution node, they must be listed in the InputSandbox attribute: InputSandbox = {"", ... ,"fileN"}; The files to be transferred back to the UI after the job is finished can be specified using the OutputSandbox attribute: OutputSandbox = {"std.out","std.err"}; Wildcards are allowed only in the InputSandbox attribute. Absolute paths cannot be specified in the OutputSandbox attribute.

The InputSandbox cannot contain two files with the same name, even if they have a different absolute path, as when transferred they would overwrite each other. The Requirements attribute can be used to express constraints on the resources where the job should run. Its value is a Boolean expression that must evaluate to true for a job to run on that specific CE. Note: Only one Requirements attribute can be specified (if there are more than one, only the last one is considered). If several conditions must be applied to the job, then they all must be combined in a single Requirements attribute. For example, let us suppose that the user wants to run on a CE using PBS as batch system, and whose WNs have at least two

CPUs. He will write then in the job description file: Requirements = other.GlueCEInfoLRMSType == "PBS" && other.GlueCEInfoTotalCPUs > 1; The WMS can be also asked to send a job to a particular queue in a CE with the following expression: Requirements = other.GlueCEUniqueID == ""; It is also possible to use regular expressions when expressing a requirement. Let us suppose for example that the user wants all his jobs to run on any CE in the domain This can be achieved putting in the JDL file the following expression:

Requirements = RegExp("",other.GlueCEUniqueID); The opposite can be required by using: Requirements = (!RegExp("", other.GlueCEUniqueID)); If the job must run on a CE where a particular experiment software is installed and this information is published by the CE, something like the following must be written: Requirements = Member(BLAST-1.0.3, other.GlueHostApplicationSoftwareRunTimeEnvironment); Note: The Member operator is used to test if its first argument (a scalar value) is a member of its second argument (a list). In fact, the GlueHostApplicationSoftwareRunTimeEnvironment

attribute is a list of strings and is used to publish any VOspecific information relative to the CE (typically, information on the VO software available on that CE). Advanced job types Job Collection: a set of independent jobs that user can submit and monitor as it was a single job [ Type = Collection"; nodes={ [ Executable = "/bin/hostname"; Arguments = -f"; StdOutput = "hostname.out"; StdError = "hostname.err"; OutputSandbox = {"hostname.err","hostname.out"}; ],[ Executable = "/bin/sh";

Arguments = ""; StdOutput = povray.out"; StdError = povray.err"; InputSandbox = {"}; OutputSandbox = {povray.err",povray.out"}; ] Requirements = Member (POVRAY-3,5, other.GlueHostApplicationSoftwareRunTimeEnvironment); ] }; Advanced job types Parametric Job: a job collection where the jobs are identical but for the value of a running parameter JobType = "Parametric"; Executable = /bin/echo";

Arguments = _PARAM_; StdOutput = "myoutput_PARAM_.txt"; StdError = "myerror_PARAM_.txt"; Parameters = 3; ParameterStep = 1; ParameterStart = 1; OutputSandbox = {myoutput_PARAM_.txt}; Advanced job types DAG is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other ones The jobs are nodes (vertices) in the graph the edges (arcs) identify the dependencies Type = "dag"; max_nodes_running = 5;

InputSandbox = {"/tmp/foo/*.exe", "/home/larocca/bar", "gsi ", "file:///tmp/myconf"}; InputSandboxBaseURI = "gsi"; nodes = [ nodeA = [ description = [ JobType = "Normal"; Executable = "a.exe"; InputSandbox = { "/home/larocca/myfile.txt", root.InputSandbox}; ]; ]; nodeF = [ description = [ JobType = "Normal"; Executable = "b.exe"; nodeA Arguments = "1 2 3"; OutputSandbox = {"myoutput.txt", "myerror.txt" };

]; ]; nodeD = [ description = [ JobType = "Checkpointable"; Executable = "b.exe"; nodeB nodeC NodeF Arguments = "1 2 3"; InputSandbox = { "file:///home/larocca/data.txt", root.nodes.nodeF.description.OutputSandbox[0] }; ]; ]; nodeC = [ file = "/home/larocca/nodec.jdl"; ];

nodeB = [ file = "foo.jdl"; ]; ]; nodeD dependencies = { { nodeA, nodeB }, { nodeA, nodeC }, {nodeA, nodeF }, { { nodeB, nodeC, nodeF }, nodeD } }; References WMProxy Users guide https :// e-v0-3.pdf JDL Attributes Specification .pdf

.pdf gLite 3.1 users guide Complex jobs WMProxy API usage The gLite stack: overview Storage Elements The Storage Element is the service which allows a user or an application to store/retrieve data for future retrieval. The DMS provides services to locate, access and transfer files User does not need to know the physical location of file, just its

logical file name; Files can be replicated or transferred to several locations (SEs) as needed; Files are shared with all the members of the given VO. Files stored in a SE are written-once, read-many Files cannot be changed unless remove or replaced; Protocols The GSIFTP protocol offers the functionalities of FTP, but with support for GSI. It is responsible for secure, fast and efficient file transfers to/from Storage Elements. RFIO was developed to access tape archiving systems, such as CASTOR (CERN Advanced STORage manager) and it comes in a secure and an insecure version. The gsidcap protocol is the GSI enabled version of the dCache native access protocol, dcap.

SRM The Storage Resource Manager The gLite Storage Element Grid file referencing schemes LFN GUID SURL TURL Logical File Name (LFN)

lfn:/grid/gilda/tutorials/input-file Grid Unique IDentifier (GUID) guid:4d57edef-fa5c-4512-a345-1c838916b357 Storage URL (for a specific replica, on a specific Storage Element) srm:// fileb366f371-b2c0-485d-b12c-c114edaf4db4 sfn:// Transport URL (for a specific replica, on an SE, with a specific protocol) gsi fileb366f371-b2c0-485d-b12c-c114edaf4db4 Needles in a haystack

How do I keep track of all files I have on Grid ? How does the Grid keep track of the mapping between LFN(s), GUID and SURL(s) ? LFC File Catalogue LFC = LCG File Catalogue LCG = LHC Compute Grid LHC = Large Hadron Collider The LCG File Catalogue is the service which maintains mappings between LFN(s), GUID and SURL(s). LFC File Catalogue It consists of a unique catalogue, where the LFN is the main key. Further LFNs can be added as symlinks to the main LFN. Looks like a top-level directory in the Grid

For each of the supported VO a separate subdirectory does exist under /grid directory All the members of the VO have read/write permissions System metadata are supported, while for user metadata only a single string entry is available The catalogue publishes its endpoint in the Information Service so that it can be discovered by Data Management tools and other services (the WMS for example). LFC Commands lfc-chmod Change access mode of the LFC file/directory lfc-chown

Change owner and group of the LFC file/directory lfc-delcomment Delete the comment associated with the file/directory lfc-getacl Get file/directory access control lists lfc-ln Make a symbolic link to a file/directory lfc-ls

List file/directory entries in a directory lfc-mkdir Create a directory lfc-rename Rename a file/directory lfc-rm Remove a file/directory lfc-setacl

Set file/directory access control lists lfc-setcomment Add/replace a comment lfc-ls Listing the entries of a LFC directory lfc-ls [-cdiLlRTu] [--class] [--comment] [--deleted] [--display_side] [--ds] path where path specifies the LFN pathname (mandatory) Remember that LFC has a directory tree structure /grid// All members of a VO have read-write permissions under their directory LFC Namespace Defined by the user You can set LFC_HOME to use relative paths

lfc-ls export lfc-ls lfc-ls /grid/gilda/tutorials/taipei02 LFC_HOME=/grid/gilda/tutorials -l taipei02 -l -R /grid lfc-mkdir Creating directories in the LFC lfc-mkdir [-m mode] [-p] path... Where path specifies the LFC pathname Remember that while registering a new file (using lcgcr, for example) the corresponding destination directory must be created in the catalog beforehand. Examples:

lfc-mkdir /grid/gilda/ Created by the user lfc-ln Creating a symbolic link lfc-ln -s file linkname lfc-ln -s directory linkname Create a link to the specified file or directory with linkname Examples: lfc-ln -s /grid/gilda/test /grid/gilda/aLink Lets check the link using lfc-ls with long listing

lfc-ls -l aLink lrwxrwxrwx 1 19122 test Original File 1077 Symbolic Link 0 Jun 14 11:58 aLink -> /grid/gilda/ Adding metadata information The lfc-setcomment and lfc-delcomment commands allow the user to associate a comment with a catalogue entry and delete such comment. This is the only user-defined metadata that can

be associated with catalogue entries. The comments for the files may be listed using the --comment option of the lfc-ls command. This is shown in the following example: $ lfc-setcomment /grid/gilda/file1 My metadata $ lfc-ls --comment /grid/gilda/file1 /grid/gilda/file1 My metadata LCG Data Management Client Tools The LCG Data Management tools allow users to copy files between UI, WN and a SE, to register entries in the file catalogue and replicate files between SEs. lcg-cp Copies a Grid file to a local destination lcg-cr

Copies a file to a SE and registers it in the catalogue lcg-del Deletes one file (either one replica or all the replicas) lcg-rep Copies a file from one SE to another SE and registers it in the catalogue lcg-gt Gets the TURL for a given SURL and transfer protocol lcg-aa

Adds an alias in the catalogue for a given GUID lcg-ra Removes an alias in the catalogue for a given GUID lcg-rf Registers in the catalogue a file residing on a SE lcg-uf Unregisters in the catalogue a file residing on a SE lcg-la

Lists the aliases for a given LFN, GUID or SURL lcg-lg Gets the GUID for a given LFN or SURL lcg-lr Lists the replicas for a given LFN, GUID or SURL Environment variables /1 The --vo option, to specify the virtual organisation of the user, is present in all commands, except for lcg-gt. Its usage is mandatory unless the variable LCG_GFAL_VO is set (e.g.: export LCG_GFAL_VO=gilda) Timeouts

The commands lcg-cr, lcg-del, lcg-gt, lcg-rf, lcg-sd and lcg-rep all have timeouts implemented. By using the option -t, the user can specify a number of seconds for the timeout. The default is 0 seconds, that is no timeout. If we got a times out during the performing of an operation, all actions performed till that moment are rolled back, so no broken files are left on a SE and no existing files are not registered in the catalogues. Environment variables /2 For all lcg-* commands to work, the environment variable LCG_GFAL_INFOSYS must be set to point to a top BDII in the format :, so that the commands can retrieve the necessary information export

The VO__DEFAULT_SE variable specifies the default SE for the VO. export Uploading a file to the Grid /1 $ lcg-cr --vo gilda -d \ file:/home/larocca/file1 guid:6ac491ea-684c-11d8-8f12-9c97cebf582a where the only argument is the local file to be uploaded and the -d option indicates the SE used as the destination for the file. The command returns the file GUID. If no destination is given, the SE specified by the VO__DEFAULT_SE environmental variable is taken. The -P option allows the user to specify a relative path name for the file in the SE. If no -P option is given, the relative path is automatically generated.

Uploading a file to the Grid /2 The following are examples of the different ways to specify a destination: -d -d srm:// -d -P my_dir/my_file The l option can be used to specify a LFN: $ lcg-cr --vo gilda -d -l lfn:/grid/gilda/myalias1 \ file:/home/larocca/file1 guid:db7ddbc5-613e-423f-9501-3c0c00a0ae24 \ Replicating a file $ lcg-rep -v --vo gilda -d \ guid:db7ddbc5-613e-423f-9501-3c0c00a0ae24

Source URL: sfn:// File size: 30 Destination specified: Source URL for copy: gsi Destination URL for copy: gsiftp:///data/gilda/generated/2004-07-09/ file50c0752c-f61f-4bc3-b48e-af3f22924b57 # streams: 1 Transfer took 2040 ms Destination URL registered in LRC: srm:///data/gilda/generated/2004-07-09/file50c0752c -f61f-4bc3-b48e-af3f22924b57 Listing replicas $ lcg-lr --vo gilda \

lfn:/grid/gilda/tutorials/larocca/my_alias1 srm:// srm:///data/gilda/generated/2004-07-08/ file0dcabb46-2214-4db8-9ee8-2930 Again, a LFN, the GUID or a SURL can be used to specify the file. Copying files out the Grid $ lcg-cp --vo gilda -t 100 -v lfn:/grid/gilda/tutorials/mytext.txt file:/tmp/mytext.txt Source URL: lfn:/grid/gilda/mytext.txt File size: 104857600 Source URL for copy: gsi input2.dat.10.0 Destination URL: file:///tmp/myfile # streams: 1

# set timeout to 100 (seconds) 85983232 bytes 8396.77 KB/sec avg 9216.11 Transfer took 12040 ms Deleting replicas /1 A file stored on a SE and registered in LFC can be deleted using the lcg-del command. If a SURL is provided as argument, then that particular replica will be deleted. If a LFN or GUID is given instead then the s option must be used to indicate which one of the replicas must be erased $ lcg-del --vo gilda -s \ guid:91b89dfe-ff95-4614-bad2-c538bfa28fac Deleting replicas /2 If the a option is used, all the replicas of the given file

will be deleted and unregistered from the catalog. $ lcg-del --vo gilda -a \ guid:91b89dfe-ff95-4614-bad2-c538bfa28fac Working with large data datasets The InputSandbox and OutputSandbox attributes are the basic way to move files to and from the User Interface (UI) and the Worker Node (WN). However, there are other ways to move files to and from the WN especially when large files (> 10 MB) are involved User interface Input sandbox

DataSets info Output sandbox LCG File Catalogue (LFC) t pu In WMS sa x bo nd

r ke ox db o Br s an + t tpu

Ou fo In Storage Element 2 Computing Element References gLite 3 User Guide Manual Series https :// gLite Documentation homepage

p DM subsystem documentation LFC and DPM documentation entDocumentation DM API /Data_Management_Java_API Running more realistic jobs with the GENIUS Grid portal: Porting BLAST & MrBayes applications to Grid Case study from CNR - ITB

The GENIUS Grid Portal architecture The GENIUS Grid portal (license ver 4.2 is free for educational) is built on top of the EnginFrame Java/XML framework; Its a gateway to European EGEE Project middleware (its easily customizable for other middleware); It allows to expose gLite-enabled applications via web browser as well as Web Services. What is EnginFrame ?

It is a web-based technology able to expose Grid services running on Grid infrastructures It allows organizations to provide application-oriented computing and data services to both users (via Web browsers) and applications (via SOAP/WSDL and/or RSS) Its a Grid gateway!! It greatly simplifies the development of Web Portals exposing computing services that can run on a broad range of different computational Grid systems About MrBayes MrBayes is a program for the Bayesian estimation of phylogeny. Bayesian inference of phylogeny is based on the posterior probability distribution of trees, which is the probability of a tree conditioned on the observations. To approximate the posterior probability distribution of

trees MrBayes uses a simulation technique called Markov Chain Monte Carlo (or MCMC). The program takes as input a character matrix in a NEXUS file format. The output is several files with the parameters that were sampled by the MCMC algorithm. The application is CPU demanding, especially if the MPI version of the software is used. EnginFrame & MrBayes The Users Tracking System (UTS) /1 The Users Tracking System (UTS) /2 About BLAST BLAST (Basic Local Alignment Search Tool) provides a

method for rapid searching of nucleotide and protein databases. The program compares nucleotide or protein sequences to sequence databases and calculates the statistical significance of matches. Click here to download results Thank you for your attention!

Recently Viewed Presentations

  • Kettlewell &amp; the Peppered Moth - Savitha Sastry

    Kettlewell & the Peppered Moth - Savitha Sastry

    The Peppered Moth Example of Evolution Background Information 1850's: Industrial Revolution Moths (Biston betularia) come in both light and dark colors. Until the 1850's, the dark color was rare. After almost 100 years, the darker color was the dominant form....
  • Folie 1 - vfdb

    Folie 1 - vfdb

    Durch Beimischungen zum Löschwasser von Schaummitteln oder Gas wird die Löschwirkung des Wassers erhöht oder auf spezifische Brandlasten abgestimmt. Neben Wasserlöschanlagen, der sog. ... Stickstoff und Kohlendioxid oder Mischungen wie Inergen oder Argonite zum Einsatz. Mit synthetischen ...
  • Chapter 9: Objects and Classes - Kennesaw State University

    Chapter 9: Objects and Classes - Kennesaw State University

    Sarah Created Date: 6/10/1995 5:31:50 PM ... Times New Roman Arial Monotype Sorts Courier New Forte Courier SimSun Book Antiqua International 1_International Microsoft Word Picture Bitmap Image Chapter 9 Strings and Text I/O Objectives Motivations The String Class Constructing Strings...
  • VECG Company Profile

    VECG Company Profile

    An architecture for client-server web communication. A. pplication . P. rogramming . l. nterface (API) Lets products and services communicate with each other. REST describes any simple interface that transmits data over a standardized interface (such as HTTP).
  • Etruscans and Romans - Council Rock School District

    Etruscans and Romans - Council Rock School District

    The Kingdom Period of Rome Rome's Beginnings, and the Etruscans Sections One, Two, and Three Learning Goal for Rome's Beginnings The students will explain how Rome was founded. Romulus and Remus Etruscan Learning Goals The students will understand that there...
  • WS2 - Risk-Based Approach to IT Infrastructure Security and ...

    WS2 - Risk-Based Approach to IT Infrastructure Security and ...

    * DDoS Command & Control Bots - e.g. IoT Devices * ENISA Smartphone Risks (Source: R1 Data leakage R2 Improper decommissioning R3 Unintentional data disclosure R4 Phishing R5 Spyware R6 Network spoofing attacks R7 Surveillance R8 Diallerware R9 Financial...
  • A4 Inequalities - Pretty Math

    A4 Inequalities - Pretty Math

    ML3.PA4.1 Grade 12 Identify and use basic trig identities to find trig values Use basic trig identities to simplify and rewrite trig expressions Trigonometric identities Earlier in the course you met the following trigonometric identities: We can write these identities...
  • КроссЛексика - большой электронный словарь словосочетаний и ...

    КроссЛексика - большой электронный словарь словосочетаний и ...

    Examples: Synonym (graffiti) = wall-painting Synonym (graft) = transplant Synonym (halal) = corresponding to Muslim norms Hyperonym (endometriosis) = obstetric disease SemLs help to construct collocations lacking in CL.