wiki:GridAtlas
Last modified 13 years ago Last modified on 01/21/09 07:23:13

Introduction

GridAtlas is a tool that hides and automates the process of collecting the installation properties of Applications across grid Resources by keeping track of resource and application instance details. Upon a user job submission, a job submission interface contacts GridAtlas and obtains necessary information to complement user provided information after which further steps of job submission process can take place.

Database schema

Entities used:

Resources – represent physical resources using their fully qualified domain name (FQDN). Furthermore, because different systems adopt different approaches to configuring local resource managers (LRMs), a mapping that specifies these policies is needed. For example, a resource may schedule user jobs one per node, irrespective of the number or CPUs or processing cores exist on an individual node. Alternatively, each CPU can represent a scheduling slot on a resource. This entity in GridAtlas should be used to denote how many Processing Elements (PEs) exist for each Processing Slot (PS) on given resource.

Applications – represents an individual application that is installed and available for execution. An application must be referable by name and have an installation path for each resource associated with it. Furthermore, details about an individual application instance can be provided, including application version and a short description. Tentatively, it would be interesting to assign execution permissions to each application as it is associated with a resource. Such permission could be kept for different VOs or individual users.

Application Input – represents an application input data that may be required. An example would be BLAST databases. It should be referenced by a name that would be familiar to an application user (e.g., in case of BLAST, nr would be an example of a specific database). The key information to store is availability path for selected resource. Furthermore, a field for a short description should be provided. As the case with Applications is, permissions for accessing data could be incorporated in GridAtlas but more discussion on the approach is needed.

MySQL database is used to store the information relating to Grid Atlas.

Step 1: Create new database called "gridatlas"

Login to mysql as follows:

mysql -u appdb -p appdb

It prompts for the password. Enter the password. Create a database as follows:

mysql> CREATE DATABASE gridatlas;

To use the newly created database, enter the following command:

mysql> USE gridatlas;

Step 2: Create the tables representing the Grid Atlas entities

We create 4 tables namely Application, AppInput?, Resource and IsInstalled?.

Application table captures the following attributes:

  1. AppID - A unique identifier for all the applications. It is a auto-increment number generator. This is the key for the table
  1. Application Name
  1. Version
  1. Path (Either the absolute or the relative path)
  1. Description - within 100 characters

AppInput? table captures the following attributes:

  1. Name - This is the key for the table. It represents the Application Input Name.
  1. Path
  1. Description - within 100 characters
  1. AppID - this is the foreign key from the Application table

Resource table captures the following attributes:

  1. FQDN - This is the key for the table. It represents the "Fully Qualified Domain Name" of the resources
  1. PE_PS - Represents how many Processing Elements (PEs) exist for each Processing Slot (PS) on given resource

IsInstalled? table basically captures the many-many relationship between the Application and Resources. This means the many Applications can run on a particular Resource and also a particular Application may run on many Resources.

IsInstalled? table captures the following attributes:

  1. FQDN - This is a foreign key from the Resource table
  1. AppID - This is a foreign key from the Application table

IsInstalled? specifies a mapping between the Applications and Resources

MySQL commands for creation of the above table:

create table Application(AppID integer auto_increment NOT NULL,AppName varchar(30) NOT NULL, version varchar(5) NOT NULL,path varchar(50) NOT NULL,description varchar(100), PRIMARY KEY(AppID) );

create table AppInput(Name varchar(30) NOT NULL,path varchar(50) NOT NULL,description varchar(100), AppID integer NOT NULL, PRIMARY KEY(Name), FOREIGN KEY(AppID) references Application(AppID) ON DELETE CASCASE ON UPDATE CASCADE );
create table Resource(FQDN varchar(40) NOT NULL,PE_PS integer NOT NULL,PRIMARY KEY(FQDN));
create table IsInstalled(FQDN varchar(40) NOT NULL,AppID integer NOT NULL,PRIMARY KEY(FQDN,AppID),FOREIGN KEY(FQDN) REFERENCES Resource(FQDN) ON DELETE CASCADE ON UPDATE CASCADE,FOREIGN KEY(AppID) references Application(AppID) ON DELETE CASCADE ON UPDATE CASCADE);

Installation procedure

GridAtlas is represented by a three-tier architecture. A database backend keeps relevant information and dependencies about an application and a resource. A web service API represents a method to access this information though programmatic methods. There is also a web-based interface allowing the mapping available functionality for simple browser access. Installation of GridAtlas can be done in two ways:

  1. Client only
    • This is intended for anyone interested in accessing GridAtlas service through programmatic API interface. As such, a client class will have to be developed in order to access existing services.
    • If you are interested in Client ONLY installation, skip to Step 7 below.
  2. Service installation
    • In case one wants to host the GridAtlas service and thus be a provided, the service class(es) will have to be installed on every resource whose information wants to be exposed. In that case, registered resource and service will be accessible by GridAtlas client as well as a properly configured web-interface.
    • If you want to install a GridAtlas service instance, follow instructions starting at Step 1.

Software Requirements

  1. Apache Tomcat 6.0.14
  1. Axis 1.4
  1. Hibernate 3.2
  1. MySQL Database
  1. JDK 1.6

Service installation details

Step 1:

Install Tomcat, Axis and Hibernate on the Gridportal machine (respective versions)

The installation details for Tomcat and Hibernate can be found in the following link http://projects.uabgrid.uab.edu/cclprojects/wiki/AppDb

Axis installation:

Download axis from the following site: http://ws.apache.org/axis/

Step 2:

  • Copy the axis-1.4.tar.gz downloaded in the previous step into the $CATALINA_HOME/webapps/ folder of Tomcat
  • Copy GridAtlasService?.tar.gz to $CATALINA_HOME/webapps/
  • Untar both of the above copied files:
       tar xvzf axis-1.4.tar.gz
       tar xvzf GridAtlasService.tar.gz
    
  • This will create two directories: $CATALINA_HOME/webapps/axis-1.4/ and $CATALINA_HOME/webapps/axis/

Step 3:

Set up the required classpath. The classpath should point to the following jar files:

  1. Axis jar files
  2. Hibernate jar files
  3. Tomcat jar files
  4. JDK jar files
  5. MySQL Driver & XML Parsing jar files

The classpath is set in the .bash_profile file after logging into the "gridportal" machine.

Sample .bash_profile would looks as follows:

#export JAVA_HOME=/netbin/java
export JAVA_HOME=~/jdk1.6.0_02
export PATH=${JAVA_HOME}/bin:${PATH}

export CATALINA_HOME=~/apache-tomcat-6.0.14
export PATH=${CATALINA_HOME}/bin:${PATH}

export CLASSPATH=$CLASSPATH:$CATALINA_HOME/webapps/axis-1_4/lib/wsdl4j-1.5.1.jar:$CATALINA_HOME/webapps/axis-1_4/lib/saaj.jar:$CATALINA_HOME/webapps/axis-1_4/lib/mailapi.jar:$CATALINA_HOME/webapps/axis-1_4/lib/log4j-1.2.8.jar:$CATALINA_HOME/webapps/axis-1_4/lib/jaxrpc.jar:$CATALINA_HOME/webapps/axis-1_4/lib/commons-logging-1.0.4.jar:$CATALINA_HOME/webapps/axis-1_4/lib/commons-discovery-0.2.jar:$CATALINA_HOME/webapps/axis-1_4/lib/axis.jar:$CATALINA_HOME/webapps/axis-1_4/lib/axis-ant.jar:$CATALINA_HOME/webapps/axis-1_4/lib/activation.jar:$CATALINA_HOME/webapps/axis-1_4/lib/xmlsec-1.3.0.jar:$CATALINA_HOME/webapps/axis/WEB-INF/classes/:$CATALINA_HOME/webapps/axis-1_4/lib/xerces-2.6.2.jar:.

Step 4:
GridAtlas uses a predefined table structure for MySQL. In order to import this table structure, a file (GridAtlas.sql) is provided in the root directory of GridAtlasService?.tar.gz file. To import this table structure into MySQL, the user needs to have access to MySQL and have a database created. In order to import the database structure, execute the following command:

   mysql -u <USER_NAME> -p <DATABASE_NAME> < GridAtlas.sql
   Enter password: <USER_NAME_PASSWORD>

In this folder $CATALINA_HOME/webapps/axis/WEB-INF/src update the following file to account for MySQL access:

hibernate.cfg.xml : This is a configuration file used by Hibernate, which specifies the connection to the database "gridatlas". The MySQL username and password for connection are provided in this file. This file also specifies the location of the mapping file (with .hbm.xml extension)

The hibernate.cfg.xml file looks as below:

-bash-3.1$ more hibernate.cfg.xml 
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost/[DATABASE_NAME]</property>
<property name="hibernate.connection.username">[DATABASE_USER]</property>
        <property name="hibernate.connection.password">[DATABASE_PASSWORD]</property>
        <property name="dialect">org.hibernate.dialect.MySQLDialect</property>
<property name="show_sql">true</property>
<mapping resource="Application.hbm.xml" />
</session-factory>
</hibernate-configuration>

Description of files in package

Application.hbm.xml

This is a mapping file used by Hibernate. It provides a mapping between the tables in the "gridatlas" database and the Java classes representing those tables. It also provides a mapping between the attributes of each table to "fields or variables" in the Java classes.

Note: The table IsInstalled? is automatically populated whenever a new Application is added to the database. Since a new Application is added for a given Resource, the FQDN of the Resource and the AppID of the Application are automatically added to the database whenever a new Application is inserted. This is represented by the "bold text" in the above Application.hbm.xml file

Application.java

This class corresponds to the Application table of the database. It defines instance variables representing the attributes of the table "Application". It has setter and getter methods for each instance variable

Resource.java

This class corresponds to the Resource table of the database. It defines instance variables representing the attributes of the table "Resource". It has setter and getter methods for each instance variable

AppInput?.java

This class corresponds to the AppInput? table of the database. It defines the instance variables representing the attributes of the table "AppInput?". It has setter and getter methods for each instance variable

IsInstalled?.java

This class corresponds to the IsInstalled? table of the database. It defines the instance variables representing the attributes of the table "IsInstalled?". It has setter and getter methods for each instance variable

This folder $CATALINA_HOME/webapps/axis/WEB-INF/classes has the following files:

It contains both the hibernate.cfg.xml and Application.hbm.xml files. It contains all the .class files of the classes Application.java,AppInput?.java,Resource.java and IsInstalled?.java

Step 5

Write a Axis Web Service Java class and put it in the folder $CATALINA_HOME/webapps/axis. For a template of this class, see everest_service.java or olympus_service.java.

This Service class is written as a .java class. It provides service methods to do the following functionality:

Functionality Provided by Web Service API

  1. Add a new Resource
  1. Add a new Application by specifying the Resource on which it runs
  1. Add a new AppInput? by specifying the Application to which it is associated
  1. Get application path provided application name and resource FQDN
  1. Get number of PE per each PS on a resource given resource FQDN
  1. Get list of applications installed on a resource, provided resource FQDN
  1. Get list of resources an application is installed on, provided application name
  1. Get application description, provided application name - It returns a list of application descriptions for all the Applications with the same name
  1. Get application version, provided application name - It returns a lit of application versions for all the Applications with the same name
  1. Get path of application input entity provided application name, resource FQDN and name of application input entity
  1. Get description of application input entity provided application name, resource FQDN and name of application input entity
  1. Get list of all application input entities provided application name and resource FQDN

The service class written in this case is called GridAtlas_Service.java

Step 6

  1. Start the Tomcat Server with the following command :
    $CATALINA_HOME/bin/startup.sh
    
  1. Compile the Service class as follows:
    javac <service class>.java
    
    For example, everest_service.java or olympus_service.java
  1. Convert the .java Service class (found in $CATALINA_HOME/webapps/axis) into a .jws class (in the same directory) by the following command:
cp <service_class>.java <service_class>.jws
  1. Copy the edu.uab.cis.gridportal.axis.olympus_service_jws.* package to $CATALINA_HOME/webapps/axis/WEB-INF/classes/
    cp -rf $CATALINA_HOME/webapps/axis/edu $CATALINA_HOME/webapps/axis/WEB_INF/classes/
    
  1. Now access the URL: http://gridportal.cis.uab.edu:<PORT_NUMBER>/axis/<service_class>.jws The WSDL(Web Service Definition Language) for the WebService? would be available at this site.

Step 7 If a client wants to write a Client program to use the above GridAtlas_Service methods, then he has to do the following things:

  1. Have Axis installed on his/her client machine
  1. Set the classpath to the Axis libraries

Sample classpath setting would be as follows:

export AXIS_HOME=/netbin/axis-1_4 
export AXIS_LIB=$AXIS_HOME/lib 
export AXIS_CLASSPATH=$AXIS_LIB/axis-ant.jar:$AXIS_LIB/axis.jar:$AXIS_LIB/commons-discovery-0.2.jar:$AXIS_LIB/saaj.jar:$AXIS_LIB/commons-logging-1.0.4.jar:$AXIS_LIB/jaxrpc.jar:$AXIS_LIB/log4j-1.2.8.jar:$AXIS_LIB/wsdl4j-1.5.1.jar:$AXIS_LIB/activation.jar:$AXIS_LIB/mailapi.jar:$AXIS_LIB/xmlsec-1.3.0.jar:.
export CLASSPATH=$AXIS_CLASSPATH:$CLASSPATH
  1. Execute the command on his shell:
    java org.apache.axis.wsdl.WSDL2Java http://gridportal.cis.uab.edu:<PORT_NUMER>/axis/GridAtlas_Service.jws?wsdl
    
  1. This will generate the client stub classes in the $CATALINA_HOME/webapps/axis/edu/uab/cis/gridportal/axis/<service_class>_jws folder. Go to that folder and compile the stub classes by doing javac *.java

This command also creates a folder called "DefaultNameSpace?" which contains the Resource.java file. Compile this class using the command

javac Resource.java

Then copy the folder "DefaultNameSpace?" into the folder $CATALINA_HOME/webapps/axis/edu/uab/cis/gridportal/axis/<service_class>_jws

  1. Now the user will have to write a client (here we are writing a client Java class) class to access the methods of the GridAtlas_Service class.

The Client class would look like below:

Grid Atlas Client Class

import java.util.Iterator;
import java.util.*;
import javax.xml.namespace.QName;
import org.apache.axis.*;
import edu.uab.cis.gridportal.axis.GridAtlas_Service_jws.*;

public class GridAtlas_Client {
        public static void main(String[] args) throws Exception {

                //Creating an instance of the GridAtlas_Service class
                GridAtlas_ServiceService grid = new GridAtlas_ServiceServiceLocator();
                 
                edu.uab.cis.gridportal.axis.GridAtlas_Service_jws.GridAtlas_Service service = grid.getGridAtlas_Service();

                String path = service.getAppPath("A1","olympus");
                System.out.println("The path returned is:"+ path);

                int pe_ps = service.getPE("olympus");
                System.out.println("The number of pe for a given resource is"+pe_ps);

                String[] apps = service.getApplications("olympus");
                for(int i = 0; i< apps.length; i++) {
                        System.out.println("The set of applications running on a resource is:"+apps[i]);
                }

                String rsrcs[] = service.getResources("A1");
                for(int i = 0; i< rsrcs.length; i++) {
                        System.out.println("The set of resources used by an application is:"+rsrcs[i]);
                }


                String[] descp = service.getAppDescp("A1");
                for(int i = 0; i< descp.length; i++) {
                        System.out.println("The descriptions for a given application is:"+descp[i]);
                }

                String[] version = service.getAppVersion("A1");
                for(int i = 0; i< version.length; i++) {
                        System.out.println("The versions for a given application is:"+version[i]);
                }

                path = service.getAppInputPath("nr","A1","olympus");
                System.out.println("The path for AppInput is:"+ path);
                
                String descp = service.getAppInputDescp("nr","A1","olympus");
                System.out.println("The description for AppInput is:"+ descp);

                String appinputs[] = service.getAppInputs("A1","olympus");
                for(int i = 0; i< appinputs.length; i++) {
                        System.out.println("The inputs for a given application is:"+descp[i]);
                }
        }
}
  1. When the user compiles and runs the code, the client class will access the methods defined in the Service class deployed as the .jws file. The user gets back the appropriate results.

Thus the Webservice API acts as a front end to the user to access functionality for the GridAtlas Database. Web service API is developed to include all the necessary access points for the data stored in the backend database.

The functionality provided by the webservice API is defined under the heading "Functionality Provided by Web Service API"