UPD/UPR/UPP Reconstruction for Run II


new! There is now a upd user callout module which provides a ups-style configuration file for upd. See here for details.


UPD : Requirements - Design - API - Commands
UPP : Requirements - Design - API - Use Cases

Other Product Requirements - Glossary - Perl modules

Work is progressing on the UPS reconstruction for Run II. In its wake, several products that rely on UPS will need to updated to use the new syntax and features of the new UPS. Most notably of these products are UPD, UPR, and UPP.

UPD is used by most fixed target and collider experiments to copy/unwind/declare products whose archive files (currently in tar format) live on a central distribution node. The kits database is the most widely known of these central databases, but CDF also maintains a distribution database for experiment specific products. Ideally CDF would like to also have the kits area have a special database for experiment specific software so they don't need to maintain a separate node. UPD currently uses rsh as the underlying mechanism for the copy.

UPR is currently used by SDSS to deliver software to the astrophysics machines. UPR uses rsh/pipe as the underlying mechanism for the copy. UPR is similar to UPD, with the following exceptions:

UPP is a tool that can be used to automatically notify users of new software on a distribution node. It can also automatically copy and declare products to a local database if the instance does not exist or if the archive file has a newer date than the local product. It currently exists as a socket based server that runs on the distribution node. Users need to subscribe to the server from their local machines. UPP is currently used by PAT to monitor new software versions in kits and by CDF to distribute new software to collaborators.

Since UPD and UPR are so similar, our goal is to merge the functionality into a single new UPD package.


UPD

Requirements

The requirements on the new UPD then become:

Support the current functionality of both

Requirements due to new UPS New requirements Lessened functionality:

Design

ftp

The new upd is moving away from rsh as the underlying transport mechanism. ftp will be the future protocol for several reasons: Note that the last several features are available only with the wu-ftp server (from Washington University). This is a popular choice among ftp servers.

Choosing ftp as the underlying server does present a couple of problems that may require some local modification. As is, wu-ftp can only "reget" (ie, recover in the middle from a network failure) if the target and destination are flat files. It does not work for the compress on the fly feature. We have looked at the source and believe that it is trivial to support this additional feature. We also believe that Washington University would incorporate it into future releases. UPD should not require a modified ftp server on the distribution node, but some functionality may be lost without it.

An additional concern regards security. As is, the wu-ftp server does support some offsite restrictions for anonymous ftp, but only on a per node basis. Fermilab needs the ability to restrict offsite users on a per account basis. We would therefore have local modifications to the ftp server running on the kits database. When an anonymous ftp user logs in, he needs to enter an account name and domain. The server can verify that he entered the correct domain. The username name can be checked against an allowed list with the local ftp server modification, but there is no way to ensure that the user actually typed in his own account name. Note that no special ftp software need to run on the local node.

Support for the ability to extract a single file from the archive file will be dropped, since this is not a native ftp feature. In the current upd, there is a subset are files that are extracted with a significantly higher frequency. These are the ups files, the README and INSTALL_NOTE files, and for the new ups, the table and version files. These files will be available as flat files when placed in the distribution archive database.

The new upd will take advantage of the wu-ftp server feature to tar/compress files on the fly. If the compression will use more cpu on the server node than what is saved on network bandwidth, then upd will need a way to disable compression.

Another ramification of selecting ftp as the transport protocol is that all files that need to be distributed must be visible by the ftp server. This can be accomplished by mounting those files systems in the ftp area.

Overlaid Products

Some products want to be distributed as overlaid products. By that we mean that the dependent products are unwound in the same directory (e.g., cern_bin, cern_ups, cern_lib, etc. all go in cern).

For an overlaid product, there will need to be a special keyword in the version file for a dependency that says it really belongs to a different product. The version file for cern_ups would have a keyword that says: _UPD_OVERLAY = "cern". UPD would then use the "cern" as the product name when determining the root directory.

Caveats:

perl

The new UPD will be coded in perl (previously in c). Since there are no real speed requirements for upd, a scripting language like perl should be Native perl does not have ftp support; however, there are perl extensions that are available on the web. The local node would be required to have these extensions. The version of perl that is available in the kits database does include the ftp extensions. Should we bundle this in with upd? We think not. How does the user get this the first time?

upd cannot assume that perl lives in /usr/local/bin.

Perl Fragments

Every ups/upd installation has its unique conventions on where products are unwound and in which ups database to declare them. UPD cannot make any sensible guesses that would would apply to all cases. Therefore, upd will rely on user-supplied perl fragments to determine the installation specific parameters. UPD will search for these fragments first in $HOME and then in $UPD_USERCODE.

A given experiment and its collaborators may want to share the same user supplied fragments since it is likely they will have a similar product structure. This can be accomplished by having UPD itself be an overlaid product where the base upd is one element and the configuration files are a second.

UPD will provide a certain set of variables to the perl fragments, e.g

The current list of user supplied fragments are for:

Root Directory Determination

UPD needs to determine under which directory location to unwind a product. All products may not belong to the same file system, so UPD will make a user callout to obtain this information.

It is the responsibility of this fragment to return a product home directory based on these variables. It may be as simple as returning /usr/products/$UPD_FLAVOR/$UPD_PRODUCT/$UPD_VERSION. Examples of more complicated configurations will be provided.

This design has the advantage that upd get does not need to have a local version of ups to copy and unwind any software.

Selecting a Database

For systems with multiple local ups databases, upd needs to select in which database a product should be declared. It will select a database by calling a perl fragment to determine the local ups database. Examples will be provided.

Table File Location

upd will also need to determine the location of a product's table file. Two possible choices are with the product itself or in the ups database on the local machine. This will be yet another user callout.

ups Subdirectory Location

upd will also need to determine the location of a product's ups subdirectory. In most cases, it will be underneath the product roo. This will be yet another user callout.

User Interface

The first release will only support a command line interface - see following sec tion on API. Future releases might support some web form, yet to be designed.

Proprietary Products

A user needs to obtain a group and password from the computing division to allow access of proprietary products. This information must be obtained on a per product basis. This information will be entered interactively at the start of distribution.

API

Commands

Before invoking upd commands, the following rules must apply to databases defined in $PRODUCTS. The following user commands will be supported:

Switches

The following lists the switches that are available to the various upd commands. Note that not all switches are used by all commands. Refer to the detailed command description to see which swithces apply.
-f flavor
product flavor (as -f in ups commands).
-g chain
product chain (as -g in ups commands).
-G "switch to ups declare command"
switch to be passed to ups declare command. If more than one switch is to be passed, each one must have a separate -G switch.
-h node
node name that contains the the distribution ups database.
-H localflavor
Operating system of local node passed into ups list.
-i
ignore errors. If a product dependency can't be distributed, continue on.
-I
distribute with an intermediate tar file on the local node. That is to first copy the tar file on the local node and then unwind instead of unwinding on the fly. This option is needed if you want to be able to reget a tar product because the network connection was lost in in the middle of a transfer.
-j
get only top level product (ignore dependencies).
-J filename
specific file to fetch.
-q qualifiers
product qualifiers ( as -q in ups commands).
-R
get only required dependencies (ignore optional).
-s
list what command will do, without doing it.
-v verbose
print more information (as -v in ups commands).
-z ups_database
specify path to distribution ups database.

UPD Commands

upd install [-fgGhiIjqRsvz] product [version]

Gets a product and dependencies from the distribution node and unwind it on the local node, and then declare it to the ups database.

The following actions will occur:

  1. check prerequisites:
  2. determine specifically which instance we want via a cgi script wrapper on ups list on the distribution node. The following switches are passed from the command line into this ups list call: -fgHz. If neither -f nor -H are specified, then upd builds a local operating system flavor string and passes with -H to ups list.
  3. if -j is not specified, do a remote ups-get-dependencies on the distribution node, (via a cgi script wrapper on ups) which will expand all the chains to actual versions. The following swithces from the command line are passed to ups depend: -R.
  4. Go through the above list in order to build a table of all the products we need to install and for each product instance:
  5. Go through the shortened list in order and for each product instance:

  6. Do a local ups-get-dependencies to get the product list of dependencies for the main product distributed. Warn the user about all of the dependencies that don't match the list we got from the distribution node.
Example

We ask upd to do a upd install -f IRIX+5 exmh v1_6_6, which will install the mentioned version and its dependencies. Assume the following initial conditions:

Distribution Node Database:
             Product		               Dependencies
Name        version flavor qual. chain   Name        version flavor qual.
--------------------------------------   --------------------------------
exmh        v1_6_6  IRIX+5  ""  current  expect      current IRIX+5  ""
				         ispell      current IRIX+5  ""
				         mimetools   current IRIX+5  ""
				         popclient   current IRIX+5  ""
			    	         www         current IRIX+5  ""
ispell      v3_1a   IRIX+5  ""  current
expect      v5_18   IRIX+5  ""  current  tk          current IRIX+5  "" 
tk          v4_2    IRIX+5  ""  current  tcl         current IRIX+5  ""
tcl         v7_6    IRIX+5  ""  current
www         v2_7    IRIX+5  ""  current  ghostview   current IRIX+5  ""
				         ximagetools current IRIX+5  ""
                                         xpdf        current IRIX+5  ""
ghostview   v5_1    IRIX+5  ""  current
ximagetools v3_1    IRIX+5  ""  current
xpdf        v1_0    IRIX+5  ""  current

UPD local Node Database:
             Product		               Dependencies
Name        version flavor qual. chain   Name        version flavor qual.
--------------------------------------   --------------------------------
exmh	    v1_6_4  IRIX+5  ""  current  expect      current IRIX+5  ""
                                         ispell      current IRIX+5  ""
                                         mimetools   current IRIX+5  ""
                                         www         current IRIX+5  ""
ispell      v2_1    IRIX+5  ""  current
expect      v5_13   IRIX+5  ""  current  tk          current IRIX+5  ""
tk          v4_0    IRIX+5  ""  current  tcl         current IRIX+5  ""
tcl         v7_4    IRIX+5  ""  current
www         v2_7    IRIX+5  ""  current  ghostview   current IRIX+5  ""
                                         ximagetools current IRIX+5  ""
                                         xpdf        current IRIX+5  ""
ghostview   v5_1    IRIX+5  ""  current
ximagetools v3_1    IRIX+5  ""  current
xpdf        v1_0    IRIX+5  ""  current
upd determines that the following dependency list on the distribute node exists for this instance of exmh.
	exmh 		v1_6_6	IRIX+5 ""
	ispell		v3_1a	IRIX+5 ""
	expect		v5_18	IRIX+5 ""
	tk		v4_2	IRIX+5 ""
	tcl		v7_6	IRIX+5 ""
	www		v2_7	IRIX+5 ""
	ghostview 	v5_1	IRIX+5 ""
	ximagetools 	v3_1	IRIX+5 ""
	xpdf		v1_0	IRIX+5 ""
upd install will then get all of these versions and declare them to the local ups database. It will then do a local "ups depend" to get the product list of dependencies for exmh v1_6_6. Note that this will give us:
	exmh 		v1_6_6	IRIX+5 ""	
	ispell		v2_1	IRIX+5 ""
	expect		v5_13	IRIX+5 ""
	tk		v4_0	IRIX+5 ""
	tcl		v7_4	IRIX+5 ""
	www		v2_7	IRIX+5 ""
	ghostview	v5_1	IRIX+5 ""
	ximagetools 	v3_1	IRIX+5 ""
	xpdf		v1_0	IRIX+5 ""
     
Finally it will warn the user about all the dependencies that don't match the list we got from the distribution node, presented as a list of ups commands needed to resolve the dependencies.
	ups declare -c -f IRIX+5 ispell v3_1a 
	ups declare -c -f IRIX+5 expect	v5_18 
	ups declare -c -f IRIX+5 tk v4_2  
	ups declare -c -f IRIX+5 tcl v7_6 
     
which tells us we need to declare ispell, expect, tk, and tcl as current before we will get dependencies matching the ones on the server.

upd update component [-fgGhiIjqRsvz] product [version]

Overwrites component of the product instance where component can be table file or ups subdirectory. The product instance must already be declared on the local node.

The following actions will occur:

  1. check prerequisites:
  2. determine specifically which instance we want via a cgi script wrapper on ups list on the distribution node. The following switches are passed from the command line into this ups list call: -fgHz. If neither -f nor -H are specified, then upd builds a local operating system flavor string and passes with -H to ups list.
  3. if -j is not specified, do a remote ups-get-dependencies on the distribution node, (via a cgi script wrapper on ups) which will expand all the chains to actual versions. The following swithces from the command line are passed to ups depend: -R.
  4. Go through the above list in order to build a table of all the products we need to install and for each product instance:
  5. Go through the shortened list in order, and for each product instance:

upd fetch [-fghqsvz] [-J filename] product [version]

Fetches a particular file from a distribution node. If -J is not specified, then simply list the files that are available for fetching. The following actions will occur:
  1. check prerequisites:
  2. determine specifically which instance we want to fetch file from via a cgi script wrapper on ups list on the distribution node. The following switches are passed from the command line into this ups list call: -fgHz. If neither -f nor -H are specified, then upd builds a local operating system flavor string and passes with -H to ups list.
  3. get the product home directory, table file location and ups directory for the instance using the www ups list.
  4. open an ftp connection to the server, and cd to the product home directory. Note that this value must therefore be set even for archive only distribution database.
  5. if no file is specified with the -J switch, simply do an ftp ls of the current directory, of the table file, and of the ups directory. If a file is specified, get it with ftp get.

UPP

Requirements

UPP will still be a separate executable, but it should arguably be part of the UPD product itself. It has the following requirements.

Support current functionality:

New requirements: Lessened functionality:

Design

The new UPP will just be another piece of the UPD product itself. The current client/server model will be replaced with a perl script that lives solely on the local node. It will rely on the following preexisting software: UPP will appear very similar to the current UPP. Users will need to create a subscription file that contains a list of products for which they are interested and at what level. The appropriate upp command will then be added to the user's cron job and executed at a user specified interval. UPP will need to maintain various configuration and status files on the local node. The upp subscription file will have the following keywords:
FILE = UPP, to specify file type.
MAIL_ADDRESS = user@node , to specify mail recipient.
DISTRIBUTION_NODE = nodename, to specify name of distribution node.
DISTRIBUTION_DB = database_dir, to specify name of distribution database.
NEWPRODUCT_NOTIFICATION = T|F, to select if user should be notified of any new products. DATA_DIR = dir, to specify directory to keep state information.
NEWPRODUCT_NOTIFICATION = T|F, to select if user should be notified of any new products.

Then for each product a line of:

product_name product_version product_quals other_flags

Then a list of actions for each state change the user is interested in

ACTION = chain_name , to specify behavior when new chains are declared.
ACTION = version, to specify behavior when new specific version
ACTION = NEWVERSION, to specify behavior when any new version
Followed by a list of one or more actions:

If blank, assume notify only.
reget - (optional) take again even if we already have this instance
install - (unwind and declare)
resolve - make our chains match the distribution node
notify - tell user about the change
...

API

upp upp-subscription-file

Use Cases


Requirements Imposed on Other Products

These requirements to the new distribution mechanism imposes requirements to other related products.

For cmd add/mod/del product:

UPS

Glossary of Terms