Setting up a BaBar Objectivity Server

The BaBar Objectivity Server installation consists of three processes, the AMS server, the Lock Server and the PUD.

Installing AMS

Make Directories and Links

The normal set up has files scattered over server different location on the server, this setup puts all the files under /opt/objy and puts links in so the software can find it's files in the expected places.

The structure under /opt/objy is as below:

• /opt/objy/
  • adm/ooss/logs/
  • spool/objy/
  • etc/ooss/

The links into this tree from elsewhere are:

• /var/adm/ooss &rarr /opt/objy/adm/ooss
• /usr/spool/objy &rarr /opt/objy/spool/objy
• /usr/etc/ooss &rarr /opt/objy/etc/ooss

These should all be owned by the account that the service will run as:

chown -R bfactory:babar objy

Get the Files

Copy the config files from RAL to /opt/objy/etc/ooss Copy the executables and libraries from /afs/slac.stanford.edu/package/mps/distrib/amscur/i386_rhel30 to /opt/objy/etc/ooss/amscurr

Configuring AMS

Edit StartAMS.cf to point to the correct local paths for certain utilities and to start the service as the correct user

Edit amshpss.cf to point to the local area where the databases will be held.

Increase the number of File Descriptors

Edit /etc/secutiry/limits.conf to include the following lines:

bfactory        soft    nofile  4096
bfactory        hard    nofile  50000

Where bfactory should be replaced by the account your running the service as.

Starting and Stopping the Server

To start the server

StartAMS.cf

If it's running corectly you should see:

ps -HU bfactory
  PID TTY          TIME CMD
16256 pts/5    00:00:00 bash
16310 pts/5    00:00:00   ps
16298 pts/5    00:00:00 keeplog
16297 pts/5    00:00:00 ooams

To stop the server

StopAMS.cf

Or use the yet to be released slac-ams sysV init script to stop and start the services

Running Multiple AMS Servers on One Host

A single instance of the SLAC AMS server cannot utilise the full CPU, disk or network bandwidth of a modern host. It therefore pays to run multiple instances of the AMS server to allow more clients to be supported off a single copy of the database.

Each instance of the AMS server has to run on a separate port, then the clients need to be persuaded to talk to the servers on these non-standard ports;

Running the AMS server on a Non-Standard Port

First you need to add the extra ports into /etc/services. e.g.

ooams-3a 1992/tcp
ooams-3b 1993/tcp
ooams-3c 1994/tcp
ooams-3d 1995/tcp 

Then repeatedly runnning the StartAMS script with the -s <portname> option will start the service:

StartAMS -s ooams-3a
StartAMS -s ooams-3b
StartAMS -s ooams-3c
StartAMS -s ooams-3d

Making the Clients Talk to the Servers on the Non-Standard Port

There is no client side config option to make the Objectivity client talk to the AMS server on multiple or non-standard ports. There are two ways to fool it one on the client side and one on the server side.

Client Side Config Method

Divide the client machines into N groups where N equals the number of AMS server instances you are running on each server host.

On each group of client machines edit /etc/services to redefine the normal ooams-3 port number from 6779 to one of you non-standard port numbers e.g:

ooams-3 1992/tcp

Choose a different non standard port for each client group. Each group will then talk to a different server instance on the server host.

Server Side Method (only for Linux)

Add N extra IP Addresses to the server machine bound together with a single DNS name then use IPTables to internally NAT between port 6779 on the extra IP Addresses and the non-standard ports you are running your server on.

I.e. if you server uses eth0 create extra ifcfg-eth0:[0-(N-1)] in /etc/sysconfig/network-scripts for N extra IP addresses:

DEVICE=eth0:0
IPADDR=192.168.0.1
NETMASK=255.255.255.0
NETWORK=192.168.0.0
BROADCAST=192.168.0.255
ONBOOT=yes
TYPE=Ethernet
GATEWAY=192.168.0.254

Then bind the extra IP addresses to a single DNS name.

host ams-server.your.domain
ams-server.your.domain has address 192.168.0.1
ams-server.your.domain has address 192.168.0.2
ams-server.your.domain has address 192.168.0.3
ams-server.your.domain has address 192.168.0.4

Finally use IPTables to redirect traffic comming to 192.168.0.[1234]:6779] to <server's real IP address>:<non-standard port>

iptables -t nat -A PREROUTING -d 192.168.0.1 -p tcp -m tcp --dport 6779 -j REDIRECT --to-ports 1992
iptables -t nat -A PREROUTING -d 192.168.0.2 -p tcp -m tcp --dport 6779 -j REDIRECT --to-ports 1993
iptables -t nat -A PREROUTING -d 192.168.0.3 -p tcp -m tcp --dport 6779 -j REDIRECT --to-ports 1994
iptables -t nat -A PREROUTING -d 192.168.0.4 -p tcp -m tcp --dport 6779 -j REDIRECT --to-ports 1995

If you wish to use the objectivity commands to manage the databases locally on the node you will also need to run a extra instance of the AMS server on port 6779 to listen to connections from localhost since IPTables does not seem to redirect these.

Setting up the PUD

Unpack the PUD tarball to /opt/objy/pud

create pud_Linux_<bfsite>.conf based on the RAL one

edit bdbcheckpud, pudc and BdbAdminDaemon to reflect local paths

Use the yet to be released PUD sysV init script to stop and start the services

Setting up the Lockserver

Install the objy8.0.9 distribution into /opt/objy

Use the yet to be released ools sysV init script to stop and start the services

IPTABLES Setup

PUD listens on port 3333, the AMS Server on 6779 and the Lock Server on 6780

iptables -I RH-Firewall-1-INPUT <rulenum> -s <your subnet definiition> -p tcp -m tcp --dport 3333 -j ACCEPT
iptables -I RH-Firewall-1-INPUT <rulenum> -s <your subnet definiition> -p tcp -m tcp --dport 6779 -j ACCEPT
iptables -I RH-Firewall-1-INPUT <rulenum> -s <your subnet definiition> -p tcp -m tcp --dport 6780 -j ACCEPT

--Chris brew 15:09, 26 Oct 2005 (BST)