Difference between revisions of "Setting up a BaBar Objectivity Server"
Chris brew (Talk | contribs) |
(No difference)
|
Latest revision as of 17:25, 10 November 2005
The BaBar Objectivity Server installation consists of three processes, the AMS server, the Lock Server and the PUD.
Contents
Installing AMS
Make Directories and Links
The normal set up has files scattered over server different location on the server, this setup puts all the files under /opt/objy and puts links in so the software can find it's files in the expected places.
The structure under /opt/objy is as below:
- /opt/objy/
- adm/ooss/logs/
- spool/objy/
- etc/ooss/
The links into this tree from elsewhere are:
- /var/adm/ooss &rarr /opt/objy/adm/ooss
- /usr/spool/objy &rarr /opt/objy/spool/objy
- /usr/etc/ooss &rarr /opt/objy/etc/ooss
These should all be owned by the account that the service will run as:
chown -R bfactory:babar objy
Get the Files
Copy the config files from RAL to /opt/objy/etc/ooss
Copy the executables and libraries from /afs/slac.stanford.edu/package/mps/distrib/amscur/i386_rhel30 to /opt/objy/etc/ooss/amscurr
Configuring AMS
Edit StartAMS.cf to point to the correct local paths for certain utilities and to start the service as the correct user
Edit amshpss.cf to point to the local area where the databases will be held.
Increase the number of File Descriptors
Edit /etc/secutiry/limits.conf to include the following lines:
bfactory soft nofile 4096 bfactory hard nofile 50000
Where bfactory should be replaced by the account your running the service as.
Starting and Stopping the Server
To start the server
StartAMS.cf
If it's running corectly you should see:
ps -HU bfactory PID TTY TIME CMD 16256 pts/5 00:00:00 bash 16310 pts/5 00:00:00 ps 16298 pts/5 00:00:00 keeplog 16297 pts/5 00:00:00 ooams
To stop the server
StopAMS.cf
Or use the yet to be released slac-ams sysV init script to stop and start the services
Running Multiple AMS Servers on One Host
A single instance of the SLAC AMS server cannot utilise the full CPU, disk or network bandwidth of a modern host. It therefore pays to run multiple instances of the AMS server to allow more clients to be supported off a single copy of the database.
Each instance of the AMS server has to run on a separate port, then the clients need to be persuaded to talk to the servers on these non-standard ports;
Running the AMS server on a Non-Standard Port
First you need to add the extra ports into /etc/services. e.g.
ooams-3a 1992/tcp ooams-3b 1993/tcp ooams-3c 1994/tcp ooams-3d 1995/tcp
Then repeatedly runnning the StartAMS script with the -s <portname> option will start the service:
StartAMS -s ooams-3a StartAMS -s ooams-3b StartAMS -s ooams-3c StartAMS -s ooams-3d
Making the Clients Talk to the Servers on the Non-Standard Port
There is no client side config option to make the Objectivity client talk to the AMS server on multiple or non-standard ports. There are two ways to fool it one on the client side and one on the server side.
Client Side Config Method
Divide the client machines into N groups where N equals the number of AMS server instances you are running on each server host.
On each group of client machines edit /etc/services to redefine the normal ooams-3 port number from 6779 to one of you non-standard port numbers e.g:
ooams-3 1992/tcp
Choose a different non standard port for each client group. Each group will then talk to a different server instance on the server host.
Server Side Method (only for Linux)
Add N extra IP Addresses to the server machine bound together with a single DNS name then use IPTables to internally NAT between port 6779 on the extra IP Addresses and the non-standard ports you are running your server on.
I.e. if you server uses eth0 create extra ifcfg-eth0:[0-(N-1)] in /etc/sysconfig/network-scripts for N extra IP addresses:
DEVICE=eth0:0 IPADDR=192.168.0.1 NETMASK=255.255.255.0 NETWORK=192.168.0.0 BROADCAST=192.168.0.255 ONBOOT=yes TYPE=Ethernet GATEWAY=192.168.0.254
Then bind the extra IP addresses to a single DNS name.
host ams-server.your.domain ams-server.your.domain has address 192.168.0.1 ams-server.your.domain has address 192.168.0.2 ams-server.your.domain has address 192.168.0.3 ams-server.your.domain has address 192.168.0.4
Finally use IPTables to redirect traffic comming to 192.168.0.[1234]:6779] to <server's real IP address>:<non-standard port>
iptables -t nat -A PREROUTING -d 192.168.0.1 -p tcp -m tcp --dport 6779 -j REDIRECT --to-ports 1992 iptables -t nat -A PREROUTING -d 192.168.0.2 -p tcp -m tcp --dport 6779 -j REDIRECT --to-ports 1993 iptables -t nat -A PREROUTING -d 192.168.0.3 -p tcp -m tcp --dport 6779 -j REDIRECT --to-ports 1994 iptables -t nat -A PREROUTING -d 192.168.0.4 -p tcp -m tcp --dport 6779 -j REDIRECT --to-ports 1995
If you wish to use the objectivity commands to manage the databases locally on the node you will also need to run a extra instance of the AMS server on port 6779 to listen to connections from localhost since IPTables does not seem to redirect these.
Setting up the PUD
Unpack the PUD tarball to /opt/objy/pud
create pud_Linux_<bfsite>.conf based on the RAL one
edit bdbcheckpud, pudc and BdbAdminDaemon to reflect local paths
Use the yet to be released PUD sysV init script to stop and start the services
Setting up the Lockserver
Install the objy8.0.9 distribution into /opt/objy
Use the yet to be released ools sysV init script to stop and start the services
IPTABLES Setup
PUD listens on port 3333, the AMS Server on 6779 and the Lock Server on 6780
iptables -I RH-Firewall-1-INPUT <rulenum> -s <your subnet definiition> -p tcp -m tcp --dport 3333 -j ACCEPT iptables -I RH-Firewall-1-INPUT <rulenum> -s <your subnet definiition> -p tcp -m tcp --dport 6779 -j ACCEPT iptables -I RH-Firewall-1-INPUT <rulenum> -s <your subnet definiition> -p tcp -m tcp --dport 6780 -j ACCEPT
--Chris brew 15:09, 26 Oct 2005 (BST)