Thursday, February 2, 2012

Tomcat [JBoss EWS] Clustering and Session Replication

Share it Please

This article explains you on how to configure a cluster using multiple tomcat instances and perform a session replication between them. We will be using the in-memory session replication in this article. For this demo article I have chosen RHEL 5.4 and Tomcat 6.

In order to configure clustering in tomcat. We follow the sequence,

1. Multicast?
2. Test Whether Your Kernel supports Multicast.
3. Configure Apache Http Server with mod_jk.
4. Configure Multiple Instances of Tomcat.
5. Configure Clustering in Tomcat.
6. Make your application distributable.
7. Session Replication Rules
8. Deploy the sample application.
9. Test.
10. Basic Issues.


1. Multicast?

If we have information that should be sent to various hosts on the internet or LAN, then it is called multicasting (UDP).

This is the opposite of unicast (TCP) where there will be once sender and only one receiver.

As we know, the IP address are divided into classes based on the high order of 32 bits, like

0                           31                               Address Range:
0|                    Class A Address                 0.0.0.0 - 127.255.255.255
|1 0|                Class B Address                 128.0.0.0 - 191.255.255.255
|1 1 0|             Class C Address                 192.0.0.0 - 223.255.255.255
|1 1 1 0|          MULTICAST Address            224.0.0.0 - 239.255.255.255
|1 1 1 1 0|       Reserved                            240.0.0.0 - 247.255.255.255

Class D addresses (224.0.0.0) are said to Used for multicast. The net mask is a 32 bit number with 
network part being filled with all “1” and host part with “0”.

2. Test Whether Your Kernel supports Multicast.

Once we are aware of multi casting, we need to make sure that our kernel (or network interface) supports multicasting. In order to find this, run
/sbin/ifconfig

And output may be something like this

eth0    Link encap:Ethernet  HWaddr 00:21:9B:F3:48:4D 
          inet addr:xxx.xx.xxx.xx  Bcast:183.82.191.255  Mask:255.255.192.0
          inet6 addr: fe80::221:9bff:fef3:484d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500  Metric:1
          RX packets:23231 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30417 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:19979434 (19.0 MiB)  TX bytes:4609420 (4.3 MiB)
          Interrupt:169

Note the MULTICAST attribute in the third line of the eth0 properties. If this is not present, it's possible that your kernel has not been compiled with multicast support.

Once we are sure that our kernel supports multicasting the next step is to enable multicast routing and this can be done by,

route add  224.0.0.0 netmask 240.0.0.0 dev eth0

Once this is done. You can make sure to ping the ip address, like

ping 224.0.0.8

You can also make sure about the multicasting using

[root@vx111a bin]# route -n
Kernel IP routing table
Destination       Gateway        Genmask            Flags  Metric Ref    Use     Iface
183.82.128.0  0.0.0.0           255.255.192.0   U         0        0        0      eth0
169.254.0.0    0.0.0.0           255.255.0.0       U         0        0        0      eth0
224.0.0.0       0.0.0.0          240.0.0.0          U         0        0        0     eth0
0.0.0.0            183.82.128.1 0.0.0.0               UG       0        0        0       eth0

Or

[root@vx111a bin]# netstat -nr
Kernel IP routing table
Destination         Gateway         Genmask            Flags    MSS      Window        irtt      Iface
183.82.128.0    0.0.0.0            255.255.192.0    U         0            0                0      eth0
169.254.0.0      0.0.0.0            255.255.0.0        U          0           0                0      eth0
224.0.0.0         0.0.0.0             240.0.0.0           U          0            0                0      eth0
0.0.0.0             183.82.128.1    0.0.0.0              UG        0            0                0      eth0

Or

[root@vx111a bin]# netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      all-systems.mcast.net
eth0            2      228.0.0.8
eth0            1      224.0.0.251
eth0            1      all-systems.mcast.net
lo              1      ff02::1
eth0            1      ff02::fb
eth0            1      ff02::1:fff3:484d
eth0            1      ff02::1


In future if you want to disable multicast, just drop the IP from Routes Table:

route -v delete -inet 228.0.0.4

3. Configure Apache Http Server with mod_jk.

The 3 step is to configure http Server with mod_jk.The article can be found here.
The configuration that I done on my desktop is,

conf/worker.properties

#
#Worker.Properties
#

worker.list=web1, web2, loadbalancer

#
#First Server Details
#

worker.web1.port=11009
worker.web1.host=183.82.144.26 (make sure you change your ip here)
worker.web1.type=ajp13
worker.web1.lbfactor=100

#
#Second Server Details
#

worker.web1.port=12009
worker.web1.host=183.82.137.30 (make sure you change your ip here)
worker.web1.type=ajp13
worker.web1.lbfactor=100

#
#LoadBalancer Worker
#

worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=web1, web2
worker. loadbalancer.sticky_session=1

#
#End Of the Properties File
#

conf/ mod_jk.conf

#
# Mod_Jk Configuration
#

LoadModule jk_module modules/mod_jk.so

<IfModule mod_jk.c>
    JkWorkersFile conf/workers.properties
    JkLogFile logs/mod_jk.log
    JkLogLevel error
    JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
    JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
    JkRequestLogFormat "%w %V %T"

    JkMount /*.jsp loadbalancer
    JkMount /servlet/* loadbalancer


</IfModule>

#
# Done
#


In httpd.conf, add the following line at the bottom,

Include conf/mod_jk.conf

By this the configuration on the Http Server is done. We have now configured http server with mod_jk (load balancing support)

4. Configure Multiple Instances of Tomcat.

The next step is configuring tomcat instances. The article for configuring multiple instances of tomcat can be found here or.

Download the Tomcat to a location

Export the CATALINA_HOME and CATALINA_BASE.Add these to the /root/.bashrc

export CATALINA_HOME=/usr/jboss-ews-1.0/tomcat6;
export CATALINA_BASE=/usr/jboss-ews-1.0/tomcat6;

Once added, use source command to update, source /root/.bashrc

Now create 2 directories in other location (I have create tmc1 and tmc2 in /usr/local)

Once both locations are created, copy the directories bin, conf, lib, logs, temp, webapps, and work.

In bin directory, remove all the files except, startup.sh and shutdown.sh (if windows keep their respective bat files)

Clean startup.sh and shutdown.sh and update with the following

startup.sh

export CATALINA_BASE=/usr/local/tmc2
export CATALINA_HOME=/usr/jboss-ews-1.0/tomcat6
/usr/jboss-ews-1.0/tomcat6/bin/startup.sh

And shutdown.sh

export CATALINA_BASE=/usr/local/tmc2
export CATALINA_HOME=/usr/jboss-ews-1.0/tomcat6
/usr/jboss-ews-1.0/tomcat6/bin/shutdown.sh

Once these changes are done, go to the conf location and modify the server.xml file according to your port needs. The 2 main ports to change in server.xml are shutdown port and http port. It would be good if we change all ports.


Now we have 2 instances of tomcat running. Now for the Clustering we need to modify the ports like

These are the changes that I did in the both tomcat instances

                   Shutdown      Http              Connector               jvmRoute
Tomcat 1       8005             8080             11009                     web1
Tomcat 2       8105             8180             12009                     web2

These changes are to be done in server.xml files in Tomcat (1,2)/conf/.These changes should be same as we configured in worker.properties file(port, jvmRoute…).

The JvmRoute attribute element of the Engine element allows the load balancer to match request to the exact jvm responsible for updating the session. This is done by appending the name of the jvm to the JSESSIONID of the request and matching this against the worker name provided in the worker.properties file

By this we are done with configuring multiple instances of tomcat. The next step is to configure cluster

5. Configure Clustering in Tomcat.

For configuring cluster we need to add the Cluster element in the Engine element like

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
              channelSendOptions="8">
      
  <Manager className="org.apache.catalina.ha.session.DeltaManager"
                   expireSessionsOnShutdown="false"
                   notifyListenersOnReplication="true"/>
  <Channel className="org.apache.catalina.tribes.group.GroupChannel">

  <Membership className="org.apache.catalina.tribes.membership.McastService"
                           address="228.0.0.8"
                           port="45564"
                           frequency="500"
                           dropTime="3000"/>
 
  <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                         address="auto"
                         port="4000"
                         autoBind="100"
                         selectorTimeout="5000"
                         maxThreads="6"/>
  <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
  <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
  </Sender>
 
 <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>

</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
                  filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.css;.*\.txt;"/>

<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>

 </Cluster>

This element is to be add under the

<Engine name="Catalina" defaultHost="localhost" jvmRoute="web1">

Add the cluster element to tomcat{1,2}/conf/server.xml file and restart the tomcat instances.

Since we are going for a In-Memory session replication, We need to configure 2 elemets

Cluster: this element is responsible for the actual session replication which includes sending the new session information to the group, taking incoming session information and also managing group’s membership.

Replication: The other element is the replication element which helps in reducing the session data by filtering certain request from session replication

SimpleTcpCluster is the implementation available in tomcat for In-Memory replication. This uses apache tribes for communication. This uses heartbeat mechanism to determine the group memberships.

The SimpleTcpCluster provides us with 2 managers

DeltaManager : replicate sessions across all tomcat instances
BackupManager : replicate session from master to a backup server

We can configure a manager depending on our needs.

SimpleTcpCluster uses Apache Tribes for communication between the servers or groups. Membership is established and maintained by Apache Tribles.It also handles server crash and recovery. This also offers guaranteed message delivery between members. This also carries the task of replicating the session data to all members.

6. Make Your Applications Distributable

In order to work with your application in a clustered environment, our applications need to add <distributable/> element to the web.xml file.

Your other option is to simply add the "distributable" attribute to the relevant application's Context element, as follows:

<Context distributable="true">

7. Session Replication Rules

Session attributes must implement java.io.Serializable.

HttpSession.setaAttribute() must be called any time changes are made to an object that belongs to a session, so that the session replicator can distribute the changes across the cluster.

The sessions must not be so big that they overload the server with traffic when copied.

Deploy the sample application.

Now the Cluster configuration is done, we need to deploy a sample application.

Create a web application. Download the following jsp file from here and add to the web application. Make sure you add <distributable/> element to the web.xml file

7. Test.


Stop one Tomcat server and access the url , it should still show the updated counter value.
Check the jsp file for more.

8. Basic Issues.

Some Tomcat instances are missing with tribes.jar file. Download catalina-tribes.jar file from internet.







 







No comments :

Post a Comment