Introduction
There are quite a few articles which describes the theoretical aspects of load balancing, among them [1] and [2] are good read before going through this tutorial.
This tutorial shows the steps you need to follow in order to front a WSO2 Application Server 5.0 cluster with a WSO2 Elastic Load Balancer 2.0 (WSO2 Elastic Load Balancer). Note that though I am only going to configure a WSO2 Application Server cluster in this tutorial, you can follow the same instructions to configure any WSO2 Carbon 4.0 based product/service. The following diagram gives an idea about the setup.
As the above image depicts, a client will send out a request to the WSO2 Application Server through WSO2 Load Balancer, and it is the Load Balancer who decides where the request should be headed to, i.e. to which service domain, to which node in the chosen service domain.
First let's look at configurations that are needed to be done in WSO2 Application Server 5.0 pack.
Configuration changes to WSO2 Application Server 5.0
Edits to “{$Carbon-Home}/repository/conf/axis2/axis2.xml”
1. Set “enable” attribute of the “clustering” element to “true”. This is needed mainly for membership discovery, since service nodes join WSO2 Elastic Load Balancer using Apache Axis2's clustering mechanism.
<!-- ================================================= -->
<!-- Clustering -->
<!-- ================================================= -->
<!--
To enable clustering for this node, set the value of "enable" attribute of the "clustering"
element to "true". The initialization of a node in the cluster is handled by the class
corresponding to the "class" attribute of the "clustering" element. It is also responsible for
getting this node to join the cluster.
-->
<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">
2. Under clustering element there are several things to edit. Please refer to following set of pictures, to identify them.
Note: “domain” parameter's value should be added in load balancer's configuration file.
<!--
The membership scheme used in this setup. The only values supported at the moment are
"multicast" and "wka"
1. multicast - membership is automatically discovered using multicasting
2. wka - Well-Known Address based multicasting. Membership is discovered with the help
of one or more nodes running at a Well-Known Address. New members joining a
cluster will first connect to a well-known node, register with the well-known node
and get the membership list from it. When new members join, one of the well-known
nodes will notify the others in the group. When a member leaves the cluster or
is deemed to have left the cluster, it will be detected by the Group Membership
Service (GMS) using a TCP ping mechanism.
-->
<parameter name="membershipScheme">wka</parameter>
<!--
The clustering domain/group. Nodes in the same group will belong to the same multicast
domain. There will not be interference between nodes in different groups.
-->
<parameter name="domain">wso2.as.domain</parameter>
“localMemberPort” should be a valid port other than 4000. Preferably something in the range 4000-5000. Also there should be a unique port for all the service nodes.
<!-- The host name or IP address of this member -->
<!--
<parameter name="localMemberHost">127.0.0.1</parameter>
-->
<!--
The TCP port used by this member. This is the port through which other nodes will
contact this member
-->
<parameter name="localMemberPort">4100</parameter>
Following picture depicts well-known members. “hostName” should be mapped to public IP of WSO2 Load Balancer and “port” to WSO2 Load Balancer's “localMemberPort”. You can add the relevant entry in /etc/hosts file.
<!--
The list of static or well-known members. These entries will only be valid if the
"membershipScheme" above is set to "wka"
-->
<members>
<member>
<hostName>appserver.cloud-test.wso2.com</hostName>
<port>4000</port>
</member>
<!--member>
<hostName>127.0.0.1</hostName>
<port>4001</port>
</member-->
</members>
Under transportReceiver element add proxyPort parameter, and its value should be the http port value of the WSO2 Load Balancer. What we are doing here is, we are going to proxy the WSO2 Application Server via WSO2 Elastic Load Balancer's ports.
<!-- ================================================= -->
<!-- In Transports -->
<!-- ================================================= -->
<transportReceiver name="http"
class="org.wso2.carbon.core.transports.http.HttpTransportListener">
<!--
Uncomment the following if you are deploying this within an application server. You
need to specify the HTTP port of the application server
-->
<parameter name="port">9763</parameter>
<!--
Uncomment the following to enable Apache2 mod_proxy. The port on the Apache server is 80
in this case.
-->
<parameter name="proxyPort">8290</parameter>
</transportReceiver>
Above same rule applies to https transportReceiver as well.
<transportReceiver name="https"
class="org.wso2.carbon.core.transports.http.HttpsTransportListener">
<!--
Uncomment the following if you are deploying this within an application server. You
need to specify the HTTPS port of the application server
-->
<parameter name="port">9443</parameter>
<!--
Uncomment the following to enable Apache2 mod_proxy. The port on the Apache server is 443
in this case.
-->
<parameter name="proxyPort">8243</parameter>
</transportReceiver>
Now the WSO2 Application Server is configured to be fronted by WSO2 Load Balancer. Let's look at configurations needed to be done on WSO2 Elastic Load Balancer's side.
Configuration changes to WSO2 Elastic Load Balancer 2.0
Changes to loadbalancer.conf file
Configuration file of WSO2 Load Balancer has been changed from loadbalancer.xml to loadbalancer.conf file which is in Nginx format [3], in the 2.0 release. Following is a sample of loadbalancer.conf configuration file.
#configuration details of WSO2 Load Balancer
loadbalancer {
# minimum number of load balancer instances
instances 1;
# whether autoscaling should be enabled or not, currently we do not support autoscaling
enable_autoscaler false;
# End point reference of the Autoscaler Service
# autoscaler_service_epr https://10.100.3.104:9445/services/AutoscalerService/;
# interval between two task executions in milliseconds
# autoscaler_task_interval 15000;
}
# services' details which are fronted by this WSO2 Load Balancer
services {
# default parameter values to be used in all services
defaults {
min_app_instances 1;
max_app_instances 5;
queue_length_per_node 3;
rounds_to_average 2;
instances_per_scale_up 1;
message_expiry_time 60000;
}
appserver {
# multiple hosts should be separated by a comma.
hosts appserver.cloud-test.wso2.com,as.cloud-test.wso2.com;
domains {
wso2.as1.domain {
tenant_range 1-100;
}
wso2.as2.domain {
tenant_range 101-200;
}
wso2.as.domain {
# all tenants other than 1-200 will belong to this domain.
tenant_range *;
}
}
}
esb {
# multiple hosts should be separated by a comma.
hosts esb.cloud-test.wso2.com;
domains {
wso2.esb.domain {
tenant_range *;
}
}
}
}
In the above loadbalancer.conf file's, under services section, you should specify the services which are going to be fronted using WSO2 Elastic Load Balancer. In this tutorial we are going to front a WSO2 Application Server, hence the appserver element (it can be any name) has been included. There I have specified 2 hosts for this service separated by a comma (,) as the values of hosts attribute.
Under each service there's a domains section, where you specify the list of service domains and their tenant ranges (this is introduced by the new tenant aware feature of WSO2 Elastic Load Balancer). If you do not want to make WSO2 Elastic Load Balancer tenant aware, you could simply set tenant_range's value to *.
As you can see in the above picture of loadbalancer.conf file, I have added the wso2.as.domain under domains of “appserver”, which is exactly what we added as the domain name of our WSO2 Application Server instance's axis2.xml file.
Edits to “{$Carbon-Home}/repository/conf/axis2/axis2.xml”
We need to change the ports of the WSO2 Elastic Load Balancer, to the ports what we have added as the proxy ports of the WSO2 Application Server i.e. http port = 8290 and https port = 8243.
<!-- ================================================= -->
<!-- Transport Ins (Listeners) -->
<!-- ================================================= -->
<transportReceiver name="http" class="org.wso2.carbon.transport.passthru.PassThroughHttpListener">
<parameter name="port">8290</parameter>
<parameter name="non-blocking"> true</parameter>
</transportReceiver>
<transportReceiver name="https" class="org.wso2.carbon.transport.passthru.PassThroughHttpSSLListener">
<parameter name="port" locked="false">8243</parameter>
<parameter name="non-blocking" locked="false">true</parameter>
......
Testing...
All good now, you can now start the WSO2 Elastic Load Balancer by executing wso2server.sh file which is resided in “{$Carbon-Home}/bin/” folder. In the logs that are printed at the startup, you should see a line like following appearing.
Above line basically means that WSO2 Elastic Load Balancer is ready to front Carbon instances which are configured to be in “wso2.as.domain” service domain.
Now you can start the WSO2 Application Server instance which we have configured. Note that if you're running both instances in a single machine, start WSO2 Application Server with a port offset i.e. by using following command:
“./{$Carbon-Home}/bin/wso2server.sh -DportOffset=5”
This will make sure that there are no port conflicts between two Carbon instances.
In the WSO2 Application Server startup logs, you should see following line. Note that the DNS is mapped to the public IP of the WSO2 Elastic Load Balancer as I have added an entry in my /etc/hosts.
Once WSO2 Application Server started up, you should see logs similar to following are appearing in WSO2 Load Balancer's logs.
Now, you should be able to successfully access the management console of the WSO2 Application Server, through the host name you have provided in loadbalancer.conf (note that you have to map this host name to the public IP of the WSO2 Elastic Load Balancer) i.e. in my case it is:
https://appserver.cloud-test.wso2.com:8243/
Similarly you could front multiple service nodes, domains and multiple services (WSO2 ESB etc.) using WSO2 Elastic Load Balancer 2.0.
Conclusion
In summary you need to do changes in following configuration files, in order to front Carbon 4.0 based products/services using WSO2 Elastic Load Balancer 2.0.
In “{$Carbon-Home}/repository/conf/axis2/axis2.xml” file of Carbon 4.0 based instance, you have to enable clustering, set “wka” as the membership scheme, set the “domain name” to which this Carbon instance belongs to, then change local member port to something other than that of Load Balancer (usually 4000), then add Load Balancer as a well-known member under members and change both http and https transport receivers’ proxy ports to the http, https ports of Load Balancer.
In “{$Carbon-Home}/repository/conf/loadbalancer.conf” file of WSO2 Elastic Load Balancer 2.0, you should basically add the service domains which you want to be fronted. Also in “{$Carbon-Home}/repository/conf/axis2/axis2.xml” file, make sure you have correctly specified the ports of http and https transport receivers.
If you use DNSs make sure to add entries in /etc/hosts file to reflect the IP-to-DNS mappings.
That's it! Happy Load Balancing !! :-)
References
[1] Role of a load balancer in PaaS
[2] How WSO2 load balancer works
[3] Nginx configuration file example
