I'm happy to state that the months of hard work of Stratos2 team is about to payoff, as we at WSO2 think that the revamped WSO2 Stratos (i.e. WSO2 Stratos 2.0.0) has come to a saturation point for its first general availability release.
Here's the release note...
WSO2 Stratos 2.0.0 Foundation is Released WSO2 Stratos2 team is pleased to announce the general availability (GA) of WSO2 Stratos 2.0.0 Foundation. WSO2Stratos2.0.0 comes with an easy to configure demo setup that can be run on Amazon EC2. Please refer to the Quick Start Guide of the Getting Startedsection for more information.
Following table lists down the Stratos 2.0.0 AWS EC2 AMIs available in respective regions.
EC2 Image
Asia Pacific (Singapore) Region
US East - 1 (N. Virginia) Region
Stratos 2.0
ami-c3ca8491
ami-dfb8cfb6
Stratos2.0.0 Command Line Interface (CLI) Tool is available here to download and Stratos2.0.0 Wiki Documentation is now publicly available.
WSO2Stratos2.0.0 is the next major version of WSO2Stratos 1.x, the most complete, enterprise-grade, open PaaS, with
support for more core services than any other available PaaS today.
Following are the Key features available in Stratos2.0.0.
Key Features
Plug-able architecture allows you to add support for new Cartridges, easily.
Support for PHP, Tomcat and MySQL and WSO2 Carbon cartridges (AS, ESB, BPS etc.)
Support for puppet based cartridge creation for WSO2 Carbon cartridges.
Artifact Distribution Coordinator (ADC) with support for external Git and GitHub repositories.
Tenant-Aware Elastic Load Balancer (ELB).
Cloud Controller (CC) provides support for multiple IaaSes (EC2, OpenStack, vCloud) through jclouds APIs.
Cloud Controller (CC) can be easily extended to support any IaaS that jclouds supports.
Policy based Auto-scaling allows you to calibrate the frequency of scale ups/downs.
Git-based Deployment Synchronizer, synchronizes artifacts across all the nodes in your service cluster.
Interactive CLI Tool and a Graphical User Interface, for your tenants to perform various operations on Stratos2 Foundation.
Custom domain mapping support for your Cartridge subscriptions.
Demo purpose internal GIT repository support.
Usage metering and Billing.
Thoroughly written Wiki Documentation (User Guide, Architecture Guide etc.)
Demo Ready, Public, AWS EC2 Stratos2.0 setup.
Installing
WSO2 Stratos 2.0.0 installation scripts are available to be downloaded
for AWS EC2 IaaS and OpenStack IaaS, from our product page http://wso2.com/cloud/stratos/
WSO2
Inc. offers a variety of development and production support programs,
ranging from Web-based support up through normal business hours, to
premium 24x7 phone support. For additional support information please
refer to http://wso2.com/support. For more information on WSO2 Stratos 2.0.0 release please visit http://wso2.com/cloud/stratos/.
When your JAVA process started to spin your CPU, you need to immediately issue following two commands and get the invaluable information required to tackle the issue.
After getting those two files, what you can do is,
1. find the thread ID (which belongs to the corresponding PID ) which takes the highest CPU usage by examine thread-usage.txt file.
%CPU CPU NI S TIME PID TID
..........
0.0 - 0 S 00:00:00 1519 1602
0.0 - 0 S 00:00:00 1519 1603 24.8 - 0 R 00:06:19 1519 1604
2.4 - 0 S 00:00:37 1519 1605
0.0 - 0 S 00:00:00 1519 1606
..........
2. convert the decimal value (in this case 1604) to hexadecimal - (online converter - http://easycalculation.com/decimal-converter.php)
Hex - 644
3. search for the hexadecimal obtained (in this case 644) in the thread-dump.txt (thread-dump.txt should have that value as a thread id of one thread) and that is the thread which spins.
4. that thread usually has a stack trace, and that's the lead to find the culprit.
In this case the stack trace of the thread that spins is:
"HTTPS-Sender I/O dispatcher-1" prio=10 tid=0x00007fb54c010000 nid=0x644 runnable [0x00007fb534e20000]
java.lang.Thread.State: RUNNABLE
at org.apache.http.impl.nio.reactor.IOSessionImpl.getEventMask(IOSessionImpl.java:139)
- locked <0x00000006cd91fef8> (a org.apache.http.impl.nio.reactor.IOSessionImpl)
at org.apache.http.nio.reactor.ssl.SSLIOSession.updateEventMask(SSLIOSession.java:300)
at org.apache.http.nio.reactor.ssl.SSLIOSession.inboundTransport(SSLIOSession.java:402)
- locked <0x00000006cd471df8> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Thread.java:722)
I wrote a WSO2 Wiki article on explaining the minimum configuration instructions required to configure WSO2 ESB in a distributed setup with separated nodes as management node and worker node/s.
Shown below is the deployment diagram of this setup. The cluster consists of two sub cluster domains as worker/manager and is fronted by a single load balancer. Altogether, we will be configuring three service instances.
WSO2 Stratos-2.0.0 Beta-1 is Released - 11th February 2013!
WSO2 Stratos2 team is pleased to announce the release of WSO2 Stratos 2.0.0 Beta-1 version.
WSO2 Stratos 2.0.0 Beta-1 is now available for
download at [1] and the documentation is available at [2]. Stratos 2.0.0 Beta-1 installed, Oracle VirtualBox image, is downloadable at [3].
WSO2 Application Server Cartridge
(LXC based), which is required at the run-time of the Stratos 2.0.0 Beta-1 VirtualBox image, is downloadable
at [4].
WSO2 Stratos 2.0.0 is the next major version of WSO2 Stratos 1.x, the most complete, enterprise-grade, open PaaS, with
support for more core services than any other available PaaS today. Key Features
Artifact Distribution Coordinator (ADC) with Git and Git-hub integration support
Plug-able architecture support for adding new cartridges
PHP and MySQL and WSO2 carbon cartridges(ESB, AS etc) support
Elastic Load Balancer (ELB) with Cartridge support
Autoscaling into different IaaSes (EC2, Openstack)
S2 Cloud Controller
Multiple IaaS support (EC2, Openstack) through jclouds API's
Git based deployment synchronizer
Interactive CLI for tenants to manage subscriptions
UI for tenants to manage subscriptions
Custom domain mapping support
Script based Multi-node Installer
Local deployment setup
Examples
Documentation(Stratos2
Installation Guide, User Guide, Architecture Guide, Cartridge
Development Guide and Openstack Installation Guide)
In a distributed system, ability to expand or contract its resource pool is defined as scalability. A system can be scaled in two modes, horizontal and vertical. What we are interested in is horizontal scaling which is adding more nodes to a clustered distributed system.
In this article, you will learn the auto-scaling algorithm used in WSO2 Elastic Load Balancer, few tips you should keep in mind when calibrating auto-scaling decision making variables and also a brief explanation on a sample scenario.
What is auto-scaling?
When there is a sudden peak of requests coming to an application, we should ideally increase the amount of resources we have provided for that application. There comes a solution call auto-scaling. In an auto-scaling enabled system, system itself should detect such peaks and start-up new server instances, to cater the requirements, without any manual interception.
With the revolutionization of Cloud, today we can easily start new instances and terminate already existing instances at any given moment, that makes auto-scaling a possibility in a Cloud environment.
Where does this autoscale decision making task reside?
The ‘autoscaling decision making’ task currently resides in WSO2 Elastic Load Balancer. Default implementation is “org.wso2.carbon.mediator.autoscale.lbautoscale.task.ServiceRequestsInFlightAutoscaler”.
What is the basis for autoscaling?
Current default implementation (ServiceRequestsInFlightAutoscaler) considers number of requests in-flight as the basis for making autoscaling decisions. We follow the paradigm; “scale up early and scale down slowly” in the default algorithm.
What are the decision making variables?
There are few of them and all of the vital ones are configurable using loadbalancer.conf file. (sample configuration files are provided at the end of this document.)
autoscaler_task_interval (t) - time period between two iterations of ‘autoscaling decision making’ task. When configuring this value, you are advised to consider the time ‘that a service instance takes to join ELB’. This is in milliseconds and the default value is 30000ms.
max_requests_per_second (Rps) - number of requests, a service instance can withstand per a second. It is recommended that you calibrate this value for each service instance and may also for different scenarios. Ideal way to estimate this value could be by load testing a similar service instance. Default value is 100.
rounds_to_average (r) - an autoscaling decision will be made only after this much of iterations of ‘autoscaling decision making’ task. Default value is 10.
alarming_upper_rate (AUR)- without waiting till the service instance reach its maximum request capacity (alarming_upper_rate = 1), we scale the system up when it reaches the request capacity, corresponds to alarming_upper_rate. This value should be 0
alarming_lower_rate (ALR) - lower bound of the alarming rate, which gives us a hint; that we can think of scaling down the system. This value should be 0
scale_down_factor (SDF) - this factor is needed in order to make the scaling down process slow. We need to scale down slowly to reduce scaling down due to a false-positive event. This value should be 0
How does the number of requests in-flight gets calculated?
We keep track of the requests that come to Elastic Load Balancer (ELB) for various service clusters. For each incoming request, we add a token, against the relevant service cluster and when the message left ELB or got expired, we remove the corresponding token.
What are the decision making functions?
We always respect the minimum number of instances value and maximum number of instances value of service clusters. We make sure that the system always maintains the minimum number of service instance requirement and also system will not scale beyond its limit.
We calculate,
average requests in-flight for a particular service cluster (avg) =
total number of requests in-flight * (1/r)
Scaling up....
number of maximum requests that a service instance can withstand over an autoscaler task interval (maxRpt) =
(Rps) * (t/1000) * (AUR)
then, we decide to scale up, if,
avg > maxRpt * (number of running instances of this service cluster)
Scaling down....
imaginary lower bound value (minRpt) =
(Rps) * (t/1000) * (ALR) * (SDF)
then, we decide to scale down, if,
avg < minRpt * (number of running instances of this service cluster - 1)
Can I plug my own implementation?
You can write your own Java implementation which implements org.apache.synapse.task.Task and org.apache.synapse.ManagedLifecycle interfaces. Wrap the implementation class to an OSGi bundle and deploy in WSO2 ELB. Then, point to that class from the {ELB_HOME}/repository/conf/loadbalancer.conf file’s loadbalancer section as follows.
