When your JAVA process started to spin your CPU, you need to immediately issue following two commands and get the invaluable information required to tackle the issue.
After getting those two files, what you can do is,
1. find the thread ID (which belongs to the corresponding PID ) which takes the highest CPU usage by examine thread-usage.txt file.
%CPU CPU NI S TIME PID TID
..........
0.0 - 0 S 00:00:00 1519 1602
0.0 - 0 S 00:00:00 1519 1603 24.8 - 0 R 00:06:19 1519 1604
2.4 - 0 S 00:00:37 1519 1605
0.0 - 0 S 00:00:00 1519 1606
..........
2. convert the decimal value (in this case 1604) to hexadecimal - (online converter - http://easycalculation.com/decimal-converter.php)
Hex - 644
3. search for the hexadecimal obtained (in this case 644) in the thread-dump.txt (thread-dump.txt should have that value as a thread id of one thread) and that is the thread which spins.
4. that thread usually has a stack trace, and that's the lead to find the culprit.
In this case the stack trace of the thread that spins is:
"HTTPS-Sender I/O dispatcher-1" prio=10 tid=0x00007fb54c010000 nid=0x644 runnable [0x00007fb534e20000]
java.lang.Thread.State: RUNNABLE
at org.apache.http.impl.nio.reactor.IOSessionImpl.getEventMask(IOSessionImpl.java:139)
- locked <0x00000006cd91fef8> (a org.apache.http.impl.nio.reactor.IOSessionImpl)
at org.apache.http.nio.reactor.ssl.SSLIOSession.updateEventMask(SSLIOSession.java:300)
at org.apache.http.nio.reactor.ssl.SSLIOSession.inboundTransport(SSLIOSession.java:402)
- locked <0x00000006cd471df8> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Thread.java:722)
I wrote a WSO2 Wiki article on explaining the minimum configuration instructions required to configure WSO2 ESB in a distributed setup with separated nodes as management node and worker node/s.
Shown below is the deployment diagram of this setup. The cluster consists of two sub cluster domains as worker/manager and is fronted by a single load balancer. Altogether, we will be configuring three service instances.
WSO2 Stratos-2.0.0 Beta-1 is Released - 11th February 2013!
WSO2 Stratos2 team is pleased to announce the release of WSO2 Stratos 2.0.0 Beta-1 version.
WSO2 Stratos 2.0.0 Beta-1 is now available for
download at [1] and the documentation is available at [2]. Stratos 2.0.0 Beta-1 installed, Oracle VirtualBox image, is downloadable at [3].
WSO2 Application Server Cartridge
(LXC based), which is required at the run-time of the Stratos 2.0.0 Beta-1 VirtualBox image, is downloadable
at [4].
WSO2 Stratos 2.0.0 is the next major version of WSO2 Stratos 1.x, the most complete, enterprise-grade, open PaaS, with
support for more core services than any other available PaaS today. Key Features
Artifact Distribution Coordinator (ADC) with Git and Git-hub integration support
Plug-able architecture support for adding new cartridges
PHP and MySQL and WSO2 carbon cartridges(ESB, AS etc) support
Elastic Load Balancer (ELB) with Cartridge support
Autoscaling into different IaaSes (EC2, Openstack)
S2 Cloud Controller
Multiple IaaS support (EC2, Openstack) through jclouds API's
Git based deployment synchronizer
Interactive CLI for tenants to manage subscriptions
UI for tenants to manage subscriptions
Custom domain mapping support
Script based Multi-node Installer
Local deployment setup
Examples
Documentation(Stratos2
Installation Guide, User Guide, Architecture Guide, Cartridge
Development Guide and Openstack Installation Guide)
In a distributed system, ability to expand or contract its resource pool is defined as scalability. A system can be scaled in two modes, horizontal and vertical. What we are interested in is horizontal scaling which is adding more nodes to a clustered distributed system.
In this article, you will learn the auto-scaling algorithm used in WSO2 Elastic Load Balancer, few tips you should keep in mind when calibrating auto-scaling decision making variables and also a brief explanation on a sample scenario.
What is auto-scaling?
When there is a sudden peak of requests coming to an application, we should ideally increase the amount of resources we have provided for that application. There comes a solution call auto-scaling. In an auto-scaling enabled system, system itself should detect such peaks and start-up new server instances, to cater the requirements, without any manual interception.
With the revolutionization of Cloud, today we can easily start new instances and terminate already existing instances at any given moment, that makes auto-scaling a possibility in a Cloud environment.
Where does this autoscale decision making task reside?
The ‘autoscaling decision making’ task currently resides in WSO2 Elastic Load Balancer. Default implementation is “org.wso2.carbon.mediator.autoscale.lbautoscale.task.ServiceRequestsInFlightAutoscaler”.
What is the basis for autoscaling?
Current default implementation (ServiceRequestsInFlightAutoscaler) considers number of requests in-flight as the basis for making autoscaling decisions. We follow the paradigm; “scale up early and scale down slowly” in the default algorithm.
What are the decision making variables?
There are few of them and all of the vital ones are configurable using loadbalancer.conf file. (sample configuration files are provided at the end of this document.)
autoscaler_task_interval (t) - time period between two iterations of ‘autoscaling decision making’ task. When configuring this value, you are advised to consider the time ‘that a service instance takes to join ELB’. This is in milliseconds and the default value is 30000ms.
max_requests_per_second (Rps) - number of requests, a service instance can withstand per a second. It is recommended that you calibrate this value for each service instance and may also for different scenarios. Ideal way to estimate this value could be by load testing a similar service instance. Default value is 100.
rounds_to_average (r) - an autoscaling decision will be made only after this much of iterations of ‘autoscaling decision making’ task. Default value is 10.
alarming_upper_rate (AUR)- without waiting till the service instance reach its maximum request capacity (alarming_upper_rate = 1), we scale the system up when it reaches the request capacity, corresponds to alarming_upper_rate. This value should be 0
alarming_lower_rate (ALR) - lower bound of the alarming rate, which gives us a hint; that we can think of scaling down the system. This value should be 0
scale_down_factor (SDF) - this factor is needed in order to make the scaling down process slow. We need to scale down slowly to reduce scaling down due to a false-positive event. This value should be 0
How does the number of requests in-flight gets calculated?
We keep track of the requests that come to Elastic Load Balancer (ELB) for various service clusters. For each incoming request, we add a token, against the relevant service cluster and when the message left ELB or got expired, we remove the corresponding token.
What are the decision making functions?
We always respect the minimum number of instances value and maximum number of instances value of service clusters. We make sure that the system always maintains the minimum number of service instance requirement and also system will not scale beyond its limit.
We calculate,
average requests in-flight for a particular service cluster (avg) =
total number of requests in-flight * (1/r)
Scaling up....
number of maximum requests that a service instance can withstand over an autoscaler task interval (maxRpt) =
(Rps) * (t/1000) * (AUR)
then, we decide to scale up, if,
avg > maxRpt * (number of running instances of this service cluster)
Scaling down....
imaginary lower bound value (minRpt) =
(Rps) * (t/1000) * (ALR) * (SDF)
then, we decide to scale down, if,
avg < minRpt * (number of running instances of this service cluster - 1)
Can I plug my own implementation?
You can write your own Java implementation which implements org.apache.synapse.task.Task and org.apache.synapse.ManagedLifecycle interfaces. Wrap the implementation class to an OSGi bundle and deploy in WSO2 ELB. Then, point to that class from the {ELB_HOME}/repository/conf/loadbalancer.conf file’s loadbalancer section as follows.
One rarely get a chance to release a product he is working on, on his Birthday. I'm lucky enough (Usually luck doesn't favour me :-(), to get such a chance.
WSO2 Stratos2 alpha was released on 19th December 2012, ya, that is my birthday.
I was mainly working on WSO2 Stratos-2.0 Cloud Controller, ELB etc. And I will elaborate on Cloud Controller in my future blog posts.
This is a continuation of my series of posts on WSO2 Autoscaler Service. If you missed the part-I, please visit here. As I mentioned there, in this post I will show how we secure the confidential information specified in the configuration file.
How to use WSO2 Secure Vault to secure your confidential data?
WSO2 Secure-vault can be used to hide your confidential data, from been appearing in the configuration files as plain text. In WSO2 Autoscaler service's configuration file i.e. elastic-scaler-config.xml file, we are securing the confidential information such as the identity and credential for accessing your account on an IaaS provider.
I will go through the steps you need to follow in order to secure an example property value.
In elastic-scaler-config.xml we have an element called “identity” at “elasticScalerConfig/iaasProviders/iaasProvider[@type='ec2']/identity”. Following is the exact element structure.
Note that you don't need to provide your identity for a particular IaaS (EC2 in the case of the example) here, as plain text. Instead there is a secret alias defined as an attribute of the identity element, namely “ elastic.scaler.ec2.identity”.
Firstly, you need to add following line into the “${CARBON_HOME}/repository/conf/security/cipher-tool.properties”.
Then you need to edit the “${CARBON_HOME}/repository/conf/security/cipher-text.properties” file.
There you need to add your plain text confidential information against the secret alias.
elastic.scaler.ec2.identity=[abcd]
Structure of the above line is:
<secretAlias>=[<plainTextValue>]
Note that you need to add the plain text value within square brackets.
Now navigate to the “${CARBON_HOME}/bin” directory and run following command;
./ciphertool.sh -Dconfigure
Type primary key store password of Carbon Server, when prompted. The default value is “wso2carbon”.
Ok, that is it. Now if you revisit “${CARBON_HOME}/repository/conf/security/cipher-text.properties” file, you could see all your plain text data are replaced by cipher text.
