Apache Stratos Incubator project in Google Summer of Code 2014

The Apache Software Foundation is participating in this year's (2014) Google Summer of Code program and we, at the Apache Stratos Incubator project is lucky enough to get two outstanding project proposals, accepted.

Following are the accepted projects:

1. Improvements to Auto-scaling in Apache Stratos - Asiri Liyana Arachchi 
Auto-scaling enables users to automatically launch or terminate instances based on user-defined policies, health status checks, and schedules. This GSoC project is on improving auto-scaling.


2. Google Compute Engine support for Stratos - Suriya priya Veluchamy 
Stratos uses jclouds to integrate with various IaaS providers. This project is to provide support for GCE(Google Compute Engine) which is google’s IaaS solution, and to run large scale tests in it. 


I'd like to warmly welcome Suriya and Asiri to the Apache Stratos community and looking forward to work with them throughout this period.

Also Congratulations to both of you!!

Happy Coding!

Apache Stratos as a single distribution

In Apache Stratos, we have recently worked on merging 3 of our main products (Stratos Manager, Cloud Controller and Auto-scaler) along with WSO2 CEP (Stratos uses WSO2 CEP as its complex event processing engine) into a single distribution using the powerfulness of WSO2 Carbon Framework. By doing so, we have expected to gain following main benefits:

1. Reduce the barrier for a new developer to getting started with Stratos.

2. Make lives of Stratos developers' easier by reducing the number of JVMs.

3. Deploying a distributed set-up using a single distribution in a production environment.

Earlier, in order to run Stratos, one needed to configure 3 Stratos products mentioned above, in addition to a message broker (Stratos uses message broker as its main inter-component communication channel), WSO2 CEP and WSO2 BAM (if you need monitoring capability). This would be resulted in six JVMs (assuming MB is also a JVM process) and would consume considerable amount of memory of your machine. Hence, lot of new blood, would rather give-up even without thinking of starting Stratos. 

With the introduction of this single distribution, as a developer, you can get-started with Stratos only with two JVMs namely Stratos JVM and MB (assuming it's a JVM), and in-turn would help us to attract more people to the project.

Reducing number of JVMs makes it easier to check logs, debug and makes your life way easier, as a contributor to Stratos. 

Further, you can use this single Stratos distribution and start Stratos Manager, Cloud Controller and Auto-scaler in 3 separate JVMs which will be useful in a real production deployment. In this case, of course you need to deploy WSO2 CEP and WSO2 BAM separately in addition to a Message Broker. 

Other than these, a single JVM Stratos deployment also capable of writing the data published by Stratos into a file (repository/logs/aggregate.log), so that you do not need an external business activity monitor in order to have a developer environment.

Try it out by following our wiki documentation here.

Building your own PaaS using Apache Stratos (Incubator) PaaS Framework - 2

This is a continuation of this post, where I have explained the basic steps you need to follow in order to build your own PaaS using Apache Stratos PaaS Framework. There, I have explained the very first step that you would need to perform on the PaaS Framework using our REST API. Here in this post, I am going to explain how you can perform the same step via Apache Stratos UI.

1. You need to access the Stratos Manager console via the URL that can be found once the set-up has done. eg: https://{SM-IP}:{SM-PORT}/console

Here you need to login as super-admin (user name: admin, password: admin) to the Stratos Manager.

2. Once you have logged in as super-admin, you will be redirected to the My Cartridges page of Stratos UI. This page shows the Cartridge subscriptions you have made. Since we have not done any subscriptions yet, we would see a page like below.

3. Navigate to the 'Configure Stratos' tab.

This page is the main entry point to configure the Apache Stratos PaaS Framework. We have implemented a Configuration Wizard which will walk you through a set of well-defined steps and ultimately help you to configure Stratos.

4. Click on the 'Take the configuration wizard' button and let it begin the wizard.

The first step of the wizard is the Partition Deployment and it is the intention of this blog post, if you can recall. We have provided a sample json file too, in the right hand corner, in order to let you started quickly.

5. You can copy the sample Partition json file, I have used in the post 1, and paste it in the 'Partition Configuration' text box. The text box has an inbuilt validation for json format, so that you cannot proceed by providing an invalid json.

6. Once you have correctly pasted your Partition json, you can click 'Next' to proceed to the next step of the configuration wizard.

Once you have clicked on 'Next', Stratos will validate your Partition configuration and then deploy it, if it is valid. Also you will see a message on top in yellow back-ground if it is successful and in case, your Partition is not valid, you will get to see the error message in a red back-ground.

That's it for now, if you like to explore more please check out our documentation. See you in the next post.

Adding support for a new IaaS provider in Apache Stratos

I have recently added a new wiki page to Apache Stratos documentation on providing a support for a new IaaS provider. You can check that here.

Prerequisite for this would be to have a good knowledge on the IaaS you are going to provide support for and also have some basic understanding of corresponding Apache Jclouds APIs.

Building your own PaaS using Apache Stratos (Incubator) PaaS Framework

This is a start of a series of blog posts I am planning to do on the topic "Building your own PaaS using Apache Stratos (Incubator) [1] PaaS Framework". 

PaaS, wondering what it is? It stands for Platform as a Service. It is the layer on top of the Infrastructure as a Service (IaaS) layer in the Cloud Computing Stack. Rackspace has published a white paper on the Cloud Computing Stack, and you may like to read it [2]. 

With the evolution of Cloud Computing technologies, people have realized the benefits that they could bring to their Organizations using Cloud technologies. Few years back they were happy to use an existing PaaS and develop/deliver their SaaS (Software as a Service) applications on top of it. But now, the industry has come to a state where they like to customize and build their own PaaS without being depended till the PaaS vendors deliver the customizations they need.

There arises a need of a framework where you have the freedom to customize and build the PaaS you wish. In this sense, having a pluggable, extensible and more importantly free and open source PaaS framework would be ideal.  Hard to believe an existence of such framework? No worries, Apache Stratos (Incubator) is there for you! 

Before go into details on the topic I am gonna discuss, it is worth to understand how Apache Stratos looks like. Apache Stratos consists of set of core components and the diagram below depicts them.

Currently Apache Stratos internally uses 3 main communication protocols, namely AMQP, HTTP and Thrift. 

AMQP protocol is mainly used to exchange topology information across core components. 'Topology' explains the run-time state of the PaaS at a given time such as existing services, service clusters, members etc.

HTTP protocol is used to perform SOAP service calls, among components.

Thrift protocol is used to publish various statistics to the Complex Event Processing Engine.

What does Apache Stratos (Incubator) core components capable of doing? Lakmal has explained this in [3].

In this first post of series of posts to come, I will roughly go through the major work-flows you need to do perform, in order to bring up your own PaaS, using Apache Stratos. Have a look at the below diagram;

As the sequence diagram explains, to build your own PaaS, in minimum, you need to follow the steps up to 'PaaS is ready!' state. Here, I am going to discuss the very first step you need to follow; that is 'Deploy Partitions'.

Let's understand the terminology first. What you deploy via a Partition is a reference to a place in an IaaS (eg: Amazon EC2/ Openstack etc.), which is capable of giving birth to a new instance (machine/node). Still not quite understood? Don't panic, let me explain via a sample configuration.

 {  
    "id": "AWSEC2AsiaPacificPartition1",  
    "provider": "ec2",  
    "property": [  
      {  
       "name": "region",  
       "value": "ap-southeast-1"  
      },  
      {  
       "name": "zone",  
       "value": "ap-southeast-1a"  
      }  
    ]   
 }  

The above JSON defines a partition. Partition has a globally unique (among partitions) identifier ('id') and an essential element 'provider' which points to the  corresponding IaaS provider type. This sample has two properties call 'region' and 'zone'. The properties you define here should be meaningful in the context of relevant provider. For an example, in Amazon EC2, there are regions and zones, hence you can define your preferred region and zone, for this partition. So, in a nut shell, what this partition references to is, ap-southeast-1a zone in ap-southeast-1 region of Amazon EC2. Similarly, if you take Openstack, they have regions and hosts.

Above sequence diagram explains the steps that get executed when you deploy a partition. You can either use Stratos Manager REST API or Apache Stratos CLI tool or Stratos Manager UI when deploying partitions. Partition deployment is successful only if the partitions are validated against their IaaS providers at Cloud Controller. Autoscaler is the place where these Partitions get persisted and it is responsible for selecting a Partition when it decides to start up a new instance. 

Following is a sample CURL command to deploy partitions via REST API;

 curl -X POST -H “Content-Type: application/json” -d @request -k -v -u admin:admin https://{SM_HOST}:{SM_PORT}/stratos/admin/policy/deployment/partition  

@request should point to the partition json file. More information on the partition deployment can be found at [4].

That concludes the first post, await the second!

References:

[1] http://stratos.incubator.apache.org/
[2] http://www.rackspace.com/knowledge_center/whitepaper/understanding-the-cloud-computing-stack-saas-paas-iaas
[3] http://lakmalsview.blogspot.com/2013/12/sneak-peek-into-apache-stratos.html
[4] https://cwiki.apache.org/confluence/display/STRATOS/4.0.0+Deploying+a+Partition

Why I need a port.mapping property when clustering WSO2 API Manager/ WSO2 ESB?

In a dynamically clustered set-up where you front a carbon instance using a WSO2 ELB, it is a responsibility of a Carbon server to send its information to ELB. You can visualize this as a "Member object somehow getting passed to ELB from Carbon server instance". In Carbon server's clustering section, under properties, you can define any Member property. This way, you can let ELB know about the information other than the basic ones. What are the basic information, you might think. Typically those are host name, http port, https port etc.

WSO2 ESB, WSO2 API Manager etc. are bit special w.r.t. ports since they usually have 2 http ports (compare to 1 http port of WSO2 AS). Hence, here we have to somehow send this additional information to ELB. Easiest way to do that is by setting a member property. Here, we use port.mapping property. Also, in order to front these special servers we need 2 http ports in ELB too, which are exposed to outside. There's a deployment decision to be made here, i.e. which http port of ELB should map to which http port of the server (i.e. servlet http port or Nhttp http port). With that in mind, let me explain, how you should use this port.mapping property.

Let me consider only the http scenario. Say, in your API-M instance, you have used 8280 as the Nhttp transport port (axis2.xml) and 9763 as the Servlet transport port (catalina-server.xml). Also ELB has 2 http ports one is 8280 and the other is 8290. Imagine there's a Member object, and in this case Member's http port would be 8280 (usually the port defined in axis2.xml gets here). But since ELB has 2 ports, there's no way to correctly map ports, by only specifying Member's http port. There arises the importance of port.mapping property. You have to think this property from the perspective of ELB.

<property name="port.mapping.8290" value="9763"/>

Let's assume we define the above property, now this means, if a request comes to ELB, in its 8290 port (see... we're thinking from ELB's perspective), forward that request to the 9763 port of the Member. Having only this property is enough, we do not need following property,

<property name="port.mapping.8280" value="8280"/>

Let me explain why. The logic was written in a way, that port.mapping properties get higher precedence over the default ports. This means, that when a request comes to ELB, ELB will first check whether the port it received the request from, is specified as a port.mapping property. If it is, it will grab the target port from that property. If not, it will send the request to the default http port. Hence, if a request received to the 8280 port of ELB, it will be automatically get redirected to 8280 port of the Member (since it's the http port of Member).

Similarly, we should define a mapping for https servlet port.

Hope someone find this useful.

Checking the non-existent of a property using WSO2 ESB 4.6

You can't simply do a string regex matching, if you are to check the non-existence of a property. What you could do is to leverage the boolean Xpath function [1], within a filter mediator.

Please see the following sample proxy configuration:

<proxy xmlns="http://ws.apache.org/ns/synapse" name="TEST" transports="https,http" statistics="disable" trace="disable" startOnLoad="true">
   <target>
      <inSequence>
         <filter source="boolean(get-property('accept'))" regex="false">
            <then>
               <log level="custom">
                  <property name="*********" value="NULL Property Value"/>
               </log>
            </then>
            <else>
               <log level="custom">
                  <property name="*********" value="NOT NULL Property Value"/>
               </log>
            </else>
         </filter>
         <send/>
      </inSequence>
      <outSequence>
         <log/>
         <send/>
      </outSequence>
   </target>
   <description></description>
</proxy>
                             

Hope this will save some amount of time of quite a bit of you :-)

[1] http://www.w3.org/TR/xpath/#function-boolean