vault-safe-cloud-security

How to Secure (ReSTFul) Web Services

Introduction

Application security is now one of hottest topic in IT departments. We are now fully in the age of the Service Oriented Architecture (SOA) regardless of how it has been rebranded by the marketing departments. It is the norm to build services and provided them to our clients, partners and suppliers. Retailers such as Amazon have open their door through their API to third party developers to make use of their vast resources; be it cloud computing or selling and buying items. We build services and make them available through (a well designed) application programming interface (API). There is a downside in making our API available to third parties, and that is we are making ourselves vulnerable to attackers to compromise our systems. This post aims to discuss ways to secure web services against such malicious behaviour. Remember, nothing is secured in theory but let’s make it darn hard to break it in practice. The security has to be a forethought when starting to develop a new service.

There are three layers of security to be addressed while developing APIs and making them available over the internet to third party users;

Web Services Security 3 Layer

These layers are also applicable to SOAP based web services and any network distributed systems. The above diagram shall be read bottom up. Even though the above layers look and sound similar to the OSI Model, there are not the same and should not be confused.

Transport layer

When making a service available over a network, we are utilizing the transport layer. The role of the transport layer is to ensure that third party clients can make a reliable connection to our services. Therefore, the transport layer creates a communication link between our API and consumers. In the context of web services, our communications link is created over HTTP. We all know that HTTP in its barebones is not secured but we can still observe people making their APIs available over plain HTTP. It’s possible that your company has developed its own proprietary encryption which can be added to the HTTP communication but in any cases, here are the most common ones:

Legacy network applications which communicates over the internet using HTTP were mostly secured using Secure Socket Layers (SSL). Certificates vendors branded their products as SSL certificates, therefore when the Transport Layer Security (TLS) was introduced, they kept the same name as SSL certificates. TLS is the evolution of SSL, needless to say that all APIs communications shall be done over HTTPS preferably using TLS as it has fixed some of the vulnerabilities of SSL.

Presentation Layer

Clients initiates communication with web services through the transport layer by using the web services URI. Once the connection is established; the client is then forwarded to the presentation layer where requests can be made. Depending on the system architecture and the purposes of the web services, the client can either make anonymous or authenticated calls. Protected web services calls shall be available only to authenticated parties. The most common way to block anonymous callers from making restricted calls is to force them to confirm their identity. Here are three common ways to validate third party callers:

  • Basic Authentication: the web services would request that the third party caller identifies itself by providing a valid username and password combination which would be used to create a session for the duration of the communication
  • OAuth 1 or 2: OAuth allows third party client to access users resources without sharing their credentials. It is commonly used by web services such as Facebook, Twitter and LinkedIn to authorise third party applications to log onto their sites.
  • Identity Certificate: identity certificates are, in many forms, similar to SSL certificates. The Certificate Authority (CA), which could be your company, signs and endorse the certificate on behalf of a third party. You can provide your third party with an identity certificate that you have signed with your key. This could be very secured in the same way as HTTPS. Third party clients should not self signed their certificates. As the certificates are available on the callers’ devices, if a device is stolen, this can become a security risk.

A recommendation would be to secure your web services API with Basic Authentication over HTTPS, this security approach is the most popular and tested on the internet. OAuth v1 would be recommended over v2 for transmitting highly sensitive data. OAuth suitability on commercial APIs is questionable as opposed to web site which aims is amassed a large user base.

Application Layer

This layer has been ignored in most conversation about web services security. A simple Google search on web services security would show results which only addressed the transport and presentation layer. So why would you want to discuss ‘securing the web services at the application layer’? This is as important as the two previous layers. Let’s put it into context; a third party client ‘s systems have been compromised. The attackers were able to obtain some credentials to our web services. As we do not have any application layer security in place, the attacker can connect to our services and make all sort of requests. This is a fictitious yet very probable scenario. So let’s tackle how we handle user authorisation in our web services, remember that authorisation is the process of verifying that you have access to something. Here are two possible solutions:

  1. Digital Asset Manager (DAM)
  2. Custom development

The application layer security would group, roles, domain or hierarchy level security. It is faster and cheaper to build a custom solution or procure an Open Source alternative. Regardless of the which route you may venture into, application layer security has to be implemented to have a fully secured web services.

Conclusion

We have discussed the three layers of web services security. Remember that; in theory nothing is secure but we should make near impossible to break it in practice. Architects have to consider all three layers when designing APIs which would be available to third parties over the internet. All web services communication shall be conducted over a secured channel such as HTTPS. The Basic Authentication can handle most authentication requests to web services and is a secure way to exchange user credentials over HTTPS. Application Layer security shall be implemented to handle authorisation to resources. Remember, code defensively to mitigate risks of a security breach.

screwupcolor1

Web Services Architecture – When to Use SOAP vs REST

SOAP (Simple Object Access Protocol) and REST (Representation State Transfer) are popular with developers working on system integration based projects. Software architects will design the application from various perspectives and also decides, based on various reasons, which approach to take to expose new API to third party applications. As a software architect, it is good practice to involve your development team lead during system architecture process.
This article, based on my experience, will discuss when to use SOAP or REST web services to expose your API to third party clients. 

Web Services Demystified

Web services are part of the Services Oriented Architecture. Web services are used as the model for process decomposition and assembly. I have been involved in discussion where there were some misconception between web services and web API.
The W3C defines a Web Service generally as:

 

A software system designed to support interoperable machine-to-machine interaction over a network.

 

Web API also known as Server-Side Web API is a programmatic interface to a defined request-response message system, typically expressed in JSON or XML, which is exposed via the web – most commonly by means of an HTTP-based web server. (extracted from Wikipedia)

Based on the above definition, one can insinuate when SOAP should be used instead of REST and vice-versa but it is not as simple as it looks. We can agree that Web Services are not the same as Web API. Accessing an image over the web is not calling a web service but retrieving a web resources using is Universal Resource Identifier. HTML has a well-defined standard approach to serving resources to clients and does not require the use of web service in order to fulfill their request.

 

Why Use REST over SOAP

Developers are passionate people. Let’s briefly analyze some of the reasons they mentioned when considering REST over SOAP:

 

REST is easier than SOAP

I’m not sure what developers refer to when they argue that REST is easier than SOAP. Based on my experience, depending on the requirement, developing REST services can quickly become very complex just as any other SOA projects. What is your service abstracting from the client? What is the level of security required? Is your service a long running asynchronous process? And many other requirements will increase the level of complexity. Testability: apparently it easier to test RESTFul web services than their SOAP counter parts. This is only partially true; for simple REST services, developers only have to point their browser to the service endpoints and a result would be returned in the response. But what happens once you need to add the HTTP headers and passing of tokens, parameters validation… This is still testable but chances are you will require a plugin for your browser in order to test those features. If a plugin is required then the ease of testing is exactly the same as using SOAPUI for testing SOAP based services.

 

RESTFul Web Services serves JSON that is faster to parse than XML

This so called “benefit” is related to consuming web services in a browser. RESTFul web services can also serve XML and any MIME type that you desire. This article is not focused on discussing JSON vs XML; and I wouldn’t write any separate article on the topic. JSON relates to JavaScript and as JS is very closed to the web, as in providing interaction on the web with HTML and CSS, most developers automatically assumes that it also linked to interacting with RESTFul web services. If you didn’t know before, I’m sure that you can guess that RESTFul web services are language agnostic.
Regarding the speed in processing the XML markup as opposed to JSON, a performance test conducted by David Lead, Lead Engineer at MarkLogic Inc, find out to be a myth.

 

REST is built for the Web

Well this is true according to Roy Fielding dissertation; after all he is credited with the creation of REST style architecture. REST, unlike SOAP, uses the underlying technology for transport and communication between clients and servers. The architecture style is optimized for the modern web architecture. The web has outgrown is initial requirements and this can be seen through HTML5 and web sockets standardization. The web has become a platform on its own right, maybe WebOS. Some applications will require server-side state saving such as financial applications to e-commerce.

 

Caching

When using REST over HTTP, it will utilize the features available in HTTP such as caching, security in terms of TLS and authentication. Architects know that dynamic resources should not be cached. Let’s discuss this with an example; we have a RESTFul web service to serve us some stock quotes when provided with a stock ticker. Stock quotes changes per milliseconds, if we make a request for BARC (Barclays Bank), there is a chance that the quote that we have receive a minute ago would be different in two minutes. This shows that we cannot always use the caching features implemented in the protocol. HTTP Caching be useful in client requests of static content but if the caching feature of HTTP is not enough for your requirements, then you should also evaluate SOAP as you will be building your own cache either way not relying on the protocol.

 

HTTP Verb Binding

HTTP verb binding is supposedly a feature worth discussing when comparing REST vs SOAP. Much of public facing API referred to as RESTFul are more REST-like and do not implement all HTTP verb in the manner they are supposed to. For example; when creating new resources, most developers use POST instead of PUT. Even deleting resources are sent through POST request instead of DELETE.
SOAP also defines a binding to the HTTP protocol. When binding to HTTP, all SOAP requests are sent through POST request.

 

Security

Security is never mentioned when discussing the benefits of REST over SOAP. Two simples security is provided on the HTTP protocol layer such as basic authentication and communication encryption through TLS. SOAP security is well standardized through WS-SECURITY. HTTP is not secured, as seen in the news all the time, therefore web services relying on the protocol needs to implement their own rigorous security. Security goes beyond simple authentication and confidentiality, and also includes authorization and integrity. When it comes to ease of implementation, I believe that SOAP is that at the forefront.

 

Conclusion

This was meant to be a short blog post but it seems we got to passionate about the subject.
I accept that there are many other factors to consider when choosing SOAP vs REST but I will over simplify it here. For machine-to-machine communications such as business processing with BPEL, transaction security and integrity, I suggest using SOAP. SOAP binding to HTTP is possible and XML parsing is not noticeably slower than JSON on the browser. For building public facing API, REST is not the undisputed champion. Consider the actual application requirements and evaluate the benefits. People would say that REST protocol agnostic and work on anything that has URI is beside the point. According to its creator, REST was conceived for the evolution of the web. Most so-called RESTFul web services available on the internet are more truly REST-like as they do not follow the principle of the architectural style. One good thing about working with REST is that application do not need a service contract a la SOAP (WSDL). WADL was never standardized and I do not believe that developers would implement it. I remember looking for Twitter WADL to integrate it.
I will leave you to make your own conclusion. There is so much I can write in a blog post. Feel free to leave any comments to keep the discussion going.

angularjs1

Liferay and AngularJS Made Simple: Connecting AngularJS to a Backend with REST and JSON

Introduction

Liferay is the leading Open Source Enterprise Portal. One may asked what an enterprise portal is and this question is very valid as it has been asked on every single Liferay project that I have worked. This blog post is not about defining what an enterprise portal is but it wouldn’t be a crime if we provided a brief definition:

An enterprise portal is a web application which provides services required by an enterprise such as: user management, authentication and authorisation services, ability to connect to third party applications and provide a single point of access to multiple applications, hence the “portal”.

The above is my own definition and it could be extended to encompass web content management, content management system (CMS) and single sign on (SSO). This post is about Liferay and the use of its web content management system (WCMS) to create single page applications using AngularJS. The motivation to create portlets using AngularJS instead of Java is as:

  • Portlet development using Java is very expensive
  • Not many Java developers with portlet experience
  • Java portlets development requires heavy duty tools such as build tools, IDE and JVM
  • Portlet developers need to be familiar with the Portlet API, lifecycle and framework

We will focus on Liferay available RESTful web services API but do remember that you can create your own custom web services using Liferay service builder SDK.

Liferay RESTFul API and Security

Liferay ReSTFul and SOAP API implement the same security as the core library:

  • API can be secured so that only authenticated users can access them (AUTHENTICATION)
  • API can be secured so that only users with the right roles can executed certain API calls (AUTHORIZATION)

When creating your own custom API, Liferay Services Builder will create the necessary permission for the web services API.
For a list of API available in Liferay, point your browser to the following

http://<your-server-address>:<your-server-port>/api/jsonws

Liferay will provide a means of testing the services calls when the above URL is loaded. Most services execution will require authentication or a secured token to be passed on with the calls. This level of a security is required in an enterprise environment. It is possible to stop Liferay from checking for the secured token in portal-ext.properties as

Auth.token.check.enabled=false

Software developments should promote code reuse, therefore by separating the business logic from the portlet code, developers can share the business logic with third party applications.

Why Use AngularJS to Create Web Applications (Not Portlets)?

This is not a tutorial on AngularJS. Developers should use the same approach for developing any AngularJS application to developing Liferay web applications.
AngularJS is a popular JavaScript framework promoting Object Oriented Development (OOD) and Model View Controller (MVC) to the JavaScript community. Java developers are already custom with the methodology through the use of Spring MVC and JSF for front end developments. Developers familiar with Google Web toolkit (GWT) should find themselves in familiar territory. Now to answer the question of why use AngularJS to create web applications on Liferay?
AngularJS is JavaScript and therefore can be executed in the browser without recompilation and redeployment. Liferay Web Content Management System (WCMS) provides an HTML editor and content versioning. Liferay JSONWS API runs on the same server and can be accessed through the JavaScript written in the WCMS. AngularJS modules can be written in a third party editor such Notepad++ and uploaded to Liferay Content Management System (CMS). The Liferay CMS provides a link to the latest version of the file which can be referenced in the HTML/ JavaScript code. By creating the web services in Java through Liferay Services Builder, the java developer can focus on the business logic – including testing. The front end developer can utilise his skills in HTML and JavaScript to develop the user interfaces and any necessary interactions with the backend through the ReSTFul services. There is a clear separation of work and accountability. The learning curve for the Java developers to create the services will be minimal. To preview the live code, the frontend developer only has to save the content (WCMS) and refresh the page to see the latest changes.
Here is a quick example:

 

 <div ng-app="" ng-controller="companiesController">  
   <ul>  
    <li ng-repeat="x in data">{{'title: ' +x.title + ', group Id: ' + x.groupId }}</li>  
   </ul>  
 </div>  
 <script>  
   function companiesController($scope,$http) {  
    $http.post("http://localhost:8080/api/jsonws/assetentry/get-company-entries/company-id/10157/start/0/end/5?p_auth=cbSXanJ2")  
    .success(function(response) {$scope.data = response;});  
   }  
   companiesController.$inject = ['$scope', '$http'];  
 </script><script src="http://ajax.googleapis.com/ajax/libs/angularjs/1.2.15/angular.min.js"></script>  

You can copy the above in a new web content article and see the result displayed on the page. Make sure to change red bold values to your system specific:

  • Company id: for ease of testing, you can retrieve that value from the control panel
  • P_auth: this value can be looked up programmatically as it will change every time the user logs into the portal

Traditionally, the simple code above would require a JavaServer page or JSF application which is slow for rapid prototyping. In software testing, portlets controller are one of the most complex components to test. By creating a clear separation between components, testers and automated tools can test each components individually. The sample code pulls information from Liferay REST web services and displays a list of registered companies on the page.

Conclusion

Liferay has a rich set of features which allows developers to create enterprise components and applications. When working with ReSTFul services, Liferay Web Content Management editor can act as an Integrated Development Environment in the browser. AngularJS is approaching maturity and it is very popular with web developers. Liferay 7 (next release as of writing ) will be introducing single page portlets but this is already possible with AngularJS and ReSTFul web services API. Needless to say that you can use any web browser to create content in Liferay CMS and debug your code in real time using tools such as Firebug.

4099041220_b0ccec7f10

Develop High Transaction Web Applications with Java MySQL & NetBeans

This entry is a brief tutorial on how to develop high transactional web application. I will be looking at how to develop a high transactional application while maintaining data integrityFor the purpose of keeping this entry simple, I will be using some RAD tools, the NetBeans IDE, to generate most of the code.

In order to follow the tutorial, you will need the following:

  1. NetBeans IDE 6.7+
  2. Java JDK 1.6+ (my version is 1.6.0_17)
  3. MySQL 5.1+
  4. MySQL Sakila database
  5. Apache Tomcat 6.x
  6. An understanding of JPA transactions and ReSTful web services

This is my definition of High Transaction Applications:

A high transaction application is one that can serve multiple simultaneous requests from clients and keep them secure from each other. The application has only two purposes: read or write data to/from a repository either JMS or DB. The transactions have to meet the ACID criteria in order to be deployed in the real world.

Ok, the above is my definition and you are free to redefine it. I am going to build a web application that will be an n-tier application:

  • Database back-end (MySQL)
  • Entity classes
  • ReSTful Web Services to allow other developers to integrate the application with theirs
  • A web based front-end

For the simplicity of the article, I will not implement any security such as user or application level security (authentication and database table privileges). This tutorial is mostly geared toward the newbies but I am sure that more advanced developers would benefit too.

Let’s get coding.

  1. Make sure that you have loaded the MySQL Sakila database into your MySQL database. You can download the Sakila database from the NetBeans plugin centre (see here).
  2. Create a new web application and name whatever you like. I have named mine “WebApplication”. I will now refer the application as WebApplication. Make sure to choose Tomcat as your deployment server
  3. Add the MySQL driver “mysql-connector-java-5.0.7-bin” to the WebApplication libraries. NetBeans will work and connect to the DB even without the driver but once you deploy your application to a server, the application will not be able to connect to the DB and throw a ClassNotFoundException com.mysql.jdbc.driver.
  4. We are going to develop the back-end first. NetBeans makes it very easy for us to create Entity classes from database. I would recommend newbies to learn how to manually create Entity classes and configure the persistence.xml file. This tutorial makes use of JPA but one should ignore the drawbacks of JPA/ ORM frameworks.
  5. Right click on the project name and choose Entity Classes from Database… On the next screen choose the “filmActor” table and click on the Add button. Make sure that “Include Related Tables” box is checked underneath the Selected Tables panel. The screenshot does not show the “filmActor” table as I have previously generated the Entity class but I am sure you get the idea.

  6. If you are required to create a Persistence Unit, click on the persistence button -> you can accept the default name -> choose your Persistence Library -> choose your database connection -> choose “None” for Table Generation Strategy -> click Finish
  7. After choosing the table to generate the Entity from click next -> fill Package name -> tick Generate Name Query… -> click Finish
  8.  

    NetBeans generates all the Entity classes based on the database table that you have chosen. The next thing that we want to do is generate a set of ReSTful web services from the generated Entity classes. Again, NetBeans facilitates the work for us (it is important that you also know how to create the classes manually or you will not know how to debug them if there is any problem in the future).

  9. Right click on the project name “WebApplication” -> RESTful Web Services from Entity Classes… -> Choose the Entity Classes that you would like to generate the WS for and click add or add all -> click next -> on the following screen, accept the default values and click Finish

  10. You can go ahead and test your ReSTFul Web Services by right clicking on the name of the application and click on Test RESTful Web Services

  11. The previous step will launch your web browser within which you can test your web services (click on the node on the left and see the queries on the right)

  12. Back in the NetBeans IDE; right click on the project name “WebApplication” -> click on JSF Pages from Entity Classes… (The JSF pages will not use the web services as there are packaged together with the Entity Classes. This will improve performances and still allow external applications to integrate). Choose the Entity Classes that you would like to generate the pages for -> click next. On the final screen, fill in the package name for the JPA Controller and JSF Classes -> click finish

    NetBeans will generate the necessary files to create a CRUD application with a user interface. I suggest that you familiarize yourself with the generated code.

  13. Expand the Configuration folder under your project name “WebApplication” and the web.xml file. At around line 38, change the content of the welcome-file-list to look as follow
    <welcome-file-list>
            <welcome-file>faces/welcomeJSF.jsp</welcome-file>
    </welcome-file-list>

    This will make the generate JSF page to be the landing page for the application when requested. Make sure that you do have a “welcomeJSF.jsp” file before making the change.

  14. Right click on the application name -> Run. The application should load in your web browser. Now go on, play around with the application. And why not create a client to send request to the web services? (not today)

You can load test your application by using Apache JMeter. It is easy to run and configure. If you want to take a look at how JPA implements the ACID features, browse to your controller classes. Here is a short introduction to JPA transaction.

Hope you enjoyed and if you need any clarification, just leave me a comment and try to get back to you ASAP (if time permits).