Cloud computing vs Open Source – Most CIO cannot tell the difference

Few weeks ago, I posted a discussion here in a CIO forum on LinkedIn and frankly some of the comments are very interesting to say the least. Now I decided to bring some technical guys to the discussion table. My argument was and still is as follow
“I have been reading many threads about Cloud computing and I came to conclude that most CIO (maybe because of their business background) do not know the difference or real benefits of Cloud computing or Open Source software.

Hi, I am not expecting everyone to agree with me but here is my argument. Most CIOs are interested in Cloud computing because of cost. COST!!? That’s very interesting because to my humble opinion, this only applies to IaaS (Infrastructure as a Service). If you do not want to spend money on hardware and be left with redundant system then IaaS is a good approach but that’s about it. I work with Cloud computing (specifically SaaS) and this next one might hurt my business but it has to be said; SaaS is not cost saving when you comparing it Open Source alternatives that you can host on your IaaS system…. Read more here

I am not sure that they understood the argument. I was not expecting anybody too agree with me and I wanted CIOs in the group to share their insight not sound like a copy of a Gartner report. Before making any conclusion, keep in mind that I am talking here about Software as a Service (SaaS), as defined here. In network architecture design, the external network  is represented by the “cloud” symbol therefore I would have to agree Wikipedia’s definition of cloud.  It is a software served over an external network regardless of a private network (private cloud) or public network (public cloud). YES – the internet was built on cloud technology. If you use any software over the internet; be it email, search, video or social network, the system is hosted by an external party which you do not have control over. The name cloud is just a marketing gimmick because hosted applications have been around for more than 20 years since Compuserve and Internet Relay Chat (IRC & BBS). So you see, “cloud” is definitely not new. Therefore software delivered over the internet such as Google apps and Salesforce can be also deployed using Open Source alternative. Cloud promises the following: scalability, peace of mind as it maintained by the providers and lower cost. CIO, I will tell you that your software, that it is deployed in the cloud or not will not be able to scale beyond the hardware and network capacity. I have another question for CIOs:
Do CIOs care or know where all their applications are running? 
That question is usually left to the technical team to decide within the budgets of the project and any other company politics. The biggest failures of IT projects is due to the fact that the CIO (or “the business” as it is usually referred as) do not know what they are trying to achieve (lack or misunderstandings of requirements) therefore it goes past the delivery and budget. 
So where does Open Source fits in, one might ask? Open Source software allows a business to take best of breed software and customised them to their own needs for free (as in free beer). If you do not want to support the Open Source software you can get support services but the software is hosted on your own system including your private cloud (Infrastructure as a Service). Let’s put it in to context; if a company requires a CRM system – they can download vTiger, customise it and either support it themselves or buy support. The same goes for an ERP system like Apache Ofbiz and MySQL for database. I am in favour of SaaS offering, after all I run a SaaS-based company. Nevertheless, CIOs do not confuse SaaS as being cheap, if you want something that has a low cost of ownership, then I suggest you go for an Open Source alternative if it exists. So why do I say that most CIO cannot tell the difference; SaaS is not cheap and does not have a lower cost of ownership when comparing it to Open Source alternatives. And by the way, you do not need to have an army of developers to build and customise your Open Source software. Do you really want the SaaS provider on a shared system to have access to your LDAP system? What happens when you want to integrate or port your existing data to your new fancy cloud based system? CIOs should not try to lie to me. I attended many meetings and conferences to know that most of them just want to please the board. What are the differences between cloud and virtualisation? What happens to your existing hardware once you moved your applications to cloud, have you thought of using them for virtualisation? So please do not make uneducated comments as “I don’t know a single CIO who can’t tell the difference between Cloud Computing & Open Source – maybe I move in different circles.” I am not talking about dictionary or Gartner definitions.  
CIOs, after reading this and before replying you should go and sit down with your technical team and ask them the difference, only after that you can provide a good educated comment. If you want to be on the “cloud” train then you are free to do so but if SaaS is not part of cloud offerings, please feel free to enlighten me as I am always willing to learn something new.
Who should make the decisions to go cloud or not? You are the CIOs, you tell me. Is it the developer, the architect, the project manager or the almighty CIOs from his ivory tower? There is no trickery here, every fool can define a word – just do this; think of a word then Google using the following “what is ‘the word’” then voila your definition. If you need someone to school you up then drop me an email. Based on the comments on LinkedIn, all I can say is WOW. Leave the technical aspect to the technical guys and you can focus on playing golf.
20090420-oracle-buys-sun1

Will Oracle really make NetBeans the BEST IDE for Java

Not yet!!! But according to their webcast they want to make NetBeans the best Java IDE. This is a statement not to be taken lightly. Oracle are now investing in three IDEs which are all established in different ways. JDeveloper is Oracle’s IDE of choice, I am currently using it in my current project as I am developing for Oracle WebCenter. Oracle is also a strategic developer and board member of the Eclipse foundation. I do not understand what exactly is a strategic developer but I know that JDeveloper is their strategic IDE. Now, introducing NetBeans, Oracle has inherited NetBeans as part of Sun acquisitions (and a ton of products and services). Until now, Oracle has never made any plug-ins for the NetBeans platform but now they promise to make it the best IDE for Java, how will they make that happen?

I am a NetBeans evangelist and as the rest of the community, it was good news to actually know where it stands in the “Oracle” vision. So Oracle will never drop JDeveloper therefore the real fight is between Eclipse and NetBeans (as it always has been). In the first step of making NetBeans the best IDE, Oracle shold immediately start porting all their Eclipse plug-ins to NetBeans. Eclipse has strength in its alliance which also includes IBM (fierce rival to Oracle). Something has to give and Eclipse will not never close shop because Oracle has left them which is not as worse as lack of investment in the NetBeans community. It would also be a foolish decision to give NetBeans to the Apache Foundation (no disrespect here but it seems that public funds do not grow on trees).

There is something funny about Oracle “choice” statement. Did anybody tried to develop for WebCenter using NetBeans or Eclipse? That’s a nightmare!!! There is no choice given, they literally forced us to use JDeveloper (Do you require any Oracle support?) in our company which is the largest in its field in Europe (I am avoiding to mention the company name but you should get it from my twitter stream).

Eclipse and NetBeans are direct competitors with Eclipse being the de facto IDE for Java, dropping Eclipse support would close doors to millions of developers and dropping NetBeans will disaster for many paying customers (Sun’s partners).  I am not sure what is going to happen in the NetBeans/ Eclipse and JDeveloper saga but something has to give.

Let me know what you think.

happy20101

Happy New Year Techies!!!

I just wanted to say happy new year to the community, without you guys, we would still be living in the dark ages. Here are a few that I am looking forward to in the new year:

  1. Oracle Sun merger:- The untold future of NetBeans, MySQL, Swing and every open source that Sun has been working on in the past. What would happen to Sun Open Source (SOS!!!) Movement.
  2. Looking forward to JavaFX1.3 release, Authoring tool and improved Composer plugin.
  3. Android uprise against iPhone (not because I can easily write Android based application) and hopefully with Visual XML builder to build Android UI. JavaFX running on Android, anybody????
  4.  Looking forward to seeing real-world JavaFX application and possibly a showcase linked to the JavaFX.com site.
  5. Java EE 6 support in Google App Engine
  6. Google Wave open to the public and how well it does against Twitter and Facebook
  7. Java Store open to Europe (well this is where I am based and want to sell applications not just provide free).
  8. My new video blog ( well that’s me trying to be more Armel 2.0 – the sequel) to better engage with the community.
  9. looking forward to networking with fellow developers and techies.
  10. looking forward to the new buzzwords (old technology, new name)

Well, 2009 is was good year for Java and tech scene. I hope that you all enjoyed the year as much as I did. Why not share what wish list with me (comment box). Subscribe to follow my blog as I will try to bring more interesting articles in the new year.

Happy 2010 New Year, wish you all; success, properity, fame (yup) and fortune. Don’t forget, if you need a server-side, UI, Android or JavaFx developer, just mail me.

4124677046_69aa7a07b3

GWT Visualization Example – Annotated Time Line Chart Tutorial

If you are a server-side Java developer such as myself, I am sure you want an easy way to create AJAX based application but pure JavaScript (ECMAScript) is not your strong skills. The beauty of JavaScript is that it runs on any web server, no need for a servlet container or anything of that sort.

So, I have been writing AJAX based UI using the GWT framework for the past year and half. I am still not a web developer but I understand enough CSS to spice up my site. Anyway, I am writing this tutorial because I think that the GWT Visualization team, even tough they did a good job, over complicates the creation of charts in their tutorial. I was trying to create an Annotated Time Line chart so I looked up my previous code I wrote for the GWT Portlet JSR-168, see here. I also decided to run a search (don’t be evil, Google is your friend) to try to find different examples and hopefully some nice custom charts. First of all, based on the search result; I thought that some of examples, including Google’s own, were confusing and over complex. Most examples uses AJAXLoader to load the Visualization API, but you shouldn’t have to use this. This is the simplest way to create a GWT Annotated Time Line Chart.

gwt annotated time line chart

1. Using your favorite development tool with support for GWT framework, create a new Java web application, you can export the compiled JavaScript later to a non-based web application, if you use an IDEW you should code-completion support.

2. Make you sure you have included the GWT Visualization API module in your classpath.

3. In your project source code root directory, you should <AppNamexxxx>.gwt.xml file. make sure to add teh following line to it in order to make the  module available to your application.

<inherits name=”com.google.gwt.visualization.Visualization”/>

4. In order to keep this tutorial short and straight to point, I have included the chart creation code in my <AppNamexxxx>EntryPoint.java class

public class MainEntryPoint implements EntryPoint
{
    
    
    /**
    * Creates a new instance of MainEntryPoint
    */
    public MainEntryPoint()
    {
        
        
    }
    
    
    
    /**
    * The entry point method, called automatically by loading a module
    * that declares an implementing class as an entry-point
    */
    public void onModuleLoad()
    {
        
        Runnable onLoadCallback = new Runnable()
        {
            
            
            public void run()
            {
                
                DataTable data = createHistoryTable(result);
                AnnotatedTimeLine.Options options = AnnotatedTimeLine.Options.create();
                options.setDisplayAnnotations(true);
                options.setDisplayZoomButtons(true);
                options.setScaleType(AnnotatedTimeLine.ScaleType.ALLFIXED);
                options.setLegendPosition(AnnotatedTimeLine.AnnotatedLegendPosition.SAME_ROW);
                AnnotatedTimeLine atl = new AnnotatedTimeLine(data, options, “600px”, “200px”);
                RootPanel.get().add(atl);
                
            }
            
            
            
        }      
        ;
        VisualizationUtils.loadVisualizationApi(onLoadCallback, AnnotatedTimeLine.PACKAGE);
        
    }
    
    
    
    // This method will create the Data used by the chart
    private DataTable createHistoryTable()
    {
        
        DataTable data = DataTable.create();
        data.addColumn(AbstractDataTable.ColumnType.DATE, “Date”);
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, “Price”);
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, “Low”);
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, “High”);
        data.addRows(5);
                  
            DateTimeFormat dtf = DateTimeFormat.getFormat(“yyyy-MM-dd”);
            data.setValue(0, 0, dtf.parse(“2009-11-21”));
            data.setValue(0, 1, 100);
            data.setValue(0, 2, 120);
            data.setValue(0, 3, 90);

            data.setValue(1, 0, dtf.parse(“2009-11-22”));
            data.setValue(1, 1, 90);
            data.setValue(1, 2, 110);
            data.setValue(1, 3, 100);

            data.setValue(2, 0, dtf.parse(“2009-11-23”));
            data.setValue(2, 1, 100);
            data.setValue(2, 2, 180);
            data.setValue(2, 3, 80);

            data.setValue(3, 0, dtf.parse(“2009-11-20”));
            data.setValue(3, 1, 130);
            data.setValue(3, 2, 100);
            data.setValue(3, 3, 130);
          

            data.setValue(4, 0, dtf.parse(“2009-11-19”));
            data.setValue(4, 1, 130);
            data.setValue(4, 2, 170);
            data.setValue(4, 3, 150);
            
        }
        return data;  
    }  
}

Don’t forget to import all the necessary classes and voila!

I hope this will save you some time in creating your charts.

 

4099041220_b0ccec7f10

Develop High Transaction Web Applications with Java MySQL & NetBeans

This entry is a brief tutorial on how to develop high transactional web application. I will be looking at how to develop a high transactional application while maintaining data integrityFor the purpose of keeping this entry simple, I will be using some RAD tools, the NetBeans IDE, to generate most of the code.

In order to follow the tutorial, you will need the following:

  1. NetBeans IDE 6.7+
  2. Java JDK 1.6+ (my version is 1.6.0_17)
  3. MySQL 5.1+
  4. MySQL Sakila database
  5. Apache Tomcat 6.x
  6. An understanding of JPA transactions and ReSTful web services

This is my definition of High Transaction Applications:

A high transaction application is one that can serve multiple simultaneous requests from clients and keep them secure from each other. The application has only two purposes: read or write data to/from a repository either JMS or DB. The transactions have to meet the ACID criteria in order to be deployed in the real world.

Ok, the above is my definition and you are free to redefine it. I am going to build a web application that will be an n-tier application:

  • Database back-end (MySQL)
  • Entity classes
  • ReSTful Web Services to allow other developers to integrate the application with theirs
  • A web based front-end

For the simplicity of the article, I will not implement any security such as user or application level security (authentication and database table privileges). This tutorial is mostly geared toward the newbies but I am sure that more advanced developers would benefit too.

Let’s get coding.

  1. Make sure that you have loaded the MySQL Sakila database into your MySQL database. You can download the Sakila database from the NetBeans plugin centre (see here).
  2. Create a new web application and name whatever you like. I have named mine “WebApplication”. I will now refer the application as WebApplication. Make sure to choose Tomcat as your deployment server
  3. Add the MySQL driver “mysql-connector-java-5.0.7-bin” to the WebApplication libraries. NetBeans will work and connect to the DB even without the driver but once you deploy your application to a server, the application will not be able to connect to the DB and throw a ClassNotFoundException com.mysql.jdbc.driver.
  4. We are going to develop the back-end first. NetBeans makes it very easy for us to create Entity classes from database. I would recommend newbies to learn how to manually create Entity classes and configure the persistence.xml file. This tutorial makes use of JPA but one should ignore the drawbacks of JPA/ ORM frameworks.
  5. Right click on the project name and choose Entity Classes from Database… On the next screen choose the “filmActor” table and click on the Add button. Make sure that “Include Related Tables” box is checked underneath the Selected Tables panel. The screenshot does not show the “filmActor” table as I have previously generated the Entity class but I am sure you get the idea.

  6. If you are required to create a Persistence Unit, click on the persistence button -> you can accept the default name -> choose your Persistence Library -> choose your database connection -> choose “None” for Table Generation Strategy -> click Finish
  7. After choosing the table to generate the Entity from click next -> fill Package name -> tick Generate Name Query… -> click Finish
  8.  

    NetBeans generates all the Entity classes based on the database table that you have chosen. The next thing that we want to do is generate a set of ReSTful web services from the generated Entity classes. Again, NetBeans facilitates the work for us (it is important that you also know how to create the classes manually or you will not know how to debug them if there is any problem in the future).

  9. Right click on the project name “WebApplication” -> RESTful Web Services from Entity Classes… -> Choose the Entity Classes that you would like to generate the WS for and click add or add all -> click next -> on the following screen, accept the default values and click Finish

  10. You can go ahead and test your ReSTFul Web Services by right clicking on the name of the application and click on Test RESTful Web Services

  11. The previous step will launch your web browser within which you can test your web services (click on the node on the left and see the queries on the right)

  12. Back in the NetBeans IDE; right click on the project name “WebApplication” -> click on JSF Pages from Entity Classes… (The JSF pages will not use the web services as there are packaged together with the Entity Classes. This will improve performances and still allow external applications to integrate). Choose the Entity Classes that you would like to generate the pages for -> click next. On the final screen, fill in the package name for the JPA Controller and JSF Classes -> click finish

    NetBeans will generate the necessary files to create a CRUD application with a user interface. I suggest that you familiarize yourself with the generated code.

  13. Expand the Configuration folder under your project name “WebApplication” and the web.xml file. At around line 38, change the content of the welcome-file-list to look as follow
    <welcome-file-list>
            <welcome-file>faces/welcomeJSF.jsp</welcome-file>
    </welcome-file-list>

    This will make the generate JSF page to be the landing page for the application when requested. Make sure that you do have a “welcomeJSF.jsp” file before making the change.

  14. Right click on the application name -> Run. The application should load in your web browser. Now go on, play around with the application. And why not create a client to send request to the web services? (not today)

You can load test your application by using Apache JMeter. It is easy to run and configure. If you want to take a look at how JPA implements the ACID features, browse to your controller classes. Here is a short introduction to JPA transaction.

Hope you enjoyed and if you need any clarification, just leave me a comment and try to get back to you ASAP (if time permits).

Develop Your Own Google with Apache Lucene (Java Nutch Solr)

Apache Lucene is Open Source API that allows a Java developer (.Net libraries available) to write indexing and full-text search capable applications. I have been writing applications based on Lucene for the last 3 years and some of the applications have been deployed at large corporations. I know there are other libraries available to developers who wish to write indexing engine but this blog will solely focus on Apache Lucene. I will not compare it to other API.

Lucene is a very mature API and can be found in NetBeans IDE, Liferay, JackRabbit among others. IBM has written a very good document about the Lucene architecture, therefore I will not delve into it here.

Lucene, alone, is pretty much useless as any other API. Let’s now introduce Nutch. Nutch is a web crawler built-on top of Lucene to provide file crawling capabilities. Nutch was designed to handle large amount of data from the internet (http). Due to its plugin architecture, it was later extended to provide local network crawling such as FTP, databases and Microsoft Windows Shares (I am the author of the protocol-smb plugin and co-author of the index-extra plugin found on the Nutch site). We had extended Nutch and turned it into an Enterprise Search application but most of the source codes were locked behind closed doors (company politics).Anyway, Nutch has evolved to become but still very complex in its inner working. The initial Nutch was developed to process data in a batch but there are ways to turn it real-time but that’s for another day. Ok, so Nutch is good for crawling and indexing of data but it does not handle search directly. There is a web application available with Nutch but it is quite poor so let’s now introduce the Solr.

Solr is a powerful web-based search server built-on top of Lucene. The application was developed by CNET Networks and donated to the Apache Foundation. I believe, not too sure so I might need some references here, Solr was powering the search feature on their site but it is definitely used internally by the company. Late 2009, Lucid Imaginations receives $7.5 million in funding to provide commercial services built around Solr (and Lucene possibly?). Here is a very good presentation about Solr. Solr is a very good indexing engine. The keyword here is “indexing engine”. It does not have any support for crawling data therefore requiring the developer to create applications that will feed it the data to index. I do believe that it is a good feature of the application as it gives the ability to integrate with various systems as long as they can post data over HTTP.

Nutch is a good crawler but it does not provide an enterprise-grade search interface to its data. Solr, in the other hand, is powerful indexer and has an enterprise grade search interface but it does not know how to gather data in its own. I am sure by now it has become obvious how we can integrate them both together.

We want Nutch to gather the data, by-pass its indexing cycle and feed the data directly to Solr. Lucid Imagination has a good tutorial about it here.

After reading the tutorial from Lucid Imagination, you will notice that Nutch is run by executing some bash files. This is something I strongly disagree with. If Nutch is based on Java (an OS independent language), why do we need to execute some UNIX/LINUX shell script. Also, the fact we need to install CygWin on MS Windows platform to be able to run is a big negative for me. I wrote a simple Java application that will launch Nutch and send the indexing to Solr but as you can see in the source code, you still need a UNIX like environment to run successfully. You can write a platform independent version by looking up Nutch API and calling the methods directly.

Well, I hope that this entry help you understand how to use Nutch and Solr built-on top of Apache Lucene. If you need any clarification, leave comments and I will try to gave ASAP if time permits.

package com.etapix.nutchsolr;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileFilter;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 *
 * @author Armel Nene
 */
public class Indexer {

    public static void main(String args[]) {
        if (args.length < 3) {
            System.out.println("Usage:" +
                    "ncrawlName        -   This will be used to store crawler files in CrawlName directory" +
                    "nurlFolder        -   The path to the folder containing the URL to crawl" +
                    "nsolrUrl          -   The URL to the Solr server");
            return;
        }
        String crawlerName = args[0];
        String urlFolder = args[1];
        String solrUrl = args[2];
        String inject = "bash bin/nutch2.sh inject " + crawlerName + "/crawldb " + urlFolder;
        String generate = "bash bin/nutch2.sh generate " + crawlerName + "/crawldb " + crawlerName + "/segments -topN 10 -numFetchers 5";
        String export = crawlerName + "/segments/";

        String invertLinks = "bash bin/nutch2.sh invertlinks " + crawlerName + "/linkdb -dir " + crawlerName + "/segments";
        String indexSolr = "bash bin/nutch2.sh solrindex " + solrUrl + " " + crawlerName + "/crawldb " + crawlerName + "/linkdb " + crawlerName + "/segments/*";
        try {
            System.out.println("Injecting URLs in crawldb");
//            int state = 0;
            InputStream in = Runtime.getRuntime().exec(inject).getInputStream();
            System.out.println(convertStreamToString(in));


//            state = Runtime.getRuntime().exec(inject).waitFor();
//            System.out.println("process completed: " + state);

            for (int i = 0; i < 3; i++) {
                System.out.println("Generating segments");
                in = Runtime.getRuntime().exec(generate).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Setting environment variable $SEGMENT");
//            String segs = convertStreamToString(Runtime.getRuntime().exec("ls -tr " + crawlerName + "/segments|tail -1").getInputStream());

                String segments = export + lastFileModified(export).getName();
                System.out.println("$segments: " + segments);
//            in = Runtime.getRuntime().exec(export + segs).getInputStream();
            System.out.println(convertStreamToString(in));

                String fetch = "bash bin/nutch2.sh fetch " + segments + " -noParsing";
                String parse = "bash bin/nutch2.sh parse " + segments;
                String update = "bash bin/nutch2.sh updatedb " + crawlerName + "/crawldb " + segments + " -filter -normalize";

                System.out.println("fetch segments");
                in = Runtime.getRuntime().exec(fetch).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Parse segments");
                in = Runtime.getRuntime().exec(parse).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Update crawldb");
                in = Runtime.getRuntime().exec(update).getInputStream();
                System.out.println(convertStreamToString(in));
            }
            System.out.println("Inverting links");
            in = Runtime.getRuntime().exec(invertLinks).getInputStream();
            System.out.println(convertStreamToString(in));

            System.out.println("Indexing contents to Solr " + solrUrl);
            in = Runtime.getRuntime().exec(indexSolr).getInputStream();
            System.out.println(convertStreamToString(in));

        } catch (Exception ex) {
            Logger.getLogger(Indexer.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    public static String convertStreamToString(InputStream is) {
        /*
         * To convert the InputStream to String we use the BufferedReader.readLine()
         * method. We iterate until the BufferedReader return null which means
         * there's no more data to read. Each line will appended to a StringBuilder
         * and returned as String.
         */
        BufferedReader reader = new BufferedReader(new InputStreamReader(is));
        StringBuilder sb = new StringBuilder();

        String line = null;
        System.out.println("Now converting inputstream to text");
        try {
            while ((line = reader.readLine()) != null) {
                sb.append(line + "n");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        System.out.println("Finish converting to text");
        return sb.toString();
    }

    public static File lastFileModified(String dir) {
        File fl = new File(dir);
        File[] files = fl.listFiles(new FileFilter() {

            public boolean accept(File file) {
                return file.isDirectory();
            }
        });
        long lastMod = Long.MIN_VALUE;
        File choice = null;
        for (File file : files) {
            if (file.lastModified() > lastMod) {
                choice = file;
                lastMod = file.lastModified();
            }
        }
        return choice;
    }
}