4124677046_69aa7a07b3

GWT Visualization Example – Annotated Time Line Chart Tutorial

If you are a server-side Java developer such as myself, I am sure you want an easy way to create AJAX based application but pure JavaScript (ECMAScript) is not your strong skills. The beauty of JavaScript is that it runs on any web server, no need for a servlet container or anything of that sort.

So, I have been writing AJAX based UI using the GWT framework for the past year and half. I am still not a web developer but I understand enough CSS to spice up my site. Anyway, I am writing this tutorial because I think that the GWT Visualization team, even tough they did a good job, over complicates the creation of charts in their tutorial. I was trying to create an Annotated Time Line chart so I looked up my previous code I wrote for the GWT Portlet JSR-168, see here. I also decided to run a search (don’t be evil, Google is your friend) to try to find different examples and hopefully some nice custom charts. First of all, based on the search result; I thought that some of examples, including Google’s own, were confusing and over complex. Most examples uses AJAXLoader to load the Visualization API, but you shouldn’t have to use this. This is the simplest way to create a GWT Annotated Time Line Chart.

gwt annotated time line chart

1. Using your favorite development tool with support for GWT framework, create a new Java web application, you can export the compiled JavaScript later to a non-based web application, if you use an IDEW you should code-completion support.

2. Make you sure you have included the GWT Visualization API module in your classpath.

3. In your project source code root directory, you should <AppNamexxxx>.gwt.xml file. make sure to add teh following line to it in order to make the  module available to your application.

<inherits name=”com.google.gwt.visualization.Visualization”/>

4. In order to keep this tutorial short and straight to point, I have included the chart creation code in my <AppNamexxxx>EntryPoint.java class

public class MainEntryPoint implements EntryPoint
{
    
    
    /**
    * Creates a new instance of MainEntryPoint
    */
    public MainEntryPoint()
    {
        
        
    }
    
    
    
    /**
    * The entry point method, called automatically by loading a module
    * that declares an implementing class as an entry-point
    */
    public void onModuleLoad()
    {
        
        Runnable onLoadCallback = new Runnable()
        {
            
            
            public void run()
            {
                
                DataTable data = createHistoryTable(result);
                AnnotatedTimeLine.Options options = AnnotatedTimeLine.Options.create();
                options.setDisplayAnnotations(true);
                options.setDisplayZoomButtons(true);
                options.setScaleType(AnnotatedTimeLine.ScaleType.ALLFIXED);
                options.setLegendPosition(AnnotatedTimeLine.AnnotatedLegendPosition.SAME_ROW);
                AnnotatedTimeLine atl = new AnnotatedTimeLine(data, options, “600px”, “200px”);
                RootPanel.get().add(atl);
                
            }
            
            
            
        }      
        ;
        VisualizationUtils.loadVisualizationApi(onLoadCallback, AnnotatedTimeLine.PACKAGE);
        
    }
    
    
    
    // This method will create the Data used by the chart
    private DataTable createHistoryTable()
    {
        
        DataTable data = DataTable.create();
        data.addColumn(AbstractDataTable.ColumnType.DATE, “Date”);
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, “Price”);
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, “Low”);
        data.addColumn(AbstractDataTable.ColumnType.NUMBER, “High”);
        data.addRows(5);
                  
            DateTimeFormat dtf = DateTimeFormat.getFormat(“yyyy-MM-dd”);
            data.setValue(0, 0, dtf.parse(“2009-11-21”));
            data.setValue(0, 1, 100);
            data.setValue(0, 2, 120);
            data.setValue(0, 3, 90);

            data.setValue(1, 0, dtf.parse(“2009-11-22”));
            data.setValue(1, 1, 90);
            data.setValue(1, 2, 110);
            data.setValue(1, 3, 100);

            data.setValue(2, 0, dtf.parse(“2009-11-23”));
            data.setValue(2, 1, 100);
            data.setValue(2, 2, 180);
            data.setValue(2, 3, 80);

            data.setValue(3, 0, dtf.parse(“2009-11-20”));
            data.setValue(3, 1, 130);
            data.setValue(3, 2, 100);
            data.setValue(3, 3, 130);
          

            data.setValue(4, 0, dtf.parse(“2009-11-19”));
            data.setValue(4, 1, 130);
            data.setValue(4, 2, 170);
            data.setValue(4, 3, 150);
            
        }
        return data;  
    }  
}

Don’t forget to import all the necessary classes and voila!

I hope this will save you some time in creating your charts.

 

4099041220_b0ccec7f10

Develop High Transaction Web Applications with Java MySQL & NetBeans

This entry is a brief tutorial on how to develop high transactional web application. I will be looking at how to develop a high transactional application while maintaining data integrityFor the purpose of keeping this entry simple, I will be using some RAD tools, the NetBeans IDE, to generate most of the code.

In order to follow the tutorial, you will need the following:

  1. NetBeans IDE 6.7+
  2. Java JDK 1.6+ (my version is 1.6.0_17)
  3. MySQL 5.1+
  4. MySQL Sakila database
  5. Apache Tomcat 6.x
  6. An understanding of JPA transactions and ReSTful web services

This is my definition of High Transaction Applications:

A high transaction application is one that can serve multiple simultaneous requests from clients and keep them secure from each other. The application has only two purposes: read or write data to/from a repository either JMS or DB. The transactions have to meet the ACID criteria in order to be deployed in the real world.

Ok, the above is my definition and you are free to redefine it. I am going to build a web application that will be an n-tier application:

  • Database back-end (MySQL)
  • Entity classes
  • ReSTful Web Services to allow other developers to integrate the application with theirs
  • A web based front-end

For the simplicity of the article, I will not implement any security such as user or application level security (authentication and database table privileges). This tutorial is mostly geared toward the newbies but I am sure that more advanced developers would benefit too.

Let’s get coding.

  1. Make sure that you have loaded the MySQL Sakila database into your MySQL database. You can download the Sakila database from the NetBeans plugin centre (see here).
  2. Create a new web application and name whatever you like. I have named mine “WebApplication”. I will now refer the application as WebApplication. Make sure to choose Tomcat as your deployment server
  3. Add the MySQL driver “mysql-connector-java-5.0.7-bin” to the WebApplication libraries. NetBeans will work and connect to the DB even without the driver but once you deploy your application to a server, the application will not be able to connect to the DB and throw a ClassNotFoundException com.mysql.jdbc.driver.
  4. We are going to develop the back-end first. NetBeans makes it very easy for us to create Entity classes from database. I would recommend newbies to learn how to manually create Entity classes and configure the persistence.xml file. This tutorial makes use of JPA but one should ignore the drawbacks of JPA/ ORM frameworks.
  5. Right click on the project name and choose Entity Classes from Database… On the next screen choose the “filmActor” table and click on the Add button. Make sure that “Include Related Tables” box is checked underneath the Selected Tables panel. The screenshot does not show the “filmActor” table as I have previously generated the Entity class but I am sure you get the idea.

  6. If you are required to create a Persistence Unit, click on the persistence button -> you can accept the default name -> choose your Persistence Library -> choose your database connection -> choose “None” for Table Generation Strategy -> click Finish
  7. After choosing the table to generate the Entity from click next -> fill Package name -> tick Generate Name Query… -> click Finish
  8.  

    NetBeans generates all the Entity classes based on the database table that you have chosen. The next thing that we want to do is generate a set of ReSTful web services from the generated Entity classes. Again, NetBeans facilitates the work for us (it is important that you also know how to create the classes manually or you will not know how to debug them if there is any problem in the future).

  9. Right click on the project name “WebApplication” -> RESTful Web Services from Entity Classes… -> Choose the Entity Classes that you would like to generate the WS for and click add or add all -> click next -> on the following screen, accept the default values and click Finish

  10. You can go ahead and test your ReSTFul Web Services by right clicking on the name of the application and click on Test RESTful Web Services

  11. The previous step will launch your web browser within which you can test your web services (click on the node on the left and see the queries on the right)

  12. Back in the NetBeans IDE; right click on the project name “WebApplication” -> click on JSF Pages from Entity Classes… (The JSF pages will not use the web services as there are packaged together with the Entity Classes. This will improve performances and still allow external applications to integrate). Choose the Entity Classes that you would like to generate the pages for -> click next. On the final screen, fill in the package name for the JPA Controller and JSF Classes -> click finish

    NetBeans will generate the necessary files to create a CRUD application with a user interface. I suggest that you familiarize yourself with the generated code.

  13. Expand the Configuration folder under your project name “WebApplication” and the web.xml file. At around line 38, change the content of the welcome-file-list to look as follow
    <welcome-file-list>
            <welcome-file>faces/welcomeJSF.jsp</welcome-file>
    </welcome-file-list>

    This will make the generate JSF page to be the landing page for the application when requested. Make sure that you do have a “welcomeJSF.jsp” file before making the change.

  14. Right click on the application name -> Run. The application should load in your web browser. Now go on, play around with the application. And why not create a client to send request to the web services? (not today)

You can load test your application by using Apache JMeter. It is easy to run and configure. If you want to take a look at how JPA implements the ACID features, browse to your controller classes. Here is a short introduction to JPA transaction.

Hope you enjoyed and if you need any clarification, just leave me a comment and try to get back to you ASAP (if time permits).

Develop Your Own Google with Apache Lucene (Java Nutch Solr)

Apache Lucene is Open Source API that allows a Java developer (.Net libraries available) to write indexing and full-text search capable applications. I have been writing applications based on Lucene for the last 3 years and some of the applications have been deployed at large corporations. I know there are other libraries available to developers who wish to write indexing engine but this blog will solely focus on Apache Lucene. I will not compare it to other API.

Lucene is a very mature API and can be found in NetBeans IDE, Liferay, JackRabbit among others. IBM has written a very good document about the Lucene architecture, therefore I will not delve into it here.

Lucene, alone, is pretty much useless as any other API. Let’s now introduce Nutch. Nutch is a web crawler built-on top of Lucene to provide file crawling capabilities. Nutch was designed to handle large amount of data from the internet (http). Due to its plugin architecture, it was later extended to provide local network crawling such as FTP, databases and Microsoft Windows Shares (I am the author of the protocol-smb plugin and co-author of the index-extra plugin found on the Nutch site). We had extended Nutch and turned it into an Enterprise Search application but most of the source codes were locked behind closed doors (company politics).Anyway, Nutch has evolved to become but still very complex in its inner working. The initial Nutch was developed to process data in a batch but there are ways to turn it real-time but that’s for another day. Ok, so Nutch is good for crawling and indexing of data but it does not handle search directly. There is a web application available with Nutch but it is quite poor so let’s now introduce the Solr.

Solr is a powerful web-based search server built-on top of Lucene. The application was developed by CNET Networks and donated to the Apache Foundation. I believe, not too sure so I might need some references here, Solr was powering the search feature on their site but it is definitely used internally by the company. Late 2009, Lucid Imaginations receives $7.5 million in funding to provide commercial services built around Solr (and Lucene possibly?). Here is a very good presentation about Solr. Solr is a very good indexing engine. The keyword here is “indexing engine”. It does not have any support for crawling data therefore requiring the developer to create applications that will feed it the data to index. I do believe that it is a good feature of the application as it gives the ability to integrate with various systems as long as they can post data over HTTP.

Nutch is a good crawler but it does not provide an enterprise-grade search interface to its data. Solr, in the other hand, is powerful indexer and has an enterprise grade search interface but it does not know how to gather data in its own. I am sure by now it has become obvious how we can integrate them both together.

We want Nutch to gather the data, by-pass its indexing cycle and feed the data directly to Solr. Lucid Imagination has a good tutorial about it here.

After reading the tutorial from Lucid Imagination, you will notice that Nutch is run by executing some bash files. This is something I strongly disagree with. If Nutch is based on Java (an OS independent language), why do we need to execute some UNIX/LINUX shell script. Also, the fact we need to install CygWin on MS Windows platform to be able to run is a big negative for me. I wrote a simple Java application that will launch Nutch and send the indexing to Solr but as you can see in the source code, you still need a UNIX like environment to run successfully. You can write a platform independent version by looking up Nutch API and calling the methods directly.

Well, I hope that this entry help you understand how to use Nutch and Solr built-on top of Apache Lucene. If you need any clarification, leave comments and I will try to gave ASAP if time permits.

package com.etapix.nutchsolr;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileFilter;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 *
 * @author Armel Nene
 */
public class Indexer {

    public static void main(String args[]) {
        if (args.length < 3) {
            System.out.println("Usage:" +
                    "ncrawlName        -   This will be used to store crawler files in CrawlName directory" +
                    "nurlFolder        -   The path to the folder containing the URL to crawl" +
                    "nsolrUrl          -   The URL to the Solr server");
            return;
        }
        String crawlerName = args[0];
        String urlFolder = args[1];
        String solrUrl = args[2];
        String inject = "bash bin/nutch2.sh inject " + crawlerName + "/crawldb " + urlFolder;
        String generate = "bash bin/nutch2.sh generate " + crawlerName + "/crawldb " + crawlerName + "/segments -topN 10 -numFetchers 5";
        String export = crawlerName + "/segments/";

        String invertLinks = "bash bin/nutch2.sh invertlinks " + crawlerName + "/linkdb -dir " + crawlerName + "/segments";
        String indexSolr = "bash bin/nutch2.sh solrindex " + solrUrl + " " + crawlerName + "/crawldb " + crawlerName + "/linkdb " + crawlerName + "/segments/*";
        try {
            System.out.println("Injecting URLs in crawldb");
//            int state = 0;
            InputStream in = Runtime.getRuntime().exec(inject).getInputStream();
            System.out.println(convertStreamToString(in));


//            state = Runtime.getRuntime().exec(inject).waitFor();
//            System.out.println("process completed: " + state);

            for (int i = 0; i < 3; i++) {
                System.out.println("Generating segments");
                in = Runtime.getRuntime().exec(generate).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Setting environment variable $SEGMENT");
//            String segs = convertStreamToString(Runtime.getRuntime().exec("ls -tr " + crawlerName + "/segments|tail -1").getInputStream());

                String segments = export + lastFileModified(export).getName();
                System.out.println("$segments: " + segments);
//            in = Runtime.getRuntime().exec(export + segs).getInputStream();
            System.out.println(convertStreamToString(in));

                String fetch = "bash bin/nutch2.sh fetch " + segments + " -noParsing";
                String parse = "bash bin/nutch2.sh parse " + segments;
                String update = "bash bin/nutch2.sh updatedb " + crawlerName + "/crawldb " + segments + " -filter -normalize";

                System.out.println("fetch segments");
                in = Runtime.getRuntime().exec(fetch).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Parse segments");
                in = Runtime.getRuntime().exec(parse).getInputStream();
                System.out.println(convertStreamToString(in));

                System.out.println("Update crawldb");
                in = Runtime.getRuntime().exec(update).getInputStream();
                System.out.println(convertStreamToString(in));
            }
            System.out.println("Inverting links");
            in = Runtime.getRuntime().exec(invertLinks).getInputStream();
            System.out.println(convertStreamToString(in));

            System.out.println("Indexing contents to Solr " + solrUrl);
            in = Runtime.getRuntime().exec(indexSolr).getInputStream();
            System.out.println(convertStreamToString(in));

        } catch (Exception ex) {
            Logger.getLogger(Indexer.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    public static String convertStreamToString(InputStream is) {
        /*
         * To convert the InputStream to String we use the BufferedReader.readLine()
         * method. We iterate until the BufferedReader return null which means
         * there's no more data to read. Each line will appended to a StringBuilder
         * and returned as String.
         */
        BufferedReader reader = new BufferedReader(new InputStreamReader(is));
        StringBuilder sb = new StringBuilder();

        String line = null;
        System.out.println("Now converting inputstream to text");
        try {
            while ((line = reader.readLine()) != null) {
                sb.append(line + "n");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        System.out.println("Finish converting to text");
        return sb.toString();
    }

    public static File lastFileModified(String dir) {
        File fl = new File(dir);
        File[] files = fl.listFiles(new FileFilter() {

            public boolean accept(File file) {
                return file.isDirectory();
            }
        });
        long lastMod = Long.MIN_VALUE;
        File choice = null;
        for (File file : files) {
            if (file.lastModified() > lastMod) {
                choice = file;
                lastMod = file.lastModified();
            }
        }
        return choice;
    }
}