Big Data Architecture Best Practices

The marketing department of software vendors have done a good job making Big Data go mainstream, whatever that means. The promise of we can achieve anything if we make use of Big Data; business insight and beating our competitions to submission. Yet, there is no well-publicised Big Data successful implementation. The question is: why not? Clearly this silver bullet where businesses have seen billions of dollars invested in but no return on investment! Who is to blame? After all, businesses do not have to publicise their internal processes or projects. I have a different view to that and the cause is on the IT department. Most Big Data projects are driven by the technologist not the business there is create lack of understanding in aligning the architecture with the business vision for the future.

The Preliminary Phase

Big Data projects are not different to any other IT projects. All projects spur out of business needs / requirements. This is not The Matrix; we cannot answer questions which have not been asked yet. Before any work begin or discussion around which technology to use, all stakeholders need to have an understanding of:

  • The organisational context
  • The key drivers and elements of the organisation
  • The requirements for architecture work
  • The architecture principles
  • The framework to be used
  • The relationships between management frameworks
  • The enterprise architecture maturity

In the majority of cases, Big Data projects involves knowing the current business technology landscape; in terms of current and future applications and services:

  • Strategies and business plans
  • Business principles, goals, and drivers
  • Major framework currently implemented in the business
  • Governance and legal frameworks
  • IT strategy
  • Pre-existing Architecture Framework, Organisational Model, and Architecture repository

The Big Data Continuum

Big Data projects are not and should never been executed in isolation. The simple fact that Big Data need to feed from other system means there should a channel of communication open across teams. In order to have a successful architecture, I came up with five simple layers/ stacks to Big Data implementation. To the more technically inclined architect, this would seem obvious:

  • Data sources
  • Big Data ETL
  • Data Services API
  • Application
  • User Interface Services
Big Data Protocol Stack

Data Sources

Current and future applications will produce more and more data which will need to be process in order to gain any competitive advantages from them. Data comes in all sorts but we can categorise them into two:

  1. Structured data – usually stored following a predefined formats such as using known and proven database techniques. Not all structured data are stored in database as there are many businesses using flat files such as Microsoft Excel or Tab Delimited files for storing data
  2. Unstructured data – businesses generates great amount of unstructured data such emails, instant messaging, video conferencing, internet, flat files such documents and images, and the list is endless. We call the data “unstructured” as they do not follow a format which will make facilitate a user to query its content.

I have spent a large part of my career working on Enterprise Search technology before even “Big Data” was coined. Understanding where the data is coming from and in what shape is valuable to a successful implementation of a Big Data ETL project. Before a single a line of programming code is written, architects will have to try and normalise the data to common format.

Big Data ETL

This is the part that excites technologists and especially the development teams. There are so many blogs and articles published every day about Big Data tools that this creates confusions among non-tech people. Everybody is excited about processing petabytes of data using the coolest kid on the block: Hadoop and its ecosystem. Before we get carried away, we first need to put some baseline in place:

  • Real-time processing
  • Batch processing
Big Data – Data Consolidation

The purpose of Extract Transform Load projects, regardless of using Hadoop or not, is to consolidate the data into a single view Master Data Management for querying on demand. Hadoop and its ecosystem deals with the ETL aspect of Big Data not the querying part. The tools used will heavily depends of processing need of the project: either Real-time or batch; i.e. Hadoop is a batch processing framework for large volume of data. Once the data has been processed, the Master Data Management system (MDM) can be stored in a data repository such as NoSQL based or RDBMS – this will only depends on the querying requirements.

Data Services API

As most of the limelight goes to the tools for ETL, a very important area is usually overlooked until later almost as a secondary thought. MDM will need to be stored in a repository in order for the information to be retrieve when needed. In a true Service Oriented Architecture spirit, the data repository should be able to expose some interfaces to external third party applications for data retrieval and manipulation. In the past, MDM were mostly created in RDBMS and retrieval and manipulation were carried out through the use of the Structured Query Language. Well this does not have to change but architects should be aware of other forms of database such NoSQL types. The following questions should be asked when choosing a database solution:

  • Is there are standard query language
  • How do we connect to the database; DB drivers or available web services
  • Will the database scale when the data grows
  • What security mechanism are in place for protecting some or whole data

Other questions specific to the project should also be included in the checklist.

Business Applications

So far, we have extracted the data, transformed and loaded it into a Master Data Management system. The normalised data is now exposed through web services (or DB drivers) to be used by third party applications. Business applications are the reason why to undertake Big Data projects in the first place. Some will argue that we should hire Data Scientists (?). According many blogs, Data Scientist roles is to understand the data, explore the data, prototype (new answers to unknown questions) and evaluate their findings. This is interesting as it reminds me the motion picture The Matrix, where the Architect knew the answers to the questions before Neo has even asked them yet and decides which one are relevant or not. Now this is not how businesses are run. It will be extremely valuable if the data scientist may suggest subconsciously (Inception) a new way to do something but most of the time the questions will come from business to be answered by the Data Scientist or whoever knows the data. The business applications will be the answer to those questions.

User Interfaces Services

User interfaces are the make or break of the project; a badly designed UI will affect adoption regardless of the data behind it, an intuitive design will increase adoption and maybe user will start questioning the quality of the data. Users will access the data differently; mobile, TV and web as an example. Users will usually focus on a certain aspect of the data and therefore they will require the data to be presented in a customised way. Some other users will want the data to be available through their current dashboard and match their current look and feel. As always, security will also be a concern. Enterprise portal have been around for a long time and they are usually used for data integration projects. Nevertheless, standards such as Web Services for Remote Portlets (WSRP) make it possible for User Interfaces to be served through Web Service calls.


This article show the importance of architecting a Big Data project before embarking on the project. The project needs to be in line with the business vision and have a good understanding of the current and future technology landscape. The data needs to bring value to the business and therefore business needs to be involved from the outset. Understanding how the data will be used is key to its success and taking a service oriented architecture approach will ensure that the data can serve many business needs.


4 Reasons Football Teams Should Use Big Data and IoT

This is a case why football teams (or soccer for my transatlantic friends) should make greater use of the Internet of Things and big data.
Here are some 5 simple reasons: 

1. Player health tracking

In 2012, then only 24 years of age, Fabrice Muamba a footballer from Bolton Wanderers suffered an heart attack on live TV during an FA Cup game. This was not the first such incident on live TV; Marc-Vivien Foe, the Cameroonian player, died on June 26 2003 during a confederation cup game against France. Not to mentioned what happens off-pitch, football club have a duty to look after their players and that’s including their health.Let’s clear, I’m not saying that current health wearable devices could have saved them but they could have played a great role. Read the signs early to avoid the disaster as prevention will always be better than the cure.So how can be big data helps us improve the athletes health? You may ask. I will simplify it for illustration purposes. Teams need to put in a place an IoT and Big Data strategy where athlete will wear various devices during training and games. Think about a heart monitoring bracelets that players can wear during training and games for the physiotherapy team to analyse later or in real time. The devices will record the time, number of foot steps (within a period – in order to differentiate between walking and running) and heart rate. Let’s say that player John Smith health is changing during the game, the physios will get alerted of the change of condition and can take appropriate actions. Maybe we can add devices to shin pads so that players do not have to wear uncomfortable devices.

2. Player positioning and technique

Great players do this naturally but not all players are born equal. I remember when I used to play football in my early teen for USO back in France. In training, we would learn how to shoot, head the ball and basic skills. The trainer would then let us know what we did wrong and how to ameliorate our techniques. Now professional players have a large team working with them daily; strikers will have coaches to help them for attacking positioning, goal keeper will have coaches too and etc… The current tech used by the teams are heavy duty machine which are employed behind close doors. How can IoT make improve on this? Again through wearable technology, we can track players movement. GPS integrated devices will track player movement over a map (football field), chin pad enabled devices will help visualise the player shooting techniques. The list of devices are only limited to your own imagination but the possibilities are endless. Players can then analyse their own data or with the help of their coach, see how they can improve. No more killing pigeons with those bad shots over the bar!

3. Team metrics

Now what is the point of all that data generated if not put to use? We have health data and game metrics (positioning and technique) about the whole team. The players health will give us an indication of who is fit for game. Being a big fan of the beautiful game, we have observed how various conditions can affect a player performance such as weather and pitch condition. The data scientists would analyse who can perform better under the current conditions. Don’t get it wrong; top players, if fit for the game, are indispensable to the team but all advices should be welcomed. You will have what we call “the best team on paper”. Gamblers use this technique in various events to pick their winners/ losers.

4. Opponents analysis

Usually we can find a wealth of information about teams from various sources such as sport sites, videos and blogs. The more you know about your opponents the better you will perform (Sun Tzu – Art of War). Team changes twice a year (transfer window) for those who can afford it. Is a team really as good as their last game? This question can be answered with big data analytics. Taking the conditions into consideration and our own team metrics, we can pick a strategy (team formation, defensive or attacking…). The big data gives (all) the information that we need in order to make decisions.


The Internet of Things (IoT) and Big Data provides an opportunity, even if not new, to gather information and make executive decisions much faster than before. There should not be any reasons not to have the full picture before hand. Football, as any other sports, is very competitive and everyone wants to create advantage. IoT devices are getting cheaper and more data can be collected for analysis. Big Data has the tools but not a silver bullet. We can prevent tragedies by tracking our players health and help them improve by analysing their techniques. Let’s realistic that winning a game is probably beyond Big Data and IoT alone but they will play a big part in shaping the future of the game.