Meaning of data exploration and its technique

One of the most conspicuous trends of today’s world is big data, thanks to the proliferation of advanced technology, there are huge amounts of data available in the system and it is very hard to analysis the data due to this. To overcome this concern, Analyst used Data exploration. It is a technique which helps for finding relevant information.

Data exploration is used for getting didactic information by data regulars for better analysis. Search used by data consumers to form true analysis from the information gathered. According to the Burnham & Anderson,(2002) Data Exploration is an indispensable and advanced procedure for analysing the data. It is a procedure, which mainly used in data warehouse. Thanks to the different systems and sources data can be seen in different format. With many customers claiming that it is very hard to get correct information from the database due to existence of big data and it creates problems for decision making process. Relevant data is needed for tasks such as statistical reporting, trend spotting and pattern spotting. Data exploration is the process of gathering such relevant data.

Techniques of data exploration

There are two main methodologies or techniques used to retrieve relevant data from large and unorganised database. They are the manual and automatic methods. Data exploration is the first step in data analysis and typically involves summarising the main characteristics of a data set. The manual method is also called data exploration. For instance, Analysts commonly use data visualisation software for data exploration because it allows users too quickly and simply view most of the relevant features of their data set. By using visualisation software users can identify variables that are likely to have interesting observations with the help of scatter plots or bar charts. -- Users can see if two or more variables correlate and determine if they are good candidates for further in-depth analysis. There are many software available in the market such as R programming and SAP business object Cloud for visualisation. In this type of methodology it can be apply to data of any type or size but because of its manual nature, there are many options available to use data exploration for smaller data sets.

Second is the automatic method which also known as data mining. It generally refers to gathering relevant data from large databases. While data exploration refers to a data user being able to find his or her way through large amounts of data in order to gather necessary information (Keim 2001)

R programming is a programming language which is mainly used in predictive analysis. It is an open source and scripting language which helps in data exploration. It was developed in1995 by Ross and Robert. This is mainly used in data exploration for getting relevant information.

Another pivotal aspect related with this fact that there is another tool available such as SAP Business Object cloud by which we can explore the data in better way for example in below diagram by using this tools we are trying to explore data that how many refugees coming in Australia from various countries.

Source- Historical Statistics 2016, Refugee Council of Australia

Difference between data exploration and predictive  analysis.

Data Exploration is the first step in any data analysis. It involves summarising the main characteristic of any database or dataset. It’s mostly done in statistical software varying in advancement levels, depending upon the complexity of the dataset. It can be conducted using visual analytics tools, which has been discussed in aforementioned arguments.

Predictive Analysis falls under advanced analytics, and is used to make predictions about unknown events that might unfold in the future. It can be said that Predictive analysis uses a host of different software, pairing them with many techniques ranging from artificial intelligence (AI), and machine learning, to analyse already existing data and make predictions regarding its course in the future. The difference is that data Exploration uncovers the complex, and oftentimes unseen relationships between measurable variables while predictive analysis offers outcomes and possibilities in the future of the variables, from the variables (Keim 2001)

Impacts of exploration of data

·         Understand the data

In this section for data exploration understanding the data is important things so that’s why there are existence of many questions; such as how many fields are available and what types of data are represented and what type of units are included in the data.

·         Organize and subset the database

After understanding the data another main step in data exploration is there are two types of tools are popular such as sort and filter by which there is possibility to sort or filter the data for making decisions about models. By filtering there is possibility to investigate large database and extract what interest us.

·         Examine individual variables and their distributions

In this section there is possibility that we can find numerical variables from lowest to highest and common way to summarize data is the histogram.

·         Calculate summary measures for individual variables

After examine individual variables and their distributions there is another process in data exploration is, Excel is also able to provide useful functions in relation with investigating individual variables and it can be also useful to identify or count specific variable.

·         Examine relationship among variable

By using graphical methods there is possibility to track relationships.

 

Steps of Data Exploration and Preparation

For building predictive model there are many steps involved to understand,clear and prepare data;

1.      Variable Identification: In this Step we have information about input and output of variables. For instance, we want to predict, whether the students will play cricket or not. Here you need to identify predictor variables, target variable, data type of variables and category of variables.

2.      Univariate Analysis: we explore variables one by one. Method to perform univariate analysis will depend on whether the variable type is categorical or continuous.

3.      Bi-variate Analysis: Bi-variate Analysis finds out the relationship between two variables. Here, we look for association and disassociation between variables at a pre-defined significance level. We can perform bi-variate analysis for any combination of categorical and continuous variables. The combination can be: Categorical & Categorical, Categorical & Continuous and Continuous & Continuous. Different methods are used to tackle these combinations during analysis process.

4.      Missing values treatment: Missing data in the training data set can reduce the power / fit of a model or can lead to a biased model because we have not analysed the behaviour and relationship with other variables correctly. It can lead to wrong prediction or classification

5.      Outlier treatment: Outlier is a commonly used terminology by analysts and data scientists as it needs close attention else it can result in wildly wrong estimations. Let’s take an example, we do customer profiling and find out that the average annual income of customers is $0.8 million. But, there are two customers having annual income of $4 and $4.2 million. These two customers annual income is much higher than rest of the population

6.      Variable transformation: Transformation refers to the replacement of a variable by a function. For instance, replacing a variable x by the square / cube root or logarithm x is a transformation. In other words, transformation is a process that changes the distribution or relationship of a variable with others.

7.      Variable creation: Variable creation is a process to generate a new variables / features based on existing variable(s). For example, say, we have date (dd-mm-yy) as an input variable in a data set. We can generate new variables like day, month, year, week, weekday that may have better relationship with target variable. This step is used to highlight the hidden relationship in a variable (Zuur 2010)

Submitted by- Harbinder Kaur Bhullar(4510860)

Savita Rani (4491997)

Submitted to- Dr Shah Miah

Date-15 September 2017       

References:-

·         Burnham, K.P. & Anderson, D.R. (2002) Model Selection and Multimodel Inference. A Practical Information–Theoretic Approach, 2nd edn. Springer, New York.

·         http://www.learn.geekinterview.com/data-warehouse/data-analysis/what-is-data-exploration.html

·         https://www.quora.com/What-is-the-difference-between-data-exploration-and-predictive-analytics

·         D. Keim (2001), “Visual exploration of large databases,” Communications of the ACM, vol. 44, no. 8, pp. 38–44, 2001

·         Zuur, A.F., Ieno, E.N. and Elphick, C.S., 2010. A protocol for data exploration to avoid common statistical problems. Methods in Ecology and Evolution, 1(1), pp.3-14.

Data Analysis in R

Data preparation is an essential activity in data analysis. There are several statistical methods that are used in data analysis which include: Linear model, logistic regression, k-means and decision trees. In this blog we will explain these statistical methods.

Data preparation

Data preparation, also known as data pre-processing is manipulation of the data in a form suitable for additional analysis and processing. Many different tasks are involved in this process and these tasks cannot be fully automated. More of the data preparation activities are tedious, routine and time consuming. According to some estimations around 60% to 80% of the time spent in data mining projects belongs to the process of data preparation. Data preparation is an essential activity for success in data mining projects. The process of data preparation improves the data quality and improves the quality of the data mining results. The process of data preparation involves several steps like checking the data for accuracy, checking or logging the data in, entering the data into the computer and transforming the data.

Linear model

Linear model or linear regression is a statistical approach for modelling of the relationship between a scalar dependent variable and one or more explanatory variables. There are two cases of linear regression:

• simple linear regression where we have one explanatory variable
• multiple linear regression where we have two or more explanatory variables.

In linear regression, data is modeled with using of linear predictor functions and unknown model parameters are estimated from the data. These models are called linear models.

Predictions

For generating predictions and residuals from the model, rxPredict() is the function which is used on various types of models. Some of the key arguments used in the formation of the syntax are as follows:

Logistic regression

Next we move on to logistic regression or a logit model. It is used to represent binary outcome variables. The log odds of the output result are represented as a linear arrangement of the predictor variables. To extract the output from R, we can use the summary command for the logit model:

Logit Models need a greater number of scenarios (greater sample size) as they utilize highest likelihood estimation techniques. In few situations, we can evaluate models for dichotomous outcomes where there are just a very few cases using exact regression analysis. It is hard to assess a logit model when the outcome is unusual. This does not depend on the size of the data-set.

k-means

Now, we move to k-means clustering. The k-mean clustering is a popular cluster analysis method in data analysis.  By using rxKmeans() function, we can easily create good visualization for k-means clustering on a certain database.

First of all about the rxKmeans function we should know is the syntax. The rxKmeans function syntax includes formula, data, outFile, numClusters…etc like below. Here we provide the explanation for the core syntax.

• formula - Specifies formulas with variables are provided in the clustering algorithm.
• data - Variables can be search and found in the formula dataset.
• outFile - Cluster IDs can be note and written down in the outFile dataset.
• numClusters - Estimation of the clusters’ number (k)
• Algorithm means additional arguments which helps  to control the k

Decision tree

The other function which is popularly used in data mining and we call it decision tree. The critical tool in data mining helps analytics to analyse large data-sets, whilst by using the decision tree, it is easier to build better visualization. The visualization mainly explores the decision rules for predicting the categorical or continuous outcome. Here we present a bit brief about the syntax we use to grow the tree:

We would like to thank our mentor, Dr Shah Jahan Miah, for providing guidance and support during the activities, which has motivated us to compile our ideas and experience in this blog.

Authors:

Aleksandar Jankulov   s4518571

Komal Bhalla               s4531062

Haoyang Di                 s3812986

Introduction to Predictive Analytics

by Bakhshash Kaur Saini,Mekete Berhanie & Nuwan Ramawickrama,

When you hear the word “Predictive Analytics” if you think about your local fortune teller, we have got it wrong, somewhere somehow. Predictive analytics is a process to identify patterns and relationships which may happen in the future by analysing the existing data. It is not fortune telling but it can become a factor that decides the fortune of your business. With the new trends such as IoT (Internet of things), The big data and one to one marketing the market place has changed drastically. The nice markets such as Uber, Airbnb are booming because of the recent burst of the technological growth.  While technology advances every second and the quick changes of the market atmosphere created a huge value for the knowledge about the future. Eventhough we would like to know the future there is no proven way of looking into the future. That is when Predictive Analytics comes into picture with an ability of predicting the trends and patterns of the market using the existing data.

Predictive analytics is heavily based on statistical application and mathematical modelling. The need of predictive modelling and analysis made the job “Data Scientist” one of the most talked about positions in the 21^st century. The HBR article “Data Scientist: The Sexiest Job of the 21st Century” by  Thomas H. Davenport and D.J. Patil talks about how important this job can be. The article further talks about data scientists becoming key player of the organisations.

A popular writer Bernard Marr in one of his articles talk about what is the skill set one should possess to become successful in data science field. The qualities are as follows.

• Multidisciplinary Its not only Maths Phd’s become successful you can come from various other fields.
• Business savvy Regardless of your higher degree you must understand business
• Analytical. naturally analytical to spot patterns
• Good at visual communications Able to make correct graph in the correct time
• Versed in computer science. Familiar with Hadoop, R,Java, Python, etc. are in high demand.
• Creative. Creative enough to find answers to questions with the existing data
• Able to add significant value to data
• A storyteller. Be able to create a story that make value to the organisation

Bearing these qualities in mind anyone who is interested in finding their future in Predictive analytics must develop the above qualities. To get a start for the process, this blog contains a brief tutorial on R basics. This R basics section talks about a case study on social health determinants and finding value from the existing data.

A bit about the Case study

SDOH: SOCIAL DETERMINANTS OF HEALTH

According to WHO(World health organization), people die young or has poor health based on where they live and what they do (WHO 2015).WHO identified nine social determinants of health that affect people with low socioeconomic status that have less accesses to SDOH (WHO 2015). The nine social determinants of health are 

• Social gradients, mortality rate is higher for communities with poor socioeconomic status.

• Stress at work or in life general.

• Early childhood development such as conditions alcohol or drug use during pregnancy.

• Social exclusion such as racism.

• Unemployment

• Social support networks

• Addiction

• Availability of healthy food.

• Availability of transportation

SDOH DATA FROM ADELAIDE UNIVERSITY 

Socioeconomic data was extracted and transformed for predictive analytics regarding social health. The data can be acquired from Adelaide university at  www.publichealth.gov.au (Adelaide 2015).

The data has demography and socioeconomic status of Australian population across the states based on LGA local area governments. The data values are percentage out of 100 populations therefore the values are scaled for statistical analysis.

R Basics

R is a versatile statistical computational package commonly used in the predictive analytics environment. In this section we will have a look at few basic functionality which can get you stated in R.

Uploading data into R

There are few ways to do the data uploading. The most common and easiest way is to use the “read.csv” command. This is how it can be used

Convert your data file into a “.csv” file.

Enter the following in your R console. To start with create an object with any name (eg: data1) and followed by the file uploading command.

data1<-read.csv(file.choose(), header=T)

“file.choose” command is one of the easy ways to browse the required file without nominating actual file path. This will open a new window for you to find the needed file.

Exploration

To start exploring we can generate a summary of the data set.

summary(data1)

A similar command which will show you the first 6 entries of ach variable is “head()” .

head(data1)

To find out what type of data your variable has you can run the following command

class(data1)

class(data1$ your variable name)

  Basic Visualisations with R

Scatter plot

To make a scatter plot you can use the plot command in the following manner. 

plot(

data1$Obese,data1$Smokers, x and y variables you are looking at

xlab="Obese People %",   X axis label

ylab="Smokers %",   Y axis label

main="Obesity vs Smoking"  The main title of the plot which will appear on the top.

)

To show the mean in your scatter plot

mean.ob<-mean(Obese)  calculates the mean of the required variable

plot(Obese~Full.time.Education.at.age.16) creates a scatter plot

abline(h=mean.ob) draws the line

Linear regression

model1<-lm(Obese~Full.time.Education.at.age.16) Creates a linear model

model1 prints the model

abline(model1,col="red") shows the regression

Multiple Linear regression

model2<-lm(Obese~Full.time.Education.at.age.16+Unemployed)

3D Scatter plots

The following commands can be used to create 3D scatter plots

install.packages("scatterplot3d") Installing required packages

require(scatterplot3d) calling the libraries

scatterplot3d(data1[3:5]) creating the 3d scatter plot

Interactive 3D scatter plot

library("rgl") Call the libraries or install them if you don’t have them already

library("RColorBrewer")

plot3d(data1$Obese,data1$Smokers,data1$Unemployed,xlab="Obesity",ylab="Smokers",col=brewer.pal(3,"Dark2"),size=8)

Basic Predictions

You can use do basic predictions using the linear model that you have created earlier.

predict (model1) r will predict the outcomes using the linear model