EDA 101 for Beginners
What is EDA?
EDA stands for Exploratory Data Analysis. In Statistics, exploratory data analysis is an approach of analysing data sets to summarize their main characteristics, often using statistical graphics and other data visualization methods.
Need of EDA
There are numerous applications of EDA. But the basic and by far the most important need of EDA is to analyse, visualise and clean data and build models out of it.
Intuition
Imagine your wolf pack decides to watch a movie you haven’t heard of. There is absolutely no debate about that, it will lead to a state where you find yourself puzzled with lot of questions which needs to be answered in order to make a decision. Being a good chieftain the first question you would ask, what is the cast and crew of the movie?As a regular practice, you would also watch the trailer of the movie on YouTube. Furthermore, you’d find out ratings and reviews the movie has received from the audience.
Whatever investigating measures you would take before finally buying popcorn for your clan in theatre, is nothing but what data scientists in their lingo call ‘Exploratory Data Analysis’.
Lets perform EDA
Step 1 : Import Libraries
Step 2: Read data from a csv file
pandas is used to read .csv files.
Lets understand EDA with a dataset.
pd.read_csv() method is used to read from a csv file and convert it into a pandas DataFrame.
Step 3: Understanding data
Observing the shape and datatypes helps better understanding of the Data.
It can be observed that there are some Numeric data and some Strings / Objects.
We have to convert Objects to Integers wherever possible for our models' better performance.
Step 4 : Create a backup of the data
It is a good practice to have a backup of the data.
What is done here is storing data from 2nd Column to data2 because 1st column of the original data was of no use.
We will talk about data cleaning later in the article.
Now, lets create a backup of the data.
.copy() method is used here. This will create a separate dataset and copy the data from original dataset.
Now, if we make changes in the new dataset , it will not be reflected in the original dataset. Hence, that can be kept for backup.
Step 5 : Cleaning Data | Converting to Integer datatypes
.to_numeric() method is used to convert objects to integers.
Weather column can be observed to have 3 types of values -'S' , 'C' , 'PS' . These can be converted into categories.
.astype() function is used here to convert objects to categorical datatype.
This data can be further cleaned by creating dummy variables for Weather column.
Step 6 : Cleaning Data | Detecting and Deleting duplicates , Renaming columns with special symbols
Step 7 : Data Visualisation
Data Visualisation is the key technique to detect Outliers , Missing Values, correlation and Trend or Distribution of data.
Data Visualisation is done using matplotlib.pyplot and seaborn libraries.
Histogram : To observe distribution
Heads up: Following plots are plotted with some different dataset.
Boxplot : To observe outliers
Heatmap : To observe correlation among variables
Step 8 : Normalising and/or Standardising dataset
Standardization
The result of standardization (or Z-score normalization) is that the features will be rescaled to ensure the mean and the standard deviation are 0 and 1, respectively.
This technique to rescale features value with the distribution value between 0 and 1 is useful for the optimization algorithms, such as gradient descent, that are used within machine-learning algorithms that weight inputs (e.g., regression and neural networks). Rescaling is also used for algorithms that use distance measurements, for example, K-nearest-neighbours (KNN).
Max/Min Normalization
Another common approach is the so-called max/min normalization (min/max scaling). This technique is to rescale features with a distribution value between 0 and 1.
For every feature, the minimum value of that feature gets transformed into 0 and the maximum value gets transformed into 1.
Comments
Post a Comment