top of page
Search
Writer's pictureRosmina Joy Cabauatan

Crafting a Data Mining Problem Statement

Updated: Mar 31, 2021

Crafting a problem statement is the initial step towards identifying the right data mining technique to use for model generation. It begins with the question, How can data solve a problem?. A problem can be solved using data by going through the following steps:

  1. Know the practical motivation of solving the problem using data

  2. Identify the data to solve the problem

  3. Formulate the data mining problem

  4. Prepare the data

  5. Explore the data

  6. Create visualizations for pattern recognition

  7. Apply inferential analysis

  8. Create a statistical inference

  9. Know the ethical considerations


Step 1. Know the practical motivation of solving the problem using data


A good problem statement can be solved using data if its answers the following questions:

  1. Is the problem related to data?

  2. Can you solve the problem using the data?


Stage 2. Identify the data to solve the problem


In collecting data, you must answer the following questions.

  1. Does the data you are trying to collect match the problem?

  2. Does the data represent reality?

  3. How to collect the relevant data?

For example,

if a poll is used to survey people's opinion about COVID 19 massive swab testing, how should this be conducted such that it can reflect reality?


It's always good to review a lesson in statistics about getting a sample of a population. The figure below shows the commonly used sampling methods.



In simple random sampling, every member of the population has an equal chance of being selected. Stratified sampling is appropriate when the population has mixed characteristics and every characteristic is proportionally represented in the sample. In systematic sampling, every member of the population is listed with a number, but instead of randomly generating numbers, individuals are chosen at regular intervals. Cluster sampling also involves dividing the population into subgroups, but each subgroup should have similar characteristics to the whole sample. Instead of sampling individuals from each subgroup, the entire subgroup is randomly selected.


Stage 3. Formulate the data mining problem


A data mining problem statement must reflect the data that will be used to solve the problem. For example,


Can age, blood type, location, and travel history predict if someone would most likely be infected with Covid19?


What about,


How do I know whether someone has infected with Covid19?


Is the statement correctly crafted? How will you restructure to make it correct?



Stage 4. Prepare the data


Preparing the data for analysis is an essential step to knowledge discovery. Data has quality if it satisfies the requirements of the intended use. The following are the factors comprising data quality.

  1. accuracy

  2. completeness

  3. consistency

  4. timeliness

  5. believability

  6. interpretability

Read the article Data Preprocessing: An Essential Step to Knowledge Discovery to know the challenges and major tasks in preparing data for analysis.


Stage 5. Explore the data


In this stage, some statistical properties of the data such as mean, median, variance, and distribution are studied. You can also create a graphical presentation that can help you gain insight from the data. An example of a question to ask can be like,


What initial insights can you extract from the distribution of Covid19 cases in the Philippines by age group?


Stage 6. Visualization and Pattern Recognition


Exploratory analysis is commonly referred to as descriptive analytics. In this type of analytics, statistical characteristics of data are presented in the form of visualization. This is the presentation of information using graphical representations to facilitate comparison of data and recognize patterns for initial decision making.


Visualization can be classified as explore or calculate and communicate. Explore and calculate requires a further analysis which requires a reason about a conveyed information. Communicate is a type that explains the information and suggests a hint for decision making. Visualization can also be classified according to purposes, such as comparison, composition, distribution, and relationship. The following questions can serve as a guide in determining which among the types is best suited for specific data.

  • How many variables are needed in the chart?

  • How many data points will you display for each variable?


Stage 7. Inferential Analysis


Inferential statistical analysis infers the properties of a population by testing hypotheses and deriving estimates. The findings can be taken from the sample group and be generalized to a larger population.


Stage 8. Statistical Inference


Statistical inference can become certain by generalizing the learning procedure or model and estimate the confidence of prediction by the general model.

For example, a claim that COVID 19 cases will go down by exactly 10% in the coming month is certainly not dependable. However, if the claim that COVID 19 cases will decrease in a range of 3%-5% in the following month with the confidence of 95%. In statistics, this is referred to as confidence interval.


Stage 9. Ethical Considerations


It is also essential to keep in mind to safeguard data that needs utmost confidentiality. Hence, in analyzing data, it is necessary to consider the privacy of individuals or organizations.

112 views0 comments

Комментарии


bottom of page