What Is Data Mining and

What Is Data Mining and
What Are Its Uses?

If you're looking for great relationships within your customer base,
this is the place to begin.

BY MARK L. LABOVITZ

Data mining is a capability consisting of the hardware, software, "warmware" (skilled labor) and data to support the recognition of previously unknown but potentially useful relationships. It supports the transformation of data to information, knowledge and wisdom, a cycle that should take place in every organization. Companies are now using this capability to understand more about their customers, to design targeted sales and marketing campaigns, to predict what and how frequently customers will buy products, and to spot trends in customer preferences that lead to new product development. It is important to note that data mining is not software alone, as some vendors would have clients believe. While software may play an important role, it is only in the context of clear objectives and careful thinking from business management along with the skill of the analyst that data mining ultimately leads to a success story rather than an embarrassing and costly failure.

Important Data-Mining Concepts

Any data-mining capability starts and ends with the business objectives and a design of the learning environment. A learning environment is a specific plan of discovery, which includes an operational or analysis-action plan to answer questions raised by pursuit of the business objectives. A partial list of questions commonly found in learning plans includes:

• 
  Who are my customers and what do they look like? (profiling)
  
• 
  Who are my most valuable customers? (determining customers' valuations, relationship discovery)
  
• 
  What products or services are my most valuable customers likely to buy next? (likelihood)
  
• 
  How should I segment my marketplace, or does my existing segmentation conform to my sales results? (market segmentation, relationship discovery or likelihood)
  
• 
  When should I contact a customer about a promotion or discount? How often and via what channel? Which products and website links should I include in e-mails to different customers? (optimizing customer contacts, explanatory behavior)
  
• 
  Given a decision to perform a specific promotion or campaign, to whom should it be promoted? (scoring or ranking individuals in terms of their probability of behaving in a variety of predictable ways, likelihood)
  

The learning plan needs to be periodically updated as the environment changes and new insights are developed and deployed.

How Does Data Mining Work?

Data mining is a capability within the business intelligence system (BIS) of an organization's information system architecture. The purpose of the BIS is to support decision making through the analysis of data consolidated together from sources, which are either outside the organization or from internal transaction processing systems (TPS). The central data store of a BIS is the data warehouse or data marts. The source for the data is usually a data warehouse or series of data marts.

However, knowledge workers and business analysts are interested in exploitable patterns or relationships in the data and not the specific values of any given set of data. The analyst uses the data to develop a generalization of the relationship (known as a model), which partially explains the data values being observed.

To achieve this, data-mining packages commonly found in the marketplace use methods from one or more of three approaches to analyze data. These approaches are:

1. 
   Statistical analysis — based on theories of distribution of errors or related probability laws.
   
2. 
   Data/knowledge discovery (KD) — includes analysis and algorithms from pattern recognition, neural networks and machine learning.
   
3. 
   Specialized applications — includes Analysis of Variance/Design of Experiments (ANOVA/DOE); the "metadata" of statistics; Geographic Information Systems (GIS) and spatial analysis; and the analysis of unstructured data such as free text, graphical /visual and audio data. Finally, visualization is used with other techniques or stand-alone.
   

KD and statistically based methods have not developed in complete isolation from one another. Many approaches that were originally statistically based have been augmented by KD methods and made more efficient and operationally feasible for use in business processes.

Evaluation Criteria

There are numerous data-mining software products available to organizations that must be evaluated to fit their needs. The following is a partial set of evaluation criteria to consider when purchasing data-mining software applications:

1. 
   User interface: How is the user interface set up? Does the interface support generic data-mining analysis or does it adapt to a specific business's needs?
   
2. 
   End user functionality and access: How easy is it for users to find and use previously created models/analyses? What support is provided for distribution of results?

   Can the tool be used to access and create models via the Web?

   
   
3. 
   Advanced analytical power: What built-in support does the tool set provide for complex analysis? Does the tool set support analyses from all the business discovery classes? Are both statistically based and KD algorithms provided?
   
4. 
   Data transfer/data store interface and file scoring: Does the application perform file scoring and does it support selection and transfer of data between the analysis engine and the data store? How does the tool ensure that the data sources, models, reports and metadata are all synchronized?
   
5. 
   Sampling: What sort of sampling design schemas does the tool support and help implement?
   
6. 
   Data preprocessing and preparation: What, if any, data preprocessing does the tool set support? Does it support variable transformation, recoding and creation?
   
7. 
   Goodness of fit and validation: Does the application provide comparison of models, modeling techniques and analyses that aid in the selection of attributes most likely to solve a given business problem? What validation methodologies does the tool support?
   
8. 
   Customization: What support is available to customize and develop applications? What can the tool do to support the implementation of models into the production environment?
   
9. 
   Availability of trained professionals: How large and how experienced is the labor pool for professionals trained in the particular tool? Does the vendor have a certification program and how is that organized?
   
10. 
    Licensing and pricing: What does the package cost, what is the total cost of ownership and how is the system licensed—by server, by processors, by named seats, etc.
    

Data Mining Issues

There are several significant issues in implementing, using and maintaining a data-mining capability that organizations need to address.

• 
  Understand What You Are Getting. Buying a data-mining software engine is insufficient to realize an ongoing data-mining capability. The software engine is a tool kit. As such, prerequisite business plans, the internal IT infrastructure and the craftsmen capable of using the tools must be secured and put into place.
  
• 
  Data Readiness for Analysis. Data-mining requires a consolidated "de-duplicated" and cleaned data store to draw from. Seventy to 85 percent of the work in building models using data mining relates to the cleaning and preparation of data prior to a specific analysis.
  
• 
  Staffing. Data-mining software is not a replacement for skilled analysts. The algorithms used in data mining are quite powerful. If the users do not understand these issues or have access to knowledgeable staff, the results can be disastrous. The staff also needs to be committed to the effort. While the level of labor required to run a data-mining capability will vary with a number of factors, data mining is not a do-once-and-then-forget activity. Learnings are incremental and environments (internal and external) change; therefore learning is a constant and ongoing process and so is data mining. The ongoing use of a data-mining capability requires business analysis and data analysis skills.
  
• 
  Taking Action. Data mining is about acting upon the results and making changes. An important issue which must be addressed before undertaking the creation of a data mining capability is the organization's ability to take the learnings, interpret the results and make the recommended changes in the business and organization. The best use of data mining will challenge an organization's change management process.
  
• 
  Financial Considerations. There is a considerable danger of underestimating the costs of implementing, using and maintaining a data-mining capability. As often occurs with other IT-enabled activities, the costs of integrating the data-mining engine and making it work in a specific environment are often underestimated by 50 percent or more. A frequently overlooked source of cost is the labor required to use and maintain the system. Data-mining software vendors have the tendency to make recommendations calling for fewer analysts of lower skill levels than will be actually required by the data-mining capability to meet company expectations. Finally, with respect to maintenance, the organization is not only faced with the maintenance activities typically associated with an IT system, but it also must expend resources to maintain other items such as models and data.
  

Data mining is a data-driven capability for extracting patterns that can provide powerful answers to critical business questions. For companies willing to make the commitment to financial and technical resources, highly qualified analytical talent, the cleanup and maintenance of clean data, and the willingness to learn and act upon the results, data mining can provide a powerful competitive advantage.