Mastering Data Mining with Python - Find patterns hidden in your data
Megan Squire
Table of Contents
Preface vii
Chapter 1: Expanding Your Data Mining Toolbox 1
What is data mining? 2
How do we do data mining? 4
The Fayyad et al. KDD process 5
The Han et al. KDD process 5
The CRISP-DM process 6
The Six Steps process 7
Which data mining methodology is the best? 8
What are the techniques used in data mining? 9
What techniques are we going to use in this book? 11
How do we set up our data mining work environment? 11
Summary 18
Chapter 2: Association Rule Mining 19
What are frequent itemsets? 20
The diapers and beer urban legend 20
Frequent itemset mining basics 21
Towards association rules 23
Support 23
Confdence 24
Association rules 24
An example with data 25
Added value – fxing a flaw in the plan 27
Methods for fnding frequent itemsets 28
A project – discovering association rules in software project tags 30
Summary 46
Chapter 3: Entity Matching 47
What is entity matching? 48
Merging data 51
Merging datasets vertically 51
Merging datasets horizontally 53
Techniques for matching 54
Attribute-based similarity matching 54
Be careful of pairwise comparisons 54
Leverage rare values 55
Methods for matching attributes 55
Range-based or distance from target 55
String edit distance 55
Hamming distance 56
Levenshtein distance 56
Soundex 57
Leveraging disjoint sets 58
Context-based similarity matching 58
Machine learning-based entity matching 59
Evaluation of entity matching techniques 60
Effciency – how long does it take to do the matching? 60
Effectiveness – how accurate are the matches that we generate? 61
Usefulness – how practical is the matching procedure to use? 63
Entity matching project 64
Diffculties with matching software projects 65
Two examples 65
Matching on project names 67
Matching on people names 67
Matching on URLs 67
Matching on topics and description keywords 68
The dataset 69
The code 70
The results 75
How many entity matches did we fnd? 76
How good are the pairs we found? 77
Summary 80
Chapter 4: Network Analysis 81
What is a network? 82
Measuring a network 85
Degree of a network 85
Diameter of a network 86
Walks, paths, and trails in a network 88
Components of a network 88
Centrality of a network 89
Closeness centrality 89
Degree centrality 90
Betweenness centrality 91
Other measures of centrality 92
Representing graph data 93
Adjacency matrix 93
Edge lists and adjacency lists 95
Differences between graph data structures 95
Importing data into a graph structure 96
Adjacency list format 97
Edge list format 97
GEXF and GraphML 98
GDF 99
Python pickle 100
JSON 100
JSON node and link series 100
JSON trees 101
Pajek format 102
A real project 103
Exploring the data 104
Generating the network fles 111
Understanding our data as a network 112
Generating simple network metrics 113
Playing with the parameters of a network 116
Analyzing subgraphs 118
Analyzing cliques and centrality in the subgraphs 121
Looking for change over time 124
Summary 134
Chapter 5: Sentiment Analysis in Text 135
What is sentiment analysis? 136
The basics of sentiment analysis 137
The structure of an opinion 137
Document-level and sentence-level analysis 139
Important features of opinions 140
Sentiment analysis algorithms 141
General-purpose data collections 142
Hu and Liu's sentiment analysis lexicon 142
SentiWordNet 143
Vader sentiment 143
Sentiment mining application 144
Motivating the project 145
Data preparation 145
Data analysis of chat messages 149
Data analysis of e-mail messages 154
Summary 160
Chapter 6: Named Entity Recognition in Text 161
Why look for named entities? 162
Techniques for named entity recognition 164
Tagging parts of speech 166
Classes of named entities 167
Building and evaluating NER systems 168
NER and partial matches 168
Handling partial matches 169
Named entity recognition project 171
A simple NER tool 172
Apache Board meeting minutes 173
Django IRC chat 175
GnuIRC summaries 179
LKML e-mails 182
Summary 183
Chapter 7: Automatic Text Summarization 185
What is automatic text summarization? 186
Tools for text summarization 187
Naive text summarization using NLTK 187
Text summarization using Gensim 190
Text summarization using Sumy 193
Sumy's Luhn summarizer 194
Sumy's TextRank summarizer 195
Sumy's LSA summarizer 196
Sumy's Edmundson summarizer 197
Summary 199
Chapter 8: Topic Modeling in Text 201
What is topic modeling? 202
Latent Dirichlet Allocation 203
Gensim for topic modeling 204
Understanding Gensim LDA topics 207
Understanding Gensim LDA passes 208
Applying a Gensim LDA model to new documents 210
Serializing Gensim LDA objects 211
Serializing a dictionary 211
Serializing a corpus 212
Serializing a model 213
Gensim LDA for a larger project 213
Summary 216
Chapter 9: Mining for Data Anomalies 217
What are data anomalies? 218
Missing data 218
Locating missing data 218
Zero values 220
Fixing missing data 220
Ignore the problem rows 220
Fix the problem manually 221
Use a fabricated value 222
Use a central measure 223
Use Last Observation Carried Forward 223
Use a similar value 224
Use the most likely value 224
Data errors 224
Truncated felds 225
Data type and character set errors 226
Logic or semantic errors 227
Outliers 228
Visual mining for outliers 230
Statistical detection of outliers 231
Summary 238
Index 23
- Log in to post comments