PrefaceTaken literally, the title “All of Statistics” is an exaggeration. But in spirit,the title is apt, as the book does cover a much broader range of topics than atypical introductory book on mathematical statistics.This book is for people who want to learn probability and statistics quickly.It is suitable for graduate or advanced undergraduate students in computerscience, mathematics, statistics, and related disciplines. The book includesmodern topics like nonparametric curve estimation, bootstrapping, and classification,topics that are usually relegated to follow-up courses. The reader ispresumed to know calculus and a little linear algebra. No previous knowledgeof probability and statistics is required.Statistics, data mining, and machine learning are all concerned withcollecting and analyzing data. For some time, statistics research was conductedin statistics departments while data mining and machine learning researchwas conducted in computer science departments. Statisticians thoughtthat computer scientists were reinventing the wheel. Computer scientiststhought that statistical theory didn’t apply to their problems.Things are changing. Statisticians now recognize that computer scientistsare making novel contributions while computer scientists now recognize thegenerality of statistical theory and methodology. Clever data mining algorithmsare more scalable than statisticians ever thought possible. Formal statisticaltheory is more pervasive than computer scientists had realized.Students who analyze data, or who aspire to develop new methods foranalyzing data, should be well grounded in basic probability and mathematicalstatistics. Using fancy tools like neural nets, boosting, and support vectormachines without understanding basic statistics is like doing brain surgerybefore knowing how to use a band-aid.But where can students learn basic probability and statistics quickly? Nowhere.At least, that was my conclusion when my computer science colleagues keptasking me: “Where can I send my students to get a good understanding ofmodern statistics quickly?” The typical mathematical statistics course spendstoo much time on tedious and uninspiring topics (counting methods, two dimensionalintegrals, etc.) at the expense of covering modern concepts (bootstrapping,curve estimation, graphical models, etc.). So I set out to redesignour undergraduate honors course on probability and mathematical statistics.This book arose from that course. Here is a summary of the main features ofthis book.1. The book is suitable for graduate students in computer science andhonors undergraduates in math, statistics, and computer science. It isalso useful for students beginning graduate work in statistics who needto fill in their background on mathematical statistics.2. I cover advanced topics that are traditionally not taught in a first course.For example, nonparametric regression, bootstrapping, density estimation,and graphical models.3. I have omitted topics in probability that do not play a central role instatistical inference. For example, counting methods are virtually absent.4. Whenever possible, I avoid tedious calculations in favor of emphasizingconcepts.5. I cover nonparametric inference before parametric inference.6. I abandon the usual “First Term = Probability” and “Second Term= Statistics” approach. Some students only take the first half and itwould be a crime if they did not see any statistical theory. Furthermore,probability is more engaging when students can see it put to work in thecontext of statistics. An exception is the topic of stochastic processeswhich is included in the later material.7. The course moves very quickly and covers much material. My colleaguesjoke that I cover all of statistics in this course and hence the title. Thecourse is demanding but I have worked hard to make the material asintuitive as possible so that the material is very understandable despitethe fast pace.8. Rigor and clarity are not synonymous. I have tried to strike a goodbalance. To avoid getting bogged down in uninteresting technical details,many results are stated without proof. The bibliographic references atthe end of each chapter point the student to appropriate sources.9. On my website are files with R code which students can use for doingall the computing. The website is:http://www.stat.cmu.edu/~larry/all-of-statisticsHowever, the book is not tied to R and any computing language can beused.Part I of the text is concerned with probability theory, the formal languageof uncertainty which is the basis of statistical inference. The basic problemthat we study in probability is:Given a data generating process, what are the properties of the outcomes?Part II is about statistical inference and its close cousins, data mining andmachine learning. The basic problem of statistical inference is the inverse ofprobability:Given the outcomes, what can we say about the process that generatedthe data?These ideas are illustrated in Figure 1. Prediction, classification, clustering,and estimation are all special cases of statistical inference. Data analysis,machine learning and data mining are various names given to the practice ofstatistical inference, depending on the context.Part III applies the ideas from Part II to specific problems such as regression,graphical models, causation, density estimation, smoothing, classification,and simulation. Part III contains one more chapter on probability thatcovers stochastic processes including Markov chains.