[{"Section": "0", "Name": "Data 101 Textbook by Tomasz Imielinski", "Details": "\r\n\r\n\r\n \r\n \r\n Data 101 Textbook \u2014 Abstract\r\n \r\n\r\n\r\n\r\n\r\n

\r\n

DATA 101 TEXTBOOK

\r\n

ABSTRACT

\r\n\r\n

\r\n The goal of this textbook is to support students in the Data 101 course. It consists of more than fifteen chapters covering basic data visualization, data exploration, and the core ideas of statistics\u2014including hypothesis testing, Bayesian reasoning, prediction models, and association rules. A major theme throughout the course is learning how not to be fooled by data. Students study randomness, sampling variability, and the dangers of misinterpreting aggregated information, such as Simpson\u2019s paradox and the ecological fallacy. Multiple hypothesis testing\u2014an essential but often overlooked topic that helps prevent false discoveries in large datasets\u2014is also covered in depth.\r\n

\r\n\r\n

\r\n Each chapter begins with PowerPoint slides, followed by a short written summary and a ten-question multiple-choice quiz with answers. In several chapters, we include a convo link to a GPT-powered chat that demonstrates how the concepts apply to a real dataset. Some chapters also feature an interactive tour, a guided, GPT-driven walkthrough of data that reinforces conceptual understanding.\r\n

\r\n\r\n

\r\n Throughout the course, we use R101, a small and beginner-friendly subset of R that allows students to focus on reasoning rather than syntax. R101 is taught in recitations, while lectures follow the Data 101 textbook and emphasize concepts and their applications. With quizzes, mini-projects, and prediction competitions, Data 101 provides an engaging and rigorous foundation for learning to think clearly with data.\r\n

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "1", "Name": "Data 101 Intro", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "2", "Name": "Language of Data", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "2.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n The Language of Data\r\n \r\n\r\n\r\n\r\n

The Language of Data

\r\n\r\n

\r\n In Data 101, we don\u2019t just learn tools; we learn a new language. Modern\r\n vibe analytics is all about asking questions in\r\n natural language — \u201cWhich students are struggling?\u201d \u201cWhich products are\r\n trending?\u201d — and translating those questions into precise operations on\r\n data. To do that, we need a shared vocabulary: the \r\n language of data, the language of statistics,\r\n and the language of data operations. This language\r\n is declarative, not procedural. We say what we want, not\r\n exactly how to do it step by step. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Columns, Values, and Data Types

\r\n\r\n

\r\n Every dataset starts with columns and\r\n values. Columns are also called variables,\r\n features, attributes, or fields. They describe what is being measured:\r\n Major, Score, City, Price.\r\n The entries inside those columns are values, sometimes called cells, data points,\r\n or observations when we look across a whole row.\r\n

\r\n\r\n

\r\n Different columns store different data types:\r\n

\r\n

Numeric: integers and doubles (e.g., 42, 3.14).
Character or string: text like \u201cNew York\u201d.
Factor/categorical: a small set of levels, like grades A\u2013F.
Logical/Boolean: TRUE/FALSE, 0/1.
Date/Time: calendar dates or timestamps.
NA/NULL: missing or unknown values.

\r\n\r\n

\r\n Even strings can hide complexity: punctuation, currency symbols, math symbols,\r\n escape characters like \\n, emojis, and typos all influence how\r\n data is cleaned and analyzed.\r\n

\r\n\r\n

Rows, Data Frames, and Metadata

\r\n\r\n

\r\n Each row of a dataset represents a unit of\r\n observation: a student, a transaction, a movie, a patient. In databases we might\r\n call it a record or a tuple; in machine learning, an instance.\r\n

\r\n\r\n

\r\n Put rows and columns together and you have a \r\n data frame: a rectangular table with labeled columns. A whole collection\r\n of such tables, often linked by keys, becomes a larger dataset or database.\r\n

\r\n\r\n

\r\n Around every dataset sits metadata — data\r\n about the data: the schema (column names and types), cardinality (how many unique\r\n values a column has), distribution, missingness, granularity (daily vs. hourly),\r\n sparsity (how many zeros or blanks), and dimensionality (how many features).\r\n

\r\n\r\n

Basic Data Operations

\r\n\r\n

\r\n Once we can describe data, we need verbs to act on it. The first set of verbs is\r\n about viewing and exploring:\r\n

\r\n

Show the first few rows.
Display the structure and types of each column.
Summarize statistics: min, max, mean, median.
Check for missing values and basic memory usage.

\r\n\r\n

\r\n Next comes subsetting and manipulating:\r\n

\r\n

Select rows that match a condition (e.g., Score > 80).
Choose specific columns to keep.
Filter rows and select columns at the same time.
Combine two datasets by a common key (merge / join).
Stack one dataset below another (add rows).
Add new columns or rename existing ones.
Remove rows with missing data.
Sort the data by one or more columns.

\r\n\r\n

\r\n Finally, we have aggregating and summarizing:\r\n

\r\n

Count how often each value appears (frequency tables).
Group by a column and compute summaries (mean score by major).
Compute mean, sum, median, min, max, standard deviation, variance.
Count the number of entries per group.

\r\n\r\n

\r\n All of this is powered by Boolean conditions:\r\n expressions like Major == \"Math\" or\r\n Score >= 70 & Attendance > 0.8 that define which rows\r\n belong to a subset. Subsetting via Boolean conditions is at the heart of\r\n exploratory queries.\r\n

\r\n\r\n

The Language of Statistics

\r\n\r\n

\r\n On top of the language of data sits the \r\n language of statistics. Once we can phrase questions like\r\n \u201cWhat is the average score by major?\u201d we move on to concepts like\r\n distributions, variability, confidence intervals, hypothesis tests,\r\n p-values, and effect sizes. The slides hint that this statistical\r\n vocabulary will layer on top of the structural words we are learning now.\r\n

\r\n\r\n

Data Pollution: When the Language Gets Noisy

\r\n\r\n

\r\n Real-world data rarely speaks clearly. It suffers from\r\n data pollution, making our tidy language\r\n messy in practice.\r\n

\r\n\r\n

\r\n Common sources of pollution include:\r\n

\r\n

Manual entry errors: typos, swapped digits, inconsistent spelling.
OCR and text recognition mistakes.
Data integration issues across different systems.
File encoding or formatting problems.
Inconsistent standards or units (e.g., meters vs. feet).

\r\n\r\n

\r\n The result is a \u201cpolluted\u201d data frame: strange characters in text fields,\r\n wrong or impossible values, inconsistent categories (\u201cNY\u201d, \u201cNew York\u201d,\r\n \u201cNewYork\u201d), missing values in critical columns. Before we do any modeling,\r\n we must recognize and clean this pollution, restoring a consistent language\r\n so our tools understand what we mean.\r\n

\r\n\r\n

Putting It All Together

\r\n\r\n

\r\n The goal of this module is simple: when you look at a table, you should be\r\n able to talk about it fluently. You should know the names for its parts\r\n (columns, rows, data types), understand how to describe its structure\r\n (schema, granularity, sparsity), and be comfortable with the basic verbs\r\n of data operations (view, subset, join, summarize).\r\n

\r\n\r\n

\r\n Once this language feels natural, \u201cvibe coding\u201d becomes possible: you can\r\n start in natural language — \u201cShow me average score by major, but only\r\n for students who attended more than 80% of classes\u201d — and translate\r\n that request into precise queries and code. This is the foundation on which\r\n all later topics in Data 101 are built.\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "2.2", "Name": "Quiz", "Details": "

\r\n Which of the following is NOT listed as an alternative name for a column?\r\n

Feature
Attribute
Instance
Field

\r\n

Correct answer: C

\r\n

\r\n\r\n

\r\n \u201cValues\u201d in a dataset (what is inside each cell) can also be called:\r\n

Entries or cells
Schemas
Granularities
Dimensions

\r\n

Correct answer: A

\r\n

\r\n\r\n

\r\n Which of the following is a data type mentioned in the slides?\r\n

Picture
Factor (categorical with levels)
Array
Matrix

\r\n

Correct answer: B

\r\n

\r\n\r\n

\r\n A row in a data frame can also be called:\r\n

Schema
Instance or record
Feature
Level

\r\n

Correct answer: B

\r\n

\r\n\r\n

\r\n Which metadata term describes how detailed the data are (for example: daily vs hourly readings)?\r\n

Cardinality
Dimensionality
Granularity
Sparsity

\r\n

Correct answer: C

\r\n

\r\n\r\n

\r\n You want to count how many times each unique value appears in a column (for example, how many students per major). \r\n This is primarily an example of:\r\n

Subsetting & manipulating data
Viewing & exploring data
Aggregating & summarizing data
Data pollution

\r\n

Correct answer: C

\r\n

\r\n\r\n

\r\n \u201cCombine two datasets by common columns or keys\u201d (for example, join student grades with student demographics) is an example of:\r\n

Stacking rows (adding one dataset below another)
Renaming columns
Subsetting via Boolean conditions
Merging or joining datasets

\r\n

Correct answer: D

\r\n

\r\n\r\n

\r\n Which of the following is listed as a cause of data pollution in the slides?\r\n

Too many rows in the dataset
Manual entry errors and OCR/text recognition issues
Using R instead of Python
Having both numeric and character columns

\r\n

Correct answer: B

\r\n

\r\n\r\n

\r\n \u201cSparsity\u201d in the metadata section refers to:\r\n

The number of unique values in a column
The proportion of empty, missing, or zero values
The total memory used by the dataset
The number of columns in a data frame

\r\n

Correct answer: B

\r\n

\r", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "3", "Name": "Data Exploration", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "3.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Exploratory Queries on the Moody Dataset\r\n \r\n\r\n\r\n\r\n

Exploratory Queries: What the Moody Data Really Says

\r\n\r\n

\r\n Exploratory data analysis is about asking simple but sharp questions of a dataset\r\n and letting the data answer before we build any fancy models. In the Moody dataset,\r\n each row is a student, and our main outcome is the final grade.\r\n With a bit of counting, visualization, and curiosity, we can uncover surprising\r\n stories about difficulty, effort, and fairness in Moody\u2019s class. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Is Moody\u2019s Class an Easy A?

\r\n\r\n

\r\n The first question students care about is simple: How hard is this class?\r\n Looking at the overall grade distribution, it becomes clear that Moody\u2019s course is\r\n not an easy A at all. The most common grade is a C,\r\n followed by a very large number of F\u2019s.\r\n In fact, about one third of the class \u2013 34.4% \u2013 failed with an F.\r\n A\u2019s exist, but they are relatively rare; D\u2019s are the rarest, but mostly because\r\n many struggling students seem to fall all the way to F.\r\n

\r\n\r\n

\r\n When we break grades down by major, the picture becomes\r\n more nuanced. Math majors have the highest fraction of A\u2019s, while Economics majors\r\n suffer the highest failure rate. C\u2019s dominate in every major, and D remains a tiny\r\n sliver across the board. The stacked bar charts of grades by major make this instantly\r\n visible: tall C segments in every bar, Math leaning slightly toward A, and Econ with\r\n a worrying block of F\u2019s.\r\n

\r\n\r\n

\r\n Before trusting these patterns, the slides show how to verify them independently:\r\n by using simple R or Python code to create cross-tabs of Major \u00d7 Grade,\r\n turning counts into percentages, or even manually checking in a spreadsheet.\r\n

\r\n\r\n

Score vs. Grade: Overlaps and Oddities

\r\n\r\n

\r\n Next, we zoom in on the relationship between score\r\n and grade. If grading were perfectly consistent,\r\n we would expect that higher scores always mean better grades, and equal scores\r\n always mean equal grades. The data tells a different story.\r\n

\r\n\r\n

\r\n There are ranges where the same score leads to different grades:\r\n

\r\n

86\u201389: some students get A, others B.
71\u201375: some get B, others C.
51\u201358: some get C, others D.

\r\n\r\n

\r\n Interestingly, there is no overlap between D and F: no score appears with both\r\n grades. At the top and bottom of the scale, the boundaries are sharp; in the\r\n middle, things are fuzzy and sometimes inconsistent.\r\n

\r\n\r\n

Do Attendance, Seating, and Questions Matter?

\r\n\r\n

\r\n To explain these overlaps, we look beyond scores. The Moody dataset also records\r\n attendance, row\r\n (seating position), and questions asked.\r\n

\r\n\r\n

\r\n For the A\u2013B overlap (scores 86\u201389), attendance is almost identical for A and B\r\n students, and the average number of questions asked is also very similar. The\r\n big difference is seating: A students at these scores\r\n sit in the very front rows, while B students with the same scores are mostly in\r\n the back. Front-row seating appears to be the decisive \u201ctie-breaker\u201d between\r\n A and B in this band.\r\n

\r\n\r\n

\r\n Between B and C, however, no clear factor stands out. Attendance, row, and questions\r\n are all similar. In this range, grading looks genuinely inconsistent: some hard-working,\r\n engaged students still land a C, while others with similar behavior get a B.\r\n

\r\n\r\n

\r\n Between C and D, and especially between D and F, attendance and participation start\r\n to matter again. Students with slightly higher engagement \u2013 more class sessions\r\n attended and more questions asked \u2013 are much more likely to avoid failing when\r\n their scores hover around the cutoff.\r\n

\r\n\r\n

The Zero-Score Mystery: When Effort Saves the Day

\r\n\r\n

\r\n One of the most striking anomalies involves students with a \r\n score of zero. You might assume that everyone with zero points fails. Not in\r\n Moody\u2019s class. The data shows that some students with score = 0 actually received\r\n a C, while others with the same score got an F.\r\n

\r\n\r\n

\r\n What separates them is not performance \u2013 everyone scored 0 \u2013 but effort:\r\n

\r\n

Passing students (C) had very high attendance (around 0.84\u20130.96) and asked 8\u201310 questions.
Failing students (F) had low attendance (around 0.23\u20130.45) and asked only 3\u20134 questions.

\r\n\r\n

\r\n At score = 0, the grade is entirely driven by engagement: show up, participate,\r\n sit where you are noticed \u2013 and you might pass despite failing every exam.\r\n This is a powerful example of how effort can outweigh raw results in this class.\r\n

\r\n\r\n

How to Get an A, B, or Simply Pass

\r\n\r\n

\r\n From all these patterns, we can extract practical \u201crules of thumb\u201d:\r\n

\r\n\r\n

\r\n To get an A: A score of 90 or higher guarantees an A.\r\n In the 86\u201389 range, front-row seating strongly tilts the odds toward an A rather than a B.\r\n
\r\n To get a B: Scores between 76 and 89 almost always lead to a B\r\n (unless you cross into the A zone). In the 71\u201375 borderline zone, grading is noisy and\r\n not well explained by attendance or questions.\r\n
\r\n To pass: Any score of 51 or more guarantees at least a D.\r\n Below that, high attendance and active participation can still rescue you, even from a\r\n score of zero.\r\n

\r\n\r\n

Luck, Unluck, and Non-Monotonic Grading

\r\n\r\n

\r\n The slides also highlight the \u201cluckiest\u201d and \u201cunluckiest\u201d students, defined as those\r\n on the edges of each grade band. Some B students are just a few points away from an A;\r\n some C students are just a few points away from a B. More dramatically, there are huge\r\n score ranges where D and F are far below C and D respectively, yet the grade C itself\r\n stretches all the way down to zero.\r\n

\r\n\r\n

\r\n The key takeaway is that the grading scheme is not monotonic\r\n with respect to score: two students with the same score can receive different grades.\r\n Hidden variables \u2013 attendance, participation, and seating \u2013 act as \u201cboosters\u201d or penalties.\r\n

\r\n\r\n

From Queries to Visual Stories

\r\n\r\n

\r\n Throughout this exploration, visual tools are essential. Bar plots and stacked bar\r\n charts reveal how grades vary by major. Boxplots capture how score, attendance, and\r\n questions differ across grades. Histograms show score distributions. Scatter plots\r\n and mosaic plots help visualize relationships between numeric and categorical variables.\r\n

\r\n\r\n

\r\n Together, these exploratory queries and visualizations turn a simple gradebook into\r\n a rich story about effort, talent, bias, and the subtle ways that behavior in class\r\n translates into final outcomes.\r\n

\r\n\r\n\r\n\r", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "3.11", "Name": "Quiz", "Details": " MCQ Quiz \u2013 Exploratory Queries and Moody Dataset

MCQ Quiz \u2013 Exploratory Queries and Moody Data

1. What is the main purpose of exploratory queries in the Moody dataset?

A) To build final prediction models
B) To identify basic questions, regularities, outliers, and surprises in the data
C) To prove a fixed hypothesis
D) To clean all missing values

Answer: B

2. According to the overall grade distribution (A: 116, B: 148, C: 371, D: 21, F: 344), which grade is most common?

A) A
B) B
C) C
D) F

Answer: C

3. In the same distribution, which grade is least common?

A) A
B) B
C) C
D) D

Answer: D

4. What fraction of the class received an F (failed) in Moody\u2019s course?

A) 11.6%
B) 21.0%
C) 34.4%
D) 50.0%

Answer: C

5. What did the exploratory queries reveal about grades by major?

A) Econ majors had the highest fraction of A\u2019s; Math the highest F\u2019s
B) Math majors had the highest fraction of A\u2019s; Econ had the highest failure rate
C) CS majors dominated both A\u2019s and F\u2019s
D) All majors had identical grade distributions

Answer: B

6. Which R command (shown in the slides) can be used to verify the counts of A\u2019s and F\u2019s per major?

A) hist(df$Grade)
B) boxplot(Grade ~ Major, data = df)
C) table(df$Major, df$Grade)
D) plot(df$Major, df$Grade)

Answer: C

7. For overlapping score ranges between grades, which pair showed no overlap in identical scores?

A) A vs B
B) B vs C
C) C vs D
D) D vs F

Answer: D

8. In the 86\u201389 score overlap (A vs B), which factor most strongly separated A\u2019s from B\u2019s?

A) Attendance (A\u2019s had much higher attendance)
B) Number of questions asked (A\u2019s asked far more)
C) Seating row (A\u2019s sat in front, B\u2019s much farther back)
D) Major (only Math majors got A\u2019s)

Answer: C

9. For students with Score = 0, what combination of factors was associated with getting a C instead of an F?

A) Low attendance and few questions
B) High attendance and active participation (many questions), often sitting in front
C) Being a CS major only
D) Random chance only

Answer: B

10. Which plot type is best suited (according to the slides) for visualizing a numeric variable by categories, such as Score by Grade?

A) Barplot
B) Pie chart
C) Boxplot
D) Mosaic plot

Answer: C

", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "3.2", "Name": "News article about Professor Moody's class", "Details": "\n\n\n \n \n Moody the Mercurial \u2014 Inside the Strange Way of Grading by Professor Reginald Moody\n \n\n\n

Moody the Mercurial

Inside the Strange Way of Grading by Professor Reginald Moody

By Clara Jennings, Campus Correspondent

NORTHBRIDGE, March 2025 — Professor Reginald Moody has long been one of Northbridge University’s most infamous instructors. Revered for his sharp lectures but loathed for his erratic grading, he inspires equal parts admiration and anger among students. His end-of-term evaluations routinely rank among the lowest in the department, filled with complaints of “unfair,” “arbitrary,” and “he just makes it up as he goes.”

The recently released Moody Class Dataset provides a rare look at how his grading works. And the results are astonishing: two students with the same score often walked away with very different grades.

\n\n

“I Got a C With a Zero”

The most shocking discovery is the case of students who scored zero points but still received a passing C.

“I couldn’t finish the exam,” recalled Emma Carter, an Economics major. “But I was always there, front row, asking questions. He called me his ‘constant chatter.’ Somehow, even with a zero, I scraped a C.”

Her classmate Alex Wong, a Math major, was less lucky:

“I showed up maybe half the time, sat in the back, never asked questions. Same score as Emma, but I got an F. Moody told me, ‘I can’t pass a ghost.’”

The numbers confirm it: Emma’s attendance was 0.96, questions asked = 10; Alex’s attendance was 0.23, questions = 3.

\n\n

Front-Row Advantage

In the 86–89 range, the contradictions are just as puzzling. Some students proudly carried home A’s, while others with identical scores were stuck with B’s.

The deciding factor? Seating position.

Those in rows 1–3 leaned toward A’s; those in rows 8–11 were pushed into B’s.

“I had an 88 and got an A,” said Rachel Stein, a CS major. “I always sat in the second row. He’d pick on me to answer, but I guess it paid off.”

“Same 88, but I sat in the back. Moody shrugged and said, ‘Out of sight, out of grade.’ I thought he was joking,” grumbled Daniel Brooks, Econ major, who got a B.

\n\n

Questions Matter at the Borderline

At the 51–58 range, another strange split appeared. Some students earned C’s, others were stuck with D’s—despite identical scores.

The difference wasn’t attendance or seating this time, but class participation.

C students: 4–10 questions asked
\n D students: 1 or 2 questions at most

“I wasn’t strong in the material,” said Sophia Hernandez, Stats major. “But I kept raising my hand, even when unsure. Moody remembered. I got a C. My roommate, same score, never spoke up. She got a D.”

\n\n

The Confusion Zone: B vs C

The 71–75 overlap remains the most baffling. Attendance, row, and questions were nearly identical.

“I had 72 and got a C,” said Marcus Lee, Econ. “My friend got a B at 72. We sat side by side. The only difference was he raised his hand now and then.”

Here, Moody’s reasoning seems most opaque, fueling resentment.

\n\n

Why Students Are Angry

It’s no wonder evaluations are harsh. Students feel trapped in a grading lottery. High attendance and effort don’t always pay off. High scores sometimes aren’t enough.

Still, the dataset paints a picture of a professor who values visibility and engagement as much as performance:

Front rows tip an A vs B.
Questions tip a C vs D.
Attendance + persistence can rescue a zero into a C.

\n\n

Moody Speaks

“A grade is not a number. It’s a memory of where you sat, how alive you were. Scores are cold. I grade the warmth.”

\n\n

Pull Quotes from Student Evaluations

“Grades are like mood rings in this class. Same score as my friend, but he got an A and I got a B. Guess I should’ve sat closer.” – Anonymous, CS Major

“I had perfect attendance, raised my hand every day, but still got a C. Meanwhile, the guy next to me skipped half the lectures and got a B. What’s the logic?” – Econ Senior

“He told me: ‘You’re too quiet. That’s a D.’ I thought he was joking. He wasn’t.” – Stats Junior

“Moody says, ‘I grade the warmth.’ I say, ‘I paid tuition for cold hard numbers.’” – Math Sophomore

“It’s the Hunger Games. May the odds—and your seat number—be ever in your favor.” – Anonymous Freshman

\n\n \n\n

Bottom Line: In Professor Moody’s class, numbers don’t speak for themselves. Presence matters. And if you’re invisible, you risk vanishing from the gradebook.

\n\n \n

\n\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "3.3", "Name": "Verification of the above article against the data set", "Details": "\n\n\n\n\nMoody the Mercurial \u2014 Verification\n\n\n\n\n

I compared the article (\u201cMoody the Mercurial\u201d) with the actual MoodyTrain2024 dataset, and here is the verification:

1. \u201cI Got a C With a Zero\u201d

Dataset evidence:

Student with Score = 0, Grade = C, Attendance = 0.96, Questions = 8, Row = 6, Major = Econ.

Another student with Score = 0, Grade = F, Attendance = 0.23, Questions = 3, Row = 2, Major = Math.

\u2705 Article is correct: some zero scorers got C\u2019s, others got F\u2019s, and the difference lines up with attendance and participation.

2. Front-Row Advantage (Scores 86\u201389 \u2192 A vs B)

Dataset evidence:

Score = 86, Row = 1, Grade = A.

Score = 88, Row = 3, Grade = A.

Score = 88, Row = 11, Grade = B.

\u2705 Article is accurate: in the same score range (86\u201389), students in front rows (1\u20133) lean A, while back rows (8\u201311) lean B.

3. Questions Matter (Scores 51\u201358 \u2192 C vs D)

Dataset evidence:

Student with Score = 51, Questions = 4, Grade = C.

Student with Score = 54, Questions = 5, Grade = D.

Students with 4\u20138 questions tend toward C\u2019s, while very few-question askers slip into D.

\u2705 Article is broadly correct, though the dataset shows some exceptions (e.g., a D with 5 questions). Still, the pattern holds: more questions = higher chance of C.

4. Confusion Zone (Scores 71\u201375 \u2192 B vs C)

Dataset evidence:

Score = 72, Row = 10, Questions = 2 \u2192 Grade C.

Score = 72, Row = 2, Questions = 1 \u2192 Grade B.

Both have very similar attendance and score, yet got different grades.

\u2705 Article is confirmed: there is overlap, and the difference isn\u2019t explained neatly by attendance, seating, or questions.

\u2705 Verdict

The article matches the dataset closely:

Zero scorers \u2192 C or F depending on attendance/questions.

86\u201389 \u2192 A in front rows, B in back.

51\u201358 \u2192 Questions often tip C vs D.

71\u201375 \u2192 true \u201cconfusion zone,\u201d with same-score students getting different grades.

The numbers quoted in the article (like 0.96 attendance, 10 questions vs 0.23 attendance, 3 questions) are directly supported by the data.

\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "4", "Name": "Fooled by Randomness", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "4.1", "Name": " False Trends in Random Data", "Details": "\r\n\r\n\r\n \r\n \r\n False Trends in Random Data\r\n \r\n\r\n\r\n

False Trends in Random Data

\r\n\r\n

\r\n The Cars2023 dataset was generated entirely at random. \r\n Each categorical attribute\u2014such as dealership, car maker, and gender\u2014follows a uniform distribution. \r\n The numerical attribute, age, was drawn from a normal distribution. \r\n Importantly, all attributes were created to be mutually independent. \r\n

\r\n\r\n

\r\n Despite this, in the following exploration we will observe what look like \r\n significant trends. These patterns are false signals\u2014the result of \r\n overly narrow queries that isolate small slices of the data. Such \u201cfindings\u201d are not \r\n real relationships but simply the noise of randomness masquerading as insight. \r\n

\r\n\r\n

Why False Trends Appear

\r\n

Law of Small Numbers: Small samples fluctuate widely, making chance differences look meaningful.
Multiple Comparisons: Searching across many combinations (city \u00d7 brand \u00d7 gender \u00d7 age) almost guarantees some \u201csignificant\u201d results by luck.
Narrow Queries: The more specific the filter, the easier it is for randomness to dominate the outcome.
Illusion of Causality: Human intuition is quick to connect patterns\u2014even when no real relationship exists.

\r\n\r\n

Reality Check

\r\n

Scramble the dataset: Randomly shuffle each column to destroy real relationships while keeping distributions intact.
Re-run the same queries: Look again at those narrow trends in the scrambled dataset.
Compare side by side: If trends vanish or reverse, they were never real to begin with\u2014they were products of chance.

\r\n\r\n

Takeaway

\r\n

\r\n This exercise is a reminder: even in a dataset designed to contain no \r\n relationships at all, we can still \u201cdiscover\u201d seemingly convincing patterns. \r\n Without proper statistical checks, these illusions can lead to wrong business actions and misguided decisions.\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "4.2", "Name": "Summary", "Details": " Fooled by Randomness

Fooled by Randomness

In data science, we love discovering patterns. We ask: who buys which car, when, and where? How do age, gender, or location relate to behavior? Behind every chart or table there is a tempting story waiting to be told. But there is a danger hidden in this curiosity: sometimes the \u201cstory\u201d we find is nothing more than random noise dressed up as meaning. The central question is: how do we know that our findings are not just random? :contentReference[oaicite:0]{index=0}

The Law of Small Numbers

Psychologist Daniel Kahneman coined the term \u201cLaw of Small Numbers\u201d to describe how people overinterpret patterns in small samples. Small datasets are naturally prone to extreme results. If you draw a few balls from an urn filled with red and black balls, it is quite easy to get all red or all black. If you draw thousands of balls, extreme imbalances become rare. :contentReference[oaicite:1]{index=1}

The same logic applies to data analysis. When we zoom in on very narrow subsets\u2014like one dealership, one season, one car brand, and one age group\u2014we are effectively \u201cdrawing a few balls from the urn.\u201d Extreme percentages or large differences in means are much more likely to appear just by chance.

A Completely Random Cars Data Set

Imagine a synthetic data frame called Cars with variables:

Dealership
Season
Car
Buyer
Buyer_Age

This data is generated to be entirely random. Car makes, dealerships, seasons, buyer ages, and genders all follow simple uniform distributions. There are no intentionally built-in patterns. It should be a \u201cboring\u201d dataset where nothing interesting is really going on. :contentReference[oaicite:2]{index=2}

And yet, if we start running many queries on this random data, \u201cdiscoveries\u201d begin to appear.

\u201cDiscoveries\u201d in Random Data

Here are some examples of surprising-looking patterns that can pop up:

In winter, among senior customers (Age > 70) at the LA dealership, 13.1% buy Toyota and only 6.25% buy GMC. It looks like seniors in LA are twice as likely to buy Toyota as GMC. :contentReference[oaicite:3]{index=3}
At the Austin dealership, among Honda buyers, the gender distribution is 54% female and 46% male. It looks like Honda \u201cfavours\u201d female customers there. :contentReference[oaicite:4]{index=4}
In Chicago, the mean age of Jeep buyers in spring is 54, while in summer it\u2019s 50. A four-year difference in average age might sound meaningful. :contentReference[oaicite:5]{index=5}
Among GMC buyers in LA in their late twenties, 61% are female and 39% are male\u2014a 22% gender gap. :contentReference[oaicite:6]{index=6}

Each of these statements sounds like a marketing insight: \u201cSeniors in LA prefer Toyota,\u201d or \u201cJeep attracts older buyers in spring.\u201d The problem is that the data is completely random. The \u201cpatterns\u201d exist only because we looked at many different subsets until something extreme appeared.

Why False Discoveries Happen

Two main forces drive these fake discoveries:

Narrow subsets (Law of Small Numbers). When we slice the data very finely\u2014 dealership + car make + season + age interval\u2014the number of rows in each subset becomes small. Small samples have high variability, so extreme percentages or big differences in means are more common.
Pattern hunting (Multiple Hypothesis Testing). There are an exponential number of possible queries: we can vary age ranges, dealership names, car makes, seasons, and more. If we keep trying different combinations and only report the ones that look \u201cinteresting,\u201d we are guaranteed to find some false positives even in pure noise. :contentReference[oaicite:7]{index=7}

This behaviour is often called p-value hunting or p-hacking. We quietly ignore the dozens of queries that show nothing special and proudly present the one that looks significant. Statistical tools like the Bonferroni correction exist precisely to account for this \u201ctry until something works\u201d process.

Pattern Hunting in Practice

In the cars example, the \u201cdiscoveries\u201d shown did not come from a single, pre-planned query. Many different queries were tried first, most of which produced unremarkable results. Only after stumbling on subsets with eye-catching numbers were those highlighted. :contentReference[oaicite:8]{index=8}

This is a realistic picture of what can happen in real projects:

You run dozens or hundreds of group-by summaries.
You focus on the few that show big gaps or large lifts.
You forget that randomness alone can generate \u201cwow\u201d moments when you search long enough.

How Not to Be Fooled by Randomness

The key lesson is not \u201cstop exploring data\u201d\u2014exploration is essential. Instead, we must learn to explore responsibly:

Be suspicious of strong claims based on very small subsets. Always check how many rows are behind the percentage or the mean.
Keep track of how many queries you tried, not only the ones that \u201cworked.\u201d The more you try, the more you must adjust your standards for what counts as surprising.
When possible, confirm discovered patterns on a fresh dataset or a held-out test set. Real patterns tend to repeat; random flukes usually do not.
Learn and apply multiple-hypothesis testing tools (like Bonferroni) when you conduct many related tests.

At the end of the day, data science is as much about skepticism as it is about discovery. By respecting the Law of Small Numbers and being honest about how many patterns we have searched for, we can avoid being fooled by randomness and focus on insights that truly reflect the underlying reality.

", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "4.21", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Fooled by Randomness\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Fooled by Randomness and Law of Small Numbers

\r\n \r\n\r\n

\r\n

1. In data science, what central danger is highlighted by \u201cFooled by randomness\u201d?

\r\n

A) Using too much data
B) Mistaking random noise for real patterns
C) Plotting data instead of computing tables
D) Using R instead of Python

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. According to the \u201claw of small numbers\u201d (Kahneman), small samples are:

\r\n

A) Always more accurate than large samples
B) Less likely to show extreme results than large samples
C) More likely to show extreme results than large samples
D) Irrelevant in data analysis

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. The \u201claw of small numbers\u201d is illustrated in the slides with which analogy?

\r\n

A) Tossing many coins
B) Selecting balls from an urn with red and black balls
C) Rolling a single die
D) Drawing cards from a full deck without replacement

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. In the synthetic Cars data set (Dealership, Season, Car, Buyer, Buyer_Age...), the content is:

\r\n

A) Designed with strong real patterns
B) Completely random, with no real patterns expected
C) Biased toward Toyota buyers
D) Collected from real car dealerships

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

5. Which of the following is one of the \u201cdiscoveries\u201d reported in the random Cars data? (Even though the data are random.)

\r\n

A) Seniors in LA never buy Toyota
B) During Winter, among seniors in LA dealership, Toyota share is about twice GMC share
C) In Spring, nobody buys Jeep in Chicago
D) All dealerships sell the same number of cars

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

6. The slides show that in random data we still observe patterns such as \u201cmean age of Jeep buyers in Chicago in spring is 54 vs 50 in summer\u201d. Why is this important?

\r\n

A) It proves Chicago customers are older in spring
B) It shows that means are useless statistics
C) It illustrates that narrow subsets can produce big differences just by chance
D) It shows season has a strong causal effect on age

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

7. What do the slides call the behavior of \u201ckeep trying different queries until you find something interesting worth reporting\u201d?

\r\n

A) Feature scaling
B) Pattern hunting or p-value hunting
C) Cross-validation
D) Gradient descent

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

8. Why does pattern hunting in the Cars data lead to many false discoveries?

\r\n

A) Because the dataset is too small to analyze
B) Because only one query is run
C) Because there are exponentially many similar queries and we only report the ones that look \u201cinteresting\u201d
D) Because we always use the full dataset without subsetting

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

9. The slides connect pattern hunting in the Cars data to which formal statistical concept?

\r\n

A) Linear regression
B) Overfitting in training vs testing
C) Multiple hypothesis testing and false positives
D) Chi-square test for independence

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

10. One of the questions posed at the end is: \u201cFind a subset of Cars (not using Buyer_Age) which has the maximum mean age.\u201d What is the hidden lesson behind this task?

\r\n

A) How to compute mean age correctly in R
B) How easy it is to use age as a predictor
C) How systematically searching many subsets can accidentally create \u201cevidence\u201d from pure randomness
D) How to remove randomness from data

\r\n

Answer: C

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "4.3", "Name": "Convo about narrow queries", "Details": "\n Narrow queries chat\n $\"ChatGPT$ \n \n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "4.4", "Name": "AI Professor ", "Details": "\n Fooled by Randomness\n $\"ChatGPT$ \n ", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "5", "Name": "Central Limit Theorem", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "5.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Central Limit Theorem and the Normal Distribution\r\n \r\n\r\n\r\n\r\n

Central Limit Theorem and the Magic of Sample Means

\r\n\r\n

\r\n In data science and statistics, one theorem quietly powers a huge part of what we do:\r\n the Central Limit Theorem (CLT). It explains why the\r\n normal distribution shows up everywhere and why we are able\r\n to use z-scores and hypothesis tests in so many different situations.\r\n

\r\n\r\n

The Normal Distribution

\r\n\r\n

\r\n The normal distribution is the famous \u201cbell-shaped curve.\u201d It is centered around a mean\r\n (average) value, and most of the data lies close to this mean. As you move farther away\r\n from the mean, values become less and less common. Many real-world measurements, like\r\n heights, test scores, or small random errors, are approximately normal.\r\n

\r\n\r\n

\r\n Key idea: A normal distribution is completely determined by two numbers:\r\n its mean (center) and standard deviation (spread).\r\n

\r\n\r\n

Z-Score: Measuring How Far a Value Is

\r\n\r\n

\r\n A z-score tells you how many standard deviations a value\r\n is away from the mean. For a value x, the z-score is:\r\n

\r\n\r\n

\r\n z = |x \u2212 mean| / standard deviation\r\n

\r\n\r\n

\r\n We take the absolute value of the difference between x and the mean, and then\r\n divide by the standard deviation. This gives us a standardized measure of \u201cdistance.\u201d\r\n Thanks to z-scores, we can put any normal distribution onto the same scale. It doesn\u2019t\r\n matter what the original mean and standard deviation were \u2014 z-scores turn everything\r\n into a common language.\r\n

\r\n\r\n

\r\n Why is this useful? Once you convert values to z-scores, you can use\r\n standard normal tables (or software) to find probabilities, compare different datasets,\r\n and perform hypothesis tests.\r\n

\r\n\r\n

From Histograms to Sample Means

\r\n\r\n

\r\n Imagine we start with some population of values and draw a simple histogram of these\r\n values. This histogram shows the original distribution of the data. It might be skewed,\r\n lopsided, or even very strange-looking \u2014 it does not have to be normal.\r\n

\r\n\r\n

\r\n Now, instead of looking at individual values, we take random samples from this population.\r\n For example, we might:\r\n

\r\n\r\n

Take a random sample of size n.
Compute the sample mean (average) for that sample.
Repeat this many, many times.

\r\n\r\n

\r\n Each time, we get a new sample mean. If we make a histogram of all these sample means,\r\n we get the distribution of sample means. This is where the\r\n Central Limit Theorem comes in.\r\n

\r\n\r\n

The Central Limit Theorem

\r\n\r\n

\r\n The Central Limit Theorem says:\r\n

\r\n\r\n

\r\n If you take sufficiently large random samples (usually n \u2265 30) from any population \u2014\r\n no matter what the original distribution looks like \u2014 the distribution of the sample means\r\n will be approximately normal (bell-shaped).\r\n

\r\n\r\n

\r\n This is an amazing statement. The original data can be skewed, heavy-tailed, or weird, but\r\n the averages of large random samples behave nicely: their distribution becomes normal as\r\n the sample size grows.\r\n

\r\n\r\n

\r\n This \u201cnormalization by averaging\u201d is what makes the normal distribution so special and so\r\n central in statistics. As the sample size n gets larger and larger, the approximation\r\n gets better and better \u2014 we say the distribution of sample means becomes normal\r\n asymptotically.\r\n

\r\n\r\n

Why the Central Limit Theorem Matters

\r\n\r\n

\r\n The Central Limit Theorem is one of the most important theorems in statistics. It is the\r\n foundation of many tools you use in data analysis:\r\n

\r\n\r\n

Confidence intervals for unknown means.
Hypothesis tests comparing means or checking if a mean is equal to some value.
Using z-scores and normal probabilities even when the original data is not normal.

\r\n\r\n

\r\n In practice, this means that if your sample size is large enough, you can safely use\r\n methods based on the normal distribution, even if the population distribution is unknown\r\n or clearly non-normal. This is why the normal distribution is \u201cvery special\u201d in statistics:\r\n it appears not only as a model for data, but as a universal model for averages.\r\n

\r\n\r\n

\r\n In short: No matter what your original data look like, if you take enough\r\n random samples and look at their means, those means will form a bell-shaped curve. This is\r\n the Central Limit Theorem in action, and it is what makes modern hypothesis testing and\r\n much of statistical inference possible.\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "5.2", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Central Limit Theorem, Normal Distribution, and Z-Scores\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Central Limit Theorem, Normal Distribution, and Z-Scores

\r\n \r\n\r\n

\r\n

1. What does the Central Limit Theorem (CLT) say about the distribution of sample means?

\r\n

A) It has the same shape as the original population, for any sample size.
B) It is always uniform, no matter what the population looks like.
C) For sufficiently large n, the distribution of sample means is approximately normal.
D) It is always skewed to the right.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

2. In practice, what is a common rule-of-thumb for \u201csufficiently large\u201d sample size n in the CLT?

\r\n

A) n \u2265 5
B) n \u2265 10
C) n \u2265 30
D) n \u2265 100

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. The z-score for a value x in a normal distribution is defined as:

\r\n

A) (x \u2212 mean) \u00d7 standard deviation
B) (mean \u2212 x) / standard deviation
C) (x \u2212 mean) / standard deviation
D) standard deviation / (x \u2212 mean)

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

4. Why is the z-score useful for normal distributions?

\r\n

A) It makes all normal distributions have mean 1 and SD 1.
B) It rescales any normal distribution to a common scale so we can use one standard normal table.
C) It removes all randomness from the data.
D) It only works for sample sizes n \u2265 30.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

5. According to the CLT, if we repeatedly take samples of size n from a population and compute the mean of each sample, then:

\r\n

A) The histogram of sample means will look more and more like a bell curve as n grows.
B) The histogram of sample means will look flatter as n grows.
C) The histogram of sample means is always the same as the original population.
D) The sample means will all be identical.

\r\n

Answer: A

\r\n

\r\n\r\n

\r\n

6. The Central Limit Theorem is especially important because:

\r\n

A) It only explains how to draw histograms.
B) It is the foundation of many hypothesis tests and confidence intervals.
C) It guarantees that the original data are normal.
D) It eliminates the need for random sampling.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

7. As sample size n increases, what happens to the spread (standard deviation) of the distribution of sample means?

\r\n

A) It stays the same as the population standard deviation.
B) It increases linearly with n.
C) It decreases; sample means cluster more tightly around the population mean.
D) It becomes infinite.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

8. In the slides, the \u201coriginal distribution\u201d is shown as a histogram of the data. What does the \u201csample means distribution\u201d represent?

\r\n

A) A histogram of all individual data points.
B) A histogram of the sample means from many random samples of size n.
C) A histogram of z-scores only for outliers.
D) A histogram of standard deviations of the samples.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

9. Which statement best compares the original data distribution and the distribution of sample means?

\r\n

A) Both must be normal for the CLT to work.
B) The original data must be normal, but the sample means can have any shape.
C) The original data can have any shape; the sample means become approximately normal for large n.
D) Both will always be uniform.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

10. Why does the CLT make the normal distribution \u201cvery special\u201d in statistics, according to the slides?

\r\n

A) Because most real-world data are exactly normal.
B) Because, regardless of the original distribution, sample means converge to a normal distribution as n grows.
C) Because normal distributions never require z-scores.
D) Because only normal distributions can be plotted as histograms.

\r\n

Answer: B

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "6", "Name": "Confidence Intervals ", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "6.01", "Name": "Formulas ", "Details": "\r\n\r\n\r\n \r\n Confidence Interval Formulas\r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n\r\n\r\n

\r\n

Confidence Interval Formulas

\r\n

Minimal handout using MathJax/LaTeX for clean math typesetting.

\r\n\r\n \r\n

\r\n

Confidence interval for the mean (z* version)

\r\n

\r\n \\[\r\n \\boxed{ \\ \\bar{x} \\;\\pm\\; z^{*}\\,\\frac{s}{\\sqrt{n}} \\ }\r\n \\]\r\n

\r\n

\$\\bar{x}\$ \u2014 sample mean.
\$z^{*}\$ \u2014 critical z-value for the chosen confidence level (e.g., \$1.96\$ for 95%).
\$s\$ \u2014 sample standard deviation (using \$s\$ in place of population \$\\sigma\$).
\$n\$ \u2014 sample size.

\r\n

\r\n\r\n \r\n

\r\n

Confidence interval for a proportion

\r\n

\r\n \\[\r\n \\boxed{ \\ \\hat{p} \\;\\pm\\; z^{*}\\,\\sqrt{\\dfrac{\\hat{p}\\,\\bigl(1-\\hat{p}\\bigr)}{n}} \\ }\r\n \\]\r\n

\r\n

\$\\hat{p}\$ \u2014 sample proportion (successes divided by \$n\$).
\$z^{*}\$ \u2014 critical z-value for the chosen confidence level.
\$n\$ \u2014 sample size (number of independent trials).

\r\n

\r\n\r\n\r\n\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "6.02", "Name": "Formuless", "Details": "\r\n\r\n\r\n \r\n Confidence Interval Effects\r\n \r\n\r\n\r\n

Confidence Interval for Means

\r\n

Sample size \u2191\u2192 MOE \u2193
Confidence Level \u2191\u2192 MOE \u2191
Standard Deviation \u2191\u2192 MOE \u2191

\r\n\r\n

Confidence Interval for Proportions

\r\n

Sample size \u2191\u2192 MOE \u2193
Confidence Level \u2191\u2192 MOE \u2191
Population Proportion \u2191\u2192 MOE ~

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "6.05", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Confidence Intervals and Margin of Error\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Confidence Intervals and Margin of Error

\r\n \r\n\r\n

\r\n

1. A confidence interval for a mean is best described as:

\r\n

A) A single best guess of the population mean
B) A range of values that likely contains the true population mean
C) The exact set of all possible sample means
D) A guarantee that the true mean is inside the range

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. In the Airbnb example, the sample mean nightly price is $150 from 500 listings. What does a 95% confidence interval of (147.37, 152.63) mean?

\r\n

A) 95% of Airbnb prices are between $147.37 and $152.63
B) 95% of all future samples will have mean $150
C) We are 95% confident that the true mean price lies between $147.37 and $152.63
D) The true mean must be exactly $150

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. The standard error of the mean (SEM) is computed as:

\r\n

A) SEM = s \u00d7 n
B) SEM = s / sqrt(n)
C) SEM = n / s
D) SEM = s\u00b2 / n

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. With sample standard deviation s = 30 and sample size n = 500, which value is closest to SEM in the slides?

\r\n

A) 0.30
B) 1.34
C) 3.00
D) 13.40

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

5. For a 95% confidence interval using the normal approximation, which critical value z* is used?

\r\n

A) 1.00
B) 1.64
C) 1.96
D) 2.58

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

6. In the Airbnb example, the margin of error (MOE) is computed as MOE = z* \u00d7 SEM = 1.96 \u00d7 1.34. Which value is closest to the MOE?

\r\n

A) 1.34
B) 2.63
C) 3.96
D) 5.00

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

7. How does the margin of error (MOE) change when the sample size n increases, keeping everything else the same?

\r\n

A) MOE increases
B) MOE stays the same
C) MOE decreases
D) MOE becomes zero

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

8. How does the margin of error change when we move from a 90% to a 95% confidence level (same data)?

\r\n

A) It becomes smaller (narrower interval)
B) It becomes larger (wider interval)
C) It stays exactly the same
D) It becomes zero

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

9. Which of the following is one of the \u201csanity checks\u201d suggested in the slides for catching suspicious CI/MOE results?

\r\n

A) If sample size goes down, MOE should go down
B) If confidence level goes up, MOE should go down
C) The CI for a mean should usually be centered at (or very near) the sample mean
D) The CI should never be checked against a plot of the data

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

10. For a confidence interval of a proportion p, the margin of error is given by:

\r\n

A) MOE = z* \u00d7 s / sqrt(n)
B) MOE = z* \u00d7 sqrt(p(1 \u2212 p)/n)
C) MOE = p(1 \u2212 p)
D) MOE = z* / sqrt(n)

\r\n

Answer: B

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "6.1", "Name": "Convo about confidence intervals of mean", "Details": "\n Confidence intervals of means\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "6.2", "Name": "Convo about confidence intervals of proportions", "Details": "\n Confidence intervals of proportions\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "6.3", "Name": "Language of Data: Confidence Intervals", "Details": "\r\n\r\n\r\n\r\nLanguage of Data \u2014 Confidence Intervals (Vibe Glossary)\r\n\r\n\r\n\r\n\r\n

\r\n

Language of Data \u2014 Confidence Intervals (Glossary for Vibing)

\r\n

Goal: Know the words to use when vibing about confidence intervals \u2014 for means or proportions.

\r\n

\r\n\r\n

\r\n

Core Terms for Vibing

\r\n

Sample Mean (x\u0304)
Population Mean (\u03bc)
Sample Size (n)
Standard Deviation (s)
Standard Error (SE or SEM)
Margin of Error (MOE)
Critical Value (z*)
Confidence Level (90%, 95%, 99%)
Confidence Interval (CI)
Lower Bound
Upper Bound
Interval Width
Center of Interval
Confidence Interval for Mean
Confidence Interval for Proportion
Sample Proportion (p*)
True Proportion (p)
SE of Proportion
MOE of Proportion
Interpretation of CI
Repeatability

\r\n\r\n

10 Vibe Prompts (Means & Proportions)

\r\n

Find the 95% confidence interval for mean Airbnb rental price using sample mean = 150, s = 30, n = 500.
Compute the margin of error for mean attendance at weekend parties (95% confidence).
Construct a 90% confidence interval for mean student score in Moody dataset.
Compare two neighborhoods: mean price in NoHo = 160, Murray Hill = 145. Build separate 95% confidence intervals and discuss overlap.
Find the 99% confidence interval for mean DJ rating in Imprezka dataset.
In an election poll, 48% support candidate A (n=1000). Build the 95% confidence interval for proportion of support.
Compute the 90% confidence interval for proportion of buyers who choose SUVs.
Estimate the 95% confidence interval for proportion of students with attendance \u2265 0.8.
Show how margin of error changes when sample size doubles (n vs 2n) for same confidence level.
Check how confidence level (90%, 95%, 99%) changes interval width for mean price of cars.

\r\n\r\n

\r\nStudents should phrase prompts like: \r\n\u201cBuild confidence interval for mean price (95%) using sample mean and SD,\u201d or \r\n\u201cFind confidence interval for proportion of voters (n=1000, p*=0.48).\u201d\r\n

\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "7", "Name": "Hypothesis Testing", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "7.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Hypothesis Testing and the Permutation Test\r\n \r\n\r\n\r\n\r\n

Hypothesis Testing and the Permutation Test

\r\n\r\n

\r\n Randomness is the biggest enemy of a data scientist. When we see a pattern in data,\r\n our first question should be: Is this real, or could it be just random noise?\r\n The goal of hypothesis testing is to help us answer\r\n that question in a disciplined, quantitative way. One of the most intuitive tools\r\n for this is the permutation test. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

The Tunnel Traffic Example

\r\n\r\n

\r\n Suppose we compare traffic through two tunnels: Lincoln and Holland. We compute the\r\n average traffic volume per minute using observed data:\r\n

\r\n\r\n

Lincoln: mean volume = 68.54 cars per minute
Holland: mean volume = 67.71 cars per minute

\r\n\r\n

\r\n The observed difference in means is\r\n D = 0.83, suggesting that Lincoln might be busier.\r\n But we must be careful: is this difference genuine, or could it easily happen just\r\n by chance if the tunnels actually have the same typical traffic?\r\n

\r\n\r\n

Null and Alternative Hypotheses

\r\n\r\n

\r\n To formalize the question, we set up two competing statements:\r\n

\r\n\r\n

\r\n Null hypothesis (H\u2080):\r\n there is no difference in traffic between the Lincoln and Holland tunnels.\r\n
\r\n Alternative hypothesis (H\u2090):\r\n the Lincoln tunnel is busier than the Holland tunnel (its mean traffic is higher).\r\n

\r\n\r\n

\r\n Hypothesis testing asks: Assuming H\u2080 is true, how surprising would our\r\n observed difference D = 0.83 be? If it is very unlikely under H\u2080, we may decide\r\n to reject H\u2080 in favor of H\u2090.\r\n

\r\n\r\n

Permutation Test: Simulating Randomness

\r\n\r\n

\r\n The permutation test gives us a way to simulate what \u201crandom\u201d would look like if the\r\n null hypothesis were true. The idea is simple:\r\n

\r\n\r\n

Take the original traffic data for both tunnels combined.
Randomly shuffle (permute) the tunnel labels on all observations.
After shuffling, reassign each traffic volume to a tunnel and recompute the\r\n difference of means; call this D_null.
Repeat this process many times (1,000, 10,000, or more) to build a distribution\r\n of D_null values that represent \u201cjust randomness.\u201d

\r\n\r\n

\r\n Each permutation completely breaks the original relationship between volume and\r\n tunnel name, which is exactly what the null hypothesis says: tunnel and traffic\r\n volume are unrelated. Each permutation is like a roll of a dice, showing us what\r\n kinds of differences in means could appear if there were no real effect.\r\n

\r\n\r\n

The p-Value: How Surprising Is Our Result?

\r\n\r\n

\r\n Once we have many simulated differences D_null, we compare them to our\r\n observed difference D = 0.83. The p-value is defined\r\n as the fraction of permutations for which\r\n D_null \u2265 D (for a \u201cgreater than\u201d alternative).\r\n In other words:\r\n

\r\n\r\n

\r\n The p-value is the probability, under the null hypothesis, of observing a difference\r\n in means at least as extreme as the one we actually saw.\r\n

\r\n\r\n

\r\n If, for example, out of 1,000 permutations we see D_null \u2265 0.83 only 3 times,\r\n then the p-value is 3/1,000 = 0.003. That means that if there were really no difference\r\n between tunnels, a difference of 0.83 or more would happen by chance only 0.3% of the time.\r\n

\r\n\r\n

Significance Level and Decision

\r\n\r\n

\r\n To decide whether a result is \u201csignificant,\u201d we choose a\r\n significance level, usually\r\n \u03b1 = 0.05. The decision rule is:\r\n

\r\n\r\n

If p \u2264 \u03b1, we reject the null hypothesis.
If p > \u03b1, we fail to reject the null hypothesis.

\r\n\r\n

\r\n In our example, if p = 0.003, then p < 0.05, so we would reject H\u2080 and conclude that\r\n the observed difference is unlikely to be random. If p were, say, 0.12, then we could\r\n not rule out randomness; the data would be consistent with \u201cno real difference.\u201d\r\n

\r\n\r\n

Common Misconceptions About p-Values

\r\n\r\n

\r\n It is important to understand what a p-value does not say:\r\n

\r\n\r\n

A small p-value does not prove that the null hypothesis is false.
A large p-value does not prove that the null hypothesis is true.

\r\n\r\n

\r\n Instead, the p-value measures the strength of evidence against H\u2080. Small p-values\r\n mean the observed data would be very unusual if H\u2080 were true, so the data provide\r\n strong evidence against H\u2080. Large p-values mean the data are compatible with H\u2080;\r\n they do not give us strong evidence to reject it.\r\n

\r\n\r\n

Beyond Tunnels: Other Difference-of-Means Questions

\r\n\r\n

\r\n The same logic applies to many situations where we compare two means:\r\n

\r\n\r\n

Difference in mean attendance for Techno vs. HipHop in the Party dataset.
Difference in mean score for CS vs. Statistics majors in the Moody dataset.
Difference in mean rating for DJ Blue vs. DJ Mski on Saturday nights.

\r\n\r\n

\r\n In each case, we specify H\u2080 (\u201cno difference\u201d), H\u2090 (\u201cone mean is larger\u201d), run a\r\n permutation test to simulate randomness, compute the p-value, and decide whether\r\n we can attribute the observed difference to chance or to a real effect.\r\n

\r\n\r\n

Why the Permutation Test Matters

\r\n\r\n

\r\n The permutation test makes randomness tangible. Instead of relying only on formulas,\r\n we actually simulate many worlds where H\u2080 is true and see how often a result\r\n like ours appears. This not only gives a valid p-value, but also builds intuition:\r\n you can literally watch the histogram of D_null values and see where your\r\n observed D falls.\r\n

\r\n\r\n

\r\n Hypothesis testing, and the permutation test in particular, is about guarding ourselves\r\n against being fooled by randomness. Only when our observed result is sufficiently rare\r\n under the null hypothesis do we dare to say: \u201cThis pattern is probably real.\u201d\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "7.01", "Name": "Formulas ", "Details": "\r\n\r\n\r\n \r\n One-Sided Two-Sample Z-Test for Means \u2014 Minimal Handout\r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n\r\n\r\n

\r\n

One-Sided Two-Sample Z-Test for Means

\r\n\r\n

\r\n

Setup & Test Statistic

\r\n

\r\n \\[\r\n H_0:\\ \\mu_1 - \\mu_2 = \\Delta_0\r\n \\qquad\r\n H_a:\\ \\mu_1 - \\mu_2 >\r\n \\Delta_0\\ \\ (\\text{right-tailed})\r\n \\quad \\text{or} \\quad\r\n H_a:\\ \\mu_1 - \\mu_2 < \\Delta_0\\ \\ (\\text{left-tailed})\r\n \\]\r\n \\[\r\n z \\;=\\; \\frac{\\bigl(\\bar{x}_1 - \\bar{x}_2\\bigr) - \\Delta_0}\r\n {\\sqrt{\\dfrac{\\sigma_1^{2}}{n_1} \\;+\\; \\dfrac{\\sigma_2^{2}}{n_2}}}\r\n \\]\r\n

\r\n

\$\\bar{x}_1,\\ \\bar{x}_2\$ \u2014 sample means of groups 1 and 2
\$\\mu_1,\\ \\mu_2\$ \u2014 population means of groups 1 and 2
\$\\Delta_0\$ \u2014 hypothesized difference (often \$0\$)
\$\\sigma_1,\\ \\sigma_2\$ \u2014 population standard deviations (large-sample substitute SE using \$s_1, s_2\$ if \$\\sigma\$ unknown)
\$n_1,\\ n_2\$ \u2014 sample sizes

\r\n

\r\n\r\n

\r\n

Decision Rule & p-Value

\r\n

\r\n \\[\r\n \\text{Reject } H_0 \\text{ if }\r\n \\begin{cases}\r\n z \\ge z^{*}_{\\alpha} & \\text{(right-tailed)}\\\\[4pt]\r\n z \\le -\\,z^{*}_{\\alpha} & \\text{(left-tailed)}\r\n \\end{cases}\r\n \\]\r\n \\[\r\n p\\text{-value} =\r\n \\begin{cases}\r\n 1-\\Phi(z) & \\text{(right-tailed)}\\\\[4pt]\r\n \\Phi(z) & \\text{(left-tailed)}\r\n \\end{cases}\r\n \\]\r\n

\r\n

\$z^{*}_{\\alpha}\$ \u2014 one-sided critical z for level \$\\alpha\$ (e.g., \$z^{*}_{0.05}\\approx 1.645\$)
\$\\Phi(\\cdot)\$ \u2014 standard normal CDF

\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "7.02", "Name": "Formuless", "Details": "\r\n\r\n\r\n \r\n Two-Sample One-Sided Mean Test \u2014 Effects on p-value\r\n \r\n \r\n\r\n\r\n

How Changes Affect the p-value

\r\n

Difference of sample means \u2191\u2192 p-value \u2193
Standard deviation of populations \u2191\u2192 p-value \u2191
z-score \u2191\u2192 p-value \u2193

\r\n\r\n \r\n

\r\n \u2191\u2193~\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "7.1", "Name": "Convo-Hypothesis Testing", "Details": "\r\n\r\n\r\n \r\n \r\n REPLAY the chat in class\r\n \r\n\r\n\r\n

\r\n

REPLAY the chat in class

\r\n

Import the dataset and then enter the prompts below.

\r\n

\r\n\r\n

\r\n

Import Data set

\r\n

airbnbN2023 (from Canvas Files)

\r\n

\r\n\r\n

\r\n

Prompts

\r\n

Show a sample of 3 tuples from this data set
Show results of 3 scrambles (permutations) of this minisample
We will use permutation test in hypothesis testing. Let us start with the first hypothesis - it is one side hypothesis that mean price of Jamaica rentals is higher than mean of Flushing? Lets start with showing what both means are. The show both null and alternative hypothesis. Run 1000 permutations, show histogram of these permutations and mark the real difference of means on this histogram. Finally display p-value and form the conclusion.
Now compare NoHo with Murray Hill, form hypothesis, calculate p-value just like above.
What was z-score of the observed difference here?
Can you explain calculations that z-test required?

\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "7.2", "Name": "Language of Data: Hypothesis testing", "Details": "\r\n\r\n\r\n\r\nLanguage of Data \u2014 Hypothesis Testing (Vibe Glossary)\r\n\r\n\r\n\r\n\r\n

\r\n

Language of Data \u2014 Hypothesis Testing (Glossary for Vibing)

\r\n

Goal: Know the words to use when vibing about hypothesis testing, either with a z-test or a permutation test.

\r\n

\r\n\r\n

\r\n

Core Terms to Use in Vibes

\r\n

Null Hypothesis (H\u2080)
Alternative Hypothesis (H\u2081)
Observed Difference (D)
Difference of Means
Permutation Test
z-Test
Simulating Randomness
Random Re-labeling
Observed Statistic
Null Distribution (D_null)
p-Value
Significance Level (\u03b1)
Reject H\u2080
Fail to Reject H\u2080
One-Sided Test
Two-Sided Test
Difference of Proportions
Standard Error (SE)
z-Score
Histogram of D_null

\r\n\r\n

Vibing Prompts (All must be z-test or permutation test)

\r\n

Test this hypothesis: mean price in NoHo > mean price in Murray Hill (use z-test or permutation test by scrambling prices).
Lincoln vs Holland tunnel \u2014 is mean traffic higher in Lincoln? Try z-test or permutation test.
Is mean Attendance higher for Techno than HipHop DJs? Test with z-test or permutation test.
Are CS majors scoring higher than Statistics majors? Use z-test or permutation test.
Is average Rating higher for DJ Blue than DJ Mski on Saturday nights? Try z-test or permutation test.
Are high-attendance students (\u22650.8) getting more A\u2019s? Compare proportions with z-test or permutation test.
Is mean Airbnb price in LA higher than in Chicago? z-test or permutation test on mean prices.
Is click-through rate greater for Campaign A than B? Use z-test or permutation test.
Do students who attend front rows score higher than back rows? Try z-test or permutation test.
Is average sales volume higher on weekends than weekdays? Use z-test or permutation test.

\r\n\r\n

Students should start their prompts with \u201cTest this hypothesis using a z-test\u201d or \u201cTest this hypothesis using a permutation test.\u201d

\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "7.21", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Hypothesis Testing and Permutation Test\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Hypothesis Testing and Permutation Test

\r\n\r\n

\r\n

1. In the Lincoln vs. Holland tunnel example, what is the observed difference of mean traffic volume per minute?

\r\n

A) 0.54 (Lincoln lower than Holland)
B) 0.83 (Lincoln higher than Holland)
C) 1.50 (Lincoln higher than Holland)
D) 2.00 (Lincoln higher than Holland)

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. What is the null hypothesis (H\u2080) in the Lincoln\u2013Holland tunnel example?

\r\n

A) Lincoln tunnel is busier than Holland tunnel.
B) Holland tunnel is busier than Lincoln tunnel.
C) There is no difference in traffic between Lincoln and Holland tunnels.
D) Both tunnels have zero traffic.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. What is the alternative hypothesis (H\u2090) in the tunnel example?

\r\n

A) There is no difference in traffic between the tunnels.
B) Lincoln tunnel is more busy than Holland tunnel.
C) Holland tunnel is more busy than Lincoln tunnel.
D) Traffic in both tunnels is completely random.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. What does a single permutation step do in the permutation test described in the slides?

\r\n

A) It changes the actual traffic volumes.
B) It swaps the names of the tunnels but keeps volumes fixed.
C) It randomly reassigns observed traffic volumes to tunnels, breaking any real relationship.
D) It deletes half of the data points at random.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

5. In the permutation test, what is D_null?

\r\n

A) The observed difference of means from the real data.
B) The difference of medians between the tunnels.
C) The difference of means for one random permutation under the null.
D) The p-value for the permutation test.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

6. How is the p-value defined in the permutation test described?

\r\n

A) The fraction of permutations with D_null exactly equal to 0.83.
B) The fraction of permutations where D_null is larger than or equal to the observed difference D.
C) The average value of D_null over all permutations.
D) The probability that the null hypothesis is true.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

7. Which of the following is the usual significance level \u03b1 used in the slides?

\r\n

A) 0.001
B) 0.01
C) 0.05
D) 0.5

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

8. According to the slides, when do we reject the null hypothesis?

\r\n

A) When p-value > \u03b1
B) When p-value = \u03b1
C) When p-value \u2264 \u03b1
D) When p-value \u2265 1 \u2212 \u03b1

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

9. Which of the following is listed as a misconception about p-values in the slides?

\r\n

A) Small p-value proves that H\u2080 is false.
B) Small p-value measures the strength of evidence against H\u2080.
C) Large p-value does not prove that H\u2080 is true.
D) p-value is the fraction of permutations with D_null \u2265 D.

\r\n

Answer: A
\r\n (Slides emphasize that small p-value does not prove H\u2080 is false, and large p-value does not prove H\u2080 is true.)

\r\n

\r\n\r\n

\r\n

10. Which of the following is not one of the example \u201cdifference of means\u201d hypothesis tests mentioned in the slides?

\r\n

A) Difference of mean volume per minute between Lincoln and Holland tunnels.
B) Difference of mean Attendance for Techno vs HipHop in Party data.
C) Difference of mean score for CS vs Statistics majors in Moody data.
D) Difference of median house prices between two cities.

\r\n

Answer: D

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "7.22", "Name": "Video", "Details": "\n Video of narrated slides\n $\"ChatGPT$ \nhttps://vimeo.com/1166396429?share=copy&fl=sv&fe=ci", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "8", "Name": "Multiple Hypotheses", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "8.1", "Name": "Summary", "Details": "\n\n\n \n Multiple Hypotheses and Bonferroni Correction\n \n\n\n\n

Multiple Hypotheses: When One p-Value Is Not Enough

\n\n

\n In data science, it is easy to fall in love with tiny p-values. We run a test,\n see p = 0.01, and feel confident that we have discovered something real.\n But what happens when we run not one test, but dozens, thousands, or even millions?\n The message of this module is simple: the more hypotheses\n you test, the more careful you must be.\n

\n\n

Jelly Beans and Acne: A Cartoon Warning

\n\n

\n The classic joke starts with a question: \u201cDo jelly beans cause acne?\u201d We test\n jelly beans by color: red, blue, green, yellow, and so on. For each color we run\n a separate hypothesis test:\n

\n\n

\n H\u2080: This color of jelly beans is not associated with acne.
\n H\u2090: This color of jelly beans is associated with acne.\n

\n\n

\n Most colors show nothing. But for green jelly beans,\n we get a p-value below 0.05 and declare, \u201cGreen jelly beans cause acne!\u201d\n The punchline: we have tested many colors, so it is not surprising that one of them\n looks significant just by chance. This is a false discovery.\n

\n\n

Family-Wise Error Rate (FWER)

\n\n

\n When we test m independent null hypotheses, each at significance level\n \u03b1 (for example, \u03b1 = 0.05), the chance that we get at least one false positive\n grows with m. This probability is called the\n Family-Wise Error Rate (FWER).\n

\n\n

\n FWER formula: FWER = 1 \u2212 (1 \u2212 \u03b1)\u1d50\n

\n\n

\n For the jelly bean example, suppose we test 9 colors at \u03b1 = 0.05.\n

\n\n

\n FWER = 1 \u2212 (1 \u2212 0.05)\u2079 \u2248 0.37\n

\n\n

\n That means there is about a 37% chance that at least\n one color will show a \u201csignificant\u201d result even if all null hypotheses are actually true.\n Our \u201cdiscovery\u201d of green jelly beans is not so impressive anymore.\n

\n\n

Expected Number of False Rejections

\n\n

\n Another way to see the problem: if all m null hypotheses are true and we\n use \u03b1 = 0.05 for each test, then on average we expect:\n

\n\n

\n Expected false rejections \u2248 \u03b1 \u00d7 m.\n

\n\n

\n If m = 10,000 tests, then 0.05 \u00d7 10,000 = 500 false positives just by chance.\n This is why \u201cfishing expeditions\u201d in wide data sets can produce long lists of\n apparently exciting but completely fake discoveries.\n

\n\n

Bonferroni Correction: Making \u03b1 Tougher

\n\n

\n How can we control the chance of at least one false positive? One simple and very\n conservative solution is the Bonferroni correction.\n

\n\n

\n Bonferroni idea: If you want to keep the overall FWER at level \u03b1,\n and you have m hypotheses, test each individual hypothesis at\n \u03b1 / m instead of \u03b1.\n

\n\n

\n Example: if \u03b1 = 0.05 and m = 100, then the corrected threshold is:\n

\n\n

\n \u03b1_corrected = 0.05 / 100 = 0.0005.\n

\n\n

\n Now each p-value must be below 0.0005 to be declared significant. This strongly\n reduces the chance of any false positives, but it also makes it harder to detect\n true effects. Bonferroni is safe, but strict.\n

\n\n

Supermarket Example: How \u201cWide\u201d Data Changes the Game

\n\n

\n Imagine a supermarket with N = 100 different items. We want to test whether each\n pair of items is associated (for example, chicken with burgers, bread with butter).\n The number of possible pairs is:\n

\n\n

\n N choose 2 = N(N \u2212 1) / 2.\n

\n\n

N = 10 \u2192 45 pairs
N = 100 \u2192 4,950 pairs
N = 1,000 \u2192 499,500 pairs
N = 10,000 \u2192 about 49 million pairs

\n\n

\n Suppose we find that the chicken\u2013burger pair has p-value = 0.000008 (8 \u00d7 10\u207b\u2076).\n Is this significant? It depends on how many tests we ran.\n

\n\n

\n If N = 100 (4,950 tests), Bonferroni says the threshold is roughly 10\u207b\u2075.\n Our p-value 8 \u00d7 10\u207b\u2076 is smaller than that, so we still reject the null:\n chicken and burgers really do go together.\n
\n If N = 1,000 (499,500 tests), Bonferroni threshold is about 10\u207b\u2078.\n Now 8 \u00d7 10\u207b\u2076 is not small enough. We fail to reject\n the null hypothesis after correction.\n

\n\n

\n The lesson: the wider your data (more features), the more\n extreme the p-values must be to pass the Bonferroni bar.\n

\n\n

Big Data and the Need for Softer Methods

\n\n

\n In modern applications, we often test huge numbers of hypotheses:\n

\n\n

Millions of SNPs in genomics.
Thousands of products in a supermarket.
Billions of combinations of ads, clicks, and user attributes.

\n\n

\n With such massive m, Bonferroni becomes almost impossible to pass.\n Very few real effects will ever reach such tiny p-values. For this reason,\n researchers often use less conservative methods, such as\n Holm\u2019s procedure or the\n Benjamini\u2013Hochberg method, which control different\n error measures (like the false discovery rate) instead of the strict FWER.\n

\n\n

When We Do (and Don\u2019t) Need Bonferroni

\n\n

\n If we decide in advance to test a small, fixed set of\n specific hypotheses, we may not need a multiple-testing correction.\n For example: comparing just Lincoln vs. Holland tunnel, or just two named\n neighborhoods, or two named drugs. The key is that these comparisons were\n chosen before looking at the data.\n

\n\n

\n However, once we let ourselves \u201clook at the data\u201d and choose the most extreme\n things — for example, the neighborhoods with the highest and lowest rent,\n or the product pair with the strongest association — we are effectively\n doing many tests. Even if we report only the best one, the selection step\n introduces multiple-testing issues. In these situations, a correction like\n Bonferroni (or a more advanced method) is necessary.\n

\n\n

The Big Picture

\n\n

\n Multiple hypothesis testing is about being honest with ourselves. The more things\n we try, the more likely we are to \u201cdiscover\u201d something that is just random noise.\n Family-wise error rate and Bonferroni correction give us a language and a tool to\n manage this risk.\n

\n\n

\n The key takeaways are:\n

\n\n

More tests \u2192 higher chance of false positives.
Bonferroni makes the p-value threshold tougher: \u03b1 / m.
Bonferroni is safe but conservative; big data often needs softer methods.
Any time we use the data to pick what to test, we should think about\n multiple hypotheses.

\n\n

\n In modern data science, avoiding false discoveries is just as important as finding\n exciting patterns. Multiple hypothesis thinking helps us do both.\n

\n\n\n\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "8.11", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Multiple Hypotheses and Bonferroni\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Multiple Hypotheses, FWER, and Bonferroni

\r\n \r\n\r\n

\r\n

1. In the \u201cacne and jelly beans\u201d story, what is the main statistical problem being illustrated?

\r\n

A) Biased sampling
B) Multiple hypothesis testing and false discoveries
C) Non-normal data
D) Small sample size

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. If we test m independent null hypotheses, each at significance level \u03b1, the Family-Wise Error Rate (FWER) is approximately:

\r\n

A) \u03b1 / m
B) m \u00d7 \u03b1
C) 1 \u2212 (1 \u2212 \u03b1)^m
D) (1 \u2212 \u03b1)^m

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. In the acne\u2013jelly beans example with 9 colors tested at \u03b1 = 0.05, the FWER is about:

\r\n

A) 0.05
B) 0.09
C) 0.37
D) 0.95

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

4. If we test m null hypotheses, all of which are actually true, and we use \u03b1 = 0.05 for each test, how many false rejections do we expect on average?

\r\n

A) 0.05
B) m / 0.05
C) 0.05 \u00d7 m
D) 1 \u2212 0.05^m

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

5. The Bonferroni correction controls FWER \u2264 \u03b1 by:

\r\n

A) Multiplying each p-value by m
B) Dividing the significance level by m and using \u03b1/m for each test
C) Ignoring half of the tests
D) Using \u03b1 = 0.5 for all tests

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

6. Suppose \u03b1 = 0.05 and we test N = 100 hypotheses. What is the Bonferroni-corrected significance level for each test?

\r\n

A) 0.05
B) 0.005
C) 0.0005
D) 0.5

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

7. A supermarket has 100 items. If we test all pairwise associations between items, how many hypotheses do we have?

\r\n

A) 100
B) 100 \u00d7 2 = 200
C) 100 choose 2 = 4,950
D) 10,000

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

8. In the supermarket example with 100 items, \u03b1 = 0.05, and 4,950 pairwise tests, the Bonferroni-corrected \u03b1 per test is approximately:

\r\n

A) 10^\u22122
B) 10^\u22123
C) 10^\u22124
D) 10^\u22125

\r\n

Answer: D (0.05 / 4950 \u2248 1\u00d710\u207b\u2075)

\r\n

\r\n\r\n

\r\n

9. When do we not need Bonferroni (or other multiple-testing) corrections, according to the slides?

\r\n

A) When sample size is large
B) When p-values are all below 0.01
C) When we only test a small fixed set of named hypotheses that were chosen before looking at the data
D) Whenever we use R instead of Python

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

10. Which statement best summarizes the trade-off of Bonferroni correction?

\r\n

A) It is very liberal: many false positives but few missed effects
B) It is very strict: reduces false positives but may miss real effects
C) It leaves the FWER unchanged
D) It only works for exactly two hypotheses

\r\n

Answer: B

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "8.2", "Name": "Jelly-beans experiment", "Details": "\r\n\r\n\r\n\r\n\r\nFWER via Permutation \u2014 Jelly Beans & Acne\r\n\r\n\r\n\r\n

\r\n

Illustrating FWER via Permutation

\r\n\r\n

1) Build the global null dataset

\r\n

Make a table T with 1000 rows.
Column 1: Acne = Yes/No (\u2248 50/50 random).
Column 2: Color = one of 10 jelly bean colors (uniform random).

\r\n

Importantly: there is no real relationship \u2014 this simulates the global null where jelly beans don\u2019t cause acne.

\r\n

\r\n\r\n

2) Define your test family

\r\n

For each color (10 total), test independence of acne and that color:

\r\n

\r\n Null hypothesis: H\u2080 : Acne \u27c2 Color
\r\n Alternative: H\u2081 : Acne depends on Color\r\n

\r\n

You obtain 10 p-values per dataset (one per color).

\r\n

\r\n\r\n

3) Permutation (simulate the null distribution)

\r\n

Scramble the Acne column relative to Color (destroys any structure).
Repeat N times (e.g., N = 1000).
For each permutation:\r\n
- Re-run all 10 tests.
- Record the minimum p-value among the 10.
\r\n

\r\n

\r\n\r\n

4) Connecting to FWER

\r\n

Across the N permutations, estimate how often the minimum p-value \u2264 \u03b1 (say 0.05). That proportion is the empirical FWER:

\r\n

\r\n FWER = P(min p \u2264 \u03b1 | H\u2080 true for all)\r\n

\r\n

Example outcome: If you test 10 colors, you might find that in about 40% of permutations at least one color produces p < 0.05.

\r\n

This aligns with the independence formula:

\r\n

\r\n 1 \u2212 (1 \u2212 0.05)\u00b9\u2070 \u2248 0.40\r\n

\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "8.4", "Name": "Bonferroni for aibnb example", "Details": "\r\n\r\n\r\n\r\n\r\nFWER Illustration with Airbnb Neighborhoods (Permutation-Based)\r\n\r\n\r\n\r\n

\r\n

FWER Illustration with Airbnb Neighborhoods (Permutation-Based)

\r\n

Math is shown with native Unicode symbols (no LaTeX rendering required).

\r\n\r\n

Setup

\r\n

Data: airbnb2023N.csv (n = 35,573 rows).
Columns used: neighbourhood, price (price > 0).
Single planned test: NoHo vs Murray Hill (one-sided: mean(NoHo) > mean(Murray Hill)).
Family for FWER demo: top 10 neighborhoods by mean price among those with \u2265100 listings \u21d2 45 pairwise tests (one-sided, higher-mean > lower-mean).
Significance level per test: \u03b1 = 0.05.

\r\n

\r\n\r\n

How to build table T and scramble (conceptually)

\r\n

Build T

\r\n

Keep only rows with valid positive price.
Keep neighbourhood as the group label.

\r\n\r\n

Single permutation test for NoHo > Murray Hill

\r\n

\r\n Let X be NoHo prices (n\u2081), Y be Murray Hill prices (n\u2082).
\r\n Observed statistic: \u0394 = x\u0304 \u2212 \u0233.\r\n

\r\n

Permutation: pool prices [X, Y]; shuffle; split back to sizes (n\u2081, n\u2082); compute \u0394*.
Repeat N times; p(one-sided) = fraction of shuffles with \u0394* \u2265 \u0394.

\r\n\r\n

FWER illustration (family of m tests)

\r\n

Choose a family (here: m = 45 pairs from the top 10 neighborhoods with \u2265100 listings).
Global null via scrambling: shuffle the entire price column across all rows (breaks association with neighborhoods; preserves group sizes).
On each scrambled table, compute one-sided p-values for all 45 pairs (Welch t or similar).
Record the minimum p across the 45 on that scramble.
Repeat N times; empirical FWER \u2248 fraction of scrambles where min-p \u2264 0.05.

\r\n

\r\n\r\n

Results from your file

\r\n

NoHo vs Murray Hill (one-sided permutation test)

\r\n

n(NoHo) = 55, n(Murray Hill) = 361
Observed mean difference: +34.85 (NoHo higher)
p(one-sided, N = 1000 permutations) = 0.042

\r\n

Interpretation: As a single, pre-specified test, this is significant at \u03b1 = 0.05. (If you had scanned many neighborhoods first, you\u2019d need multiple-testing control.)

\r\n

FWER demo with a family of 45 pairwise tests

\r\n

Family: Top 10 neighborhoods by mean price with \u2265100 listings
\r\n West Village, Murray Hill, Greenwich Village, Chelsea, Gramercy, Midtown, Kips Bay, Financial District, Tribeca, SoHo\r\n
Direction per pair: higher-mean > lower-mean (one-sided)
Empirical FWER (N = 300 global scrambles) = 0.687
Theoretical independence bound:\r\n
1 \u2212 (1 \u2212 0.05)\u2074\u2075 \u2248 0.901
\r\n Real data correlations make empirical FWER lower than the independence bound, but it\u2019s still high.\r\n

\r\n

Observed dataset\u2019s most extreme pair (one-sided p): West Village > Midtown, p \u2248 0.00022 (unadjusted)

\r\n

Bonferroni for the family: m = 45 \u21d2 \u03b1/m = 0.00111. Any pair with p \u2264 0.00111 would survive Bonferroni.

\r\n

\r\n\r\n

Why this illustrates FWER

\r\n

On scrambled (null) data, at least one pair is \u201csignificant\u201d at 0.05 in about 69% of full experiments \u2014 even when nothing is real. That\u2019s the family-wise error rate: P(\u22651 false positive in the family).

\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "8.5", "Name": "Video and narrative", "Details": "\n Video of narrated slides\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "9", "Name": "Chi-square - Independence", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "9.1", "Name": "Language of Data - Independence Hypothesis", "Details": "\r\n\r\n\r\n\r\nLanguage of Data \u2014 Chi-Square and Independence Test\r\n\r\n\r\n\r\n\r\n

\r\n

Language of Data \u2014 Chi-Square and Independence Test

\r\n

Goal: Use these words when vibing about relationships between categorical variables. \r\n Work with counts and associations.

\r\n

\r\n\r\n

\r\n

Core Terms

\r\n

Independence
Dependence
Contingency Table
Observed Counts
Expected Counts
Chi-Square Statistic
Degrees of Freedom
P-Value
Association
Relationship Strength

\r\n\r\n

Example Vibes (Chi-Square and Independence)

\r\n

Is music type independent of the day of the week in the Imprezka dataset?
Are majors independent of the grade received in the Moody dataset?
Is apartment location independent of whether rent is above $500?
Is attendance level dependent on the grade earned?
Are gender and car brand independent in the Cars dataset?
Is weather independent of whether the party rating is \u201cExcellent\u201d?
Are neighborhood and room type independent in Airbnb listings?
Check whether auction house and artist gender are independent using a chi-square test.
Are club choice and food type independent in the PartyMega dataset?
Are day of the week and DJ identity independent in the Imprezka dataset?

\r\n\r\n

\r\nChi-Square reasoning focuses on independence vs dependence of categories. \r\nEach vibe asks whether two categorical attributes show a real association beyond random chance.\r\n

\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "9.2", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Chi-Square Test of Independence\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Chi-Square Test of Independence

\r\n \r\n\r\n

\r\n

1. What is the main question the chi-square test of independence answers?

\r\n

A) Are two numerical variables linearly related?
B) Are the means of two groups different?
C) Are two categorical variables independent or related?
D) Is a sample mean equal to a known population mean?

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

2. Which of the following is a typical example for chi-square test of independence from the slides?

\r\n

A) Does quiz score depend on hours of sleep (both numeric)?
B) Does support for a candidate depend on voter\u2019s age group and ethnic background?
C) Is the average income different between two cities?
D) Are midterm scores normally distributed?

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

3. What is the null hypothesis in a chi-square test of independence?

\r\n

A) The two variables are perfectly correlated.
B) The two variables are independent (no relationship).
C) The two variables have equal means.
D) The sample size is large enough.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. In the \u201conline vs in-person vs hybrid\u201d example, which question are we asking?

\r\n

A) Does GPA depend on quiz scores?
B) Do students prefer hybrid only?
C) Do teaching preferences depend on the school (SAS vs Engineering)?
D) Are online classes better than in-person classes?

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

5. In a contingency table for \u201cSchool (SAS/Engineering)\u201d vs \u201cClass preference (Online/In-person/Hybrid)\u201d, each cell contains:

\r\n

A) A probability between 0 and 1
B) The difference between observed and expected counts
C) The number of students in that combination of school and preference
D) The chi-square value for that cell

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

6. What does the \u201cexpected count\u201d in a contingency table represent under the null hypothesis?

\r\n

A) The count we actually observe in the data
B) The count we would expect if the two variables were independent
C) The maximum possible count in that cell
D) The average of the row and column totals

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

7. The chi-square statistic measures:

\r\n

A) The sum of (observed \u2212 expected)
B) The sum of (observed \u2212 expected)\u00b2
C) The sum of (observed \u2212 expected)\u00b2 / expected over all cells
D) The difference between the largest and smallest cell count

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

8. According to the slides, a larger chi-square value indicates:

\r\n

A) Observed and expected tables are very similar (close distance)
B) Observed and expected tables are far apart (large distance)
C) The sample size must be very small
D) The variables are definitely independent

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

9. Which of the following is a correct interpretation if we reject the null hypothesis in a chi-square test of independence?

\r\n

A) The two variables are independent.
B) There is evidence that the two variables are dependent (related).
C) The expected counts were calculated incorrectly.
D) The row totals and column totals are equal.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

10. Which question from the slides is most directly answered by a chi-square test of independence on Data 101 grades?

\r\n

A) Is the average grade in Data 101 higher than in another course?
B) Does grade distribution in Data 101 depend on major?
C) Are Data 101 grades normally distributed?
D) Is the median grade equal to B?

\r\n

Answer: B

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "10", "Name": "Bayesian reasoning", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "10.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Bayesian Reasoning: It\u2019s Not as Bad as You Think\r\n \r\n\r\n\r\n\r\n

Bayesian Reasoning: \u201cIt\u2019s Not as Bad as You Think\u201d

\r\n\r\n

\r\n Bayesian reasoning is a way of thinking that helps us update our beliefs when we\r\n see new evidence. It is fundamental in data science, but it is also a tool for\r\n everyday life \u2013 especially for calming irrational pessimists who always jump to\r\n the worst-case scenario. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Beliefs, Observations, and Updates

\r\n\r\n

\r\n Bayesian reasoning always has three parts:\r\n

\r\n\r\n

Prior beliefs \u2013 what you believe before seeing new data.
Observation \u2013 something you see or measure.
Posterior beliefs \u2013 your updated belief after combining prior and observation.

\r\n\r\n

\r\n We often phrase this as a rule: if Observation, then Belief becomes more or less\r\n likely. The observation is something we know (test result, symptom, event), and\r\n the belief is something we do not yet know but care about (having covid, being guilty,\r\n a model being correct).\r\n

\r\n\r\n

Covid Example: Common Sense vs. Bayesian

\r\n\r\n

\r\n Consider the belief: \u201cI have covid.\u201d You take a test\r\n and the observation is: \u201cTested positive.\u201d\r\n

\r\n\r\n

\r\n The \u201ccommon sense\u201d reaction is panic: \u201cOMG, I have covid!\u201d But the Bayesian\r\n reaction is slower and calmer: \u201cNot so fast. What were the prior odds of me having\r\n covid, and how good is this test?\u201d\r\n

\r\n\r\n

Odds, Priors, and Test Quality

\r\n\r\n

\r\n Instead of only thinking in terms of probability, Bayes often uses\r\n odds, defined as:\r\n

\r\n\r\n

\r\n odds = p / (1 \u2212 p)\r\n

\r\n\r\n

\r\n First, we ask about the prior odds of having covid.\r\n Given your symptoms and situation, are the odds:\r\n

\r\n\r\n

1:10 (about 9% probability)?
1:100 (about 1%)?
1:1000 (0.1%)?

\r\n\r\n

\r\n Next, we describe the quality of the test:\r\n

\r\n\r\n

True Positive Rate:\r\n If you really have covid, how likely is the test to say YES?
False Positive Rate:\r\n If you do not have covid, how likely is the test to incorrectly say YES?

\r\n\r\n

\r\n Suppose the test is 99% true positive and 1% false positive. It sounds very good,\r\n but we still need to combine this with the prior odds.\r\n

\r\n\r\n

Bayes Theorem in Odds Form

\r\n\r\n

\r\n Bayes in odds form:
\r\n Posterior odds = Likelihood ratio \u00d7 Prior odds\r\n

\r\n\r\n

\r\n The pieces are:\r\n

\r\n\r\n

Prior odds:\r\n P(B) / P(~B), where B is the belief (\u201cI have covid\u201d).
Likelihood ratio:\r\n P(O|B) / P(O|~B) \u2013 the chance of seeing the observation if the belief is true\r\n divided by the chance of seeing it if the belief is false. For a positive test,\r\n this is True Positive / False Positive.
Posterior odds:\r\n P(B|O) / P(~B|O) \u2013 the updated odds of the belief after seeing the observation.

\r\n\r\n

\r\n In more standard notation:\r\n

\r\n\r\n

\r\n Prior odds = P(B) / P(~B)
\r\n Likelihood ratio = P(O|B) / P(O|~B)
\r\n Posterior odds = P(B|O) / P(~B|O)\r\n

\r\n\r\n

Covid Calculation: Not as Bad as It Looks

\r\n\r\n

\r\n Imagine your prior odds of having covid are 1:99 (about 1% probability).\r\n The test has:\r\n

\r\n\r\n

True positive rate = 99%
False positive rate = 1%

\r\n\r\n

\r\n The likelihood ratio is:\r\n

\r\n\r\n

\r\n LR = True positive / False positive = 99 / 1 = 99.\r\n

\r\n\r\n

\r\n Now apply Bayes:\r\n

\r\n\r\n

\r\n Posterior odds = 99 \u00d7 (1 / 99) = 1 / 1.\r\n

\r\n\r\n

\r\n So after a positive test, your odds are 1:1 \u2013 in other words, about a 50% chance.\r\n That is serious, but it is much better than the \u201cOMG 99%!\u201d feeling you\r\n might have had just from seeing \u201cpositive test.\u201d\r\n

\r\n\r\n

\r\n Bayesian reasoning often shows that things are not as bad as they first appear.\r\n The combination of low prior probability and imperfect tests means that many\r\n positive results are not automatic death sentences.\r\n

\r\n\r\n

Three Ways We Use Bayes in Practice

\r\n\r\n

\r\n The slides highlight three typical scenarios for computing Bayesian odds:\r\n

\r\n\r\n

\r\n Word problems: You are given all relevant\r\n probabilities in the text (e.g., prior probability, true positive, false positive).\r\n You plug them into Bayes\u2019 theorem directly \u2013 like the covid example.\r\n
\r\n From a data frame: You have raw data with\r\n individual cases (rows). You estimate probabilities directly from the data:\r\n how many had covid, how many tested positive, etc., then use these as inputs\r\n to Bayes.\r\n
\r\n From a contingency table: You have aggregate\r\n counts in a table (e.g., positive/negative vs. covid/no-covid). You derive\r\n probabilities from the counts and apply Bayes.\r\n

\r\n\r\n

Why Bayesian Reasoning Matters

\r\n\r\n

\r\n Bayesian reasoning is a powerful antidote to anxiety and overreaction. It forces us\r\n to:\r\n

\r\n\r\n

Say clearly what we believed before the evidence (prior).
Describe how good or bad our measurement is (likelihood ratio).
Compute a calm, quantitative update (posterior odds).

\r\n\r\n

\r\n In data science, this mindset is everywhere: in spam filters, medical diagnosis,\r\n A/B testing, and model comparison. In everyday life, it is the quiet voice saying:\r\n \u201cWait. Let\u2019s look at the numbers before we panic.\u201d\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "10.11", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Bayesian Reasoning and Covid Example\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Bayesian Reasoning: \u201cIt\u2019s Not as Bad as You Think\u201d

\r\n \r\n\r\n

\r\n

1. In the slides, Bayesian reasoning is described as a process that moves from:

\r\n

A) Data \u2192 Code \u2192 Plots
B) Prior beliefs \u2192 Observations \u2192 Posterior beliefs
C) Observations \u2192 Hypotheses \u2192 Data
D) Random guesses \u2192 Simulation \u2192 Conclusion

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. In the covid example, what is the belief and what is the observation?

\r\n

A) Belief: \u201cTest is accurate\u201d; Observation: \u201cI have covid\u201d
B) Belief: \u201cI have covid\u201d; Observation: \u201cTested positive\u201d
C) Belief: \u201cI have covid\u201d; Observation: \u201cCovid exists\u201d
D) Belief: \u201cTested positive\u201d; Observation: \u201cI have covid\u201d

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

3. How are odds defined in the slides?

\r\n

A) odds = 1 \u2212 p
B) odds = 1 / p
C) odds = p / (1 \u2212 p)
D) odds = p \u00d7 (1 \u2212 p)

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

4. Which quantities describe the \u201cquality of the test\u201d in the covid example?

\r\n

A) Prior probability and posterior probability
B) True positive rate and false positive rate
C) Prior odds and posterior odds
D) Mean and standard deviation

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

5. The slides define the likelihood ratio as:

\r\n

A) P(B)
B) P(O)
C) True positive / False negative
D) True positive / False positive

\r\n

Answer: D

\r\n

\r\n\r\n

\r\n

6. In odds form, Bayes\u2019 theorem is written in the slides as:

\r\n

A) Posterior odds = Prior odds + Likelihood ratio
B) Posterior odds = Prior odds \u2212 Likelihood ratio
C) Posterior odds = Likelihood ratio \u00d7 Prior odds
D) Posterior odds = Likelihood ratio \u00f7 Prior odds

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

7. In notation, the prior odds for the belief \u201cI have covid\u201d are given by:

\r\n

A) P(O|B) / P(O)
B) P(B|O) / P(~B|O)
C) P(B) / P(~B)
D) P(O|~B) / P(O|B)

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

8. In the numeric covid example, prior odds are 1:99 (probability 0.01) and the test has 99% true positive and 1% false positive. What is the likelihood ratio?

\r\n

A) 1 / 99
B) 99 / 1 = 99
C) 99 / 100
D) 1 / 100

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

9. Using those numbers, what are the posterior odds of having covid after a positive test, according to the slides?

\r\n

A) 1 : 99 (still about 1%)
B) 99 : 1 (about 99%)
C) 1 : 1 (about 50%)
D) 9 : 1 (about 90%)

\r\n

Answer: C (Posterior odds = 99 \u00d7 (1/99) = 1:1.)

\r\n

\r\n\r\n

\r\n

10. The slides list three scenarios for computing Bayesian odds. Which of the following is not one of them?

\r\n

A) From a word problem that gives all needed probabilities (no raw data)
B) Directly from a data frame (raw data)
C) From a contingency table (aggregate data)
D) From a neural network without any probabilities

\r\n

Answer: D

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "10.1", "Name": "Convo-on Bayesian Reasoning", "Details": " \n Bayesian Reasoning\n $\"ChatGPT$ \n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "10.2", "Name": "Language of Data: Bayesian Reasoning", "Details": "\r\n\r\n\r\n\r\nLanguage of Data \u2014 Bayesian Reasoning (Odds Version)\r\n\r\n\r\n\r\n\r\n

\r\n

Language of Data \u2014 Bayesian Reasoning (Odds Version)

\r\n

Goal: Use these words when vibing about beliefs and observations in data. \r\n Work with odds, not probabilities.

\r\n

\r\n\r\n

\r\n

Core Terms

\r\n

Belief
Observation
Prior Odds
Posterior Odds
Likelihood Ratio
True Positive
False Positive

\r\n\r\n

Example Vibes (Odds Version)

\r\n

What are the odds of passing the class if your score is between 50 and 55?
What are the odds if apartment rental is below $200 given that apartment is in Tribeca?
What is the likelihood ratio of apartment rental below $200 given that apartment is in Tribeca?
What are the odds of a female buyer under 30 buying a GMC truck in Chicago?
What are the posterior odds of apartment renting above $500 being on a high floor?
What is observation and what is belief in the task \u201cwhat are posterior odds of apartment renting above $500 being on high floor?\u201d

\r\n\r\n

\r\nBayesian reasoning in odds form:
\r\nPosterior Odds = Prior Odds \u00d7 Likelihood Ratio
\r\nEach vibe identifies belief and observation in a dataset context \u2014 returning computed odds or ratios.\r\n

\r\n

\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "11", "Name": "Prediction ", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "11.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Prediction Models and Decision Trees (rpart)\r\n \r\n\r\n\r\n\r\n

Prediction Models: From Ideas to Decision Trees

\r\n\r\n

\r\n A prediction model in machine learning is a mathematical\r\n recipe that learns from past data and then predicts what will happen for new, unseen\r\n cases. You feed it input variables (features, X),\r\n it learns to approximate the output variable\r\n (target, Y), and after training you can ask it questions about the future. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Three Flavors of Prediction

\r\n\r\n

\r\n Most prediction tasks fall into one of three buckets:\r\n

\r\n\r\n

\r\n Regression models predict continuous numbers:\r\n auction prices, apartment rent, student GPA, time on site.\r\n Examples include linear regression and decision tree regression.\r\n
\r\n Classification models predict categories:\r\n grade A/B/C/D/F, spam vs. not spam, churn vs. stay.\r\n Examples include logistic regression, Naive Bayes, random forests, and neural networks.\r\n
\r\n Probability models output probabilities instead\r\n of hard labels: \u201c70% chance of rain tomorrow,\u201d or \u201c85% chance the student gets at\r\n least a B.\u201d\r\n

\r\n\r\n

\r\n In all cases, we describe each case with features X and ask the model to predict a\r\n Y that is either a number, a category, or a probability distribution.\r\n

\r\n\r\n

How Prediction Models Learn

\r\n\r\n

\r\n The life of a model can be split into three phases:\r\n

\r\n\r\n

\r\n Training phase: The model sees historical data\r\n where both X and the correct Y are known. It adjusts its internal parameters to\r\n reduce prediction error.\r\n
\r\n Evaluation phase: We test the trained model on a\r\n separate dataset it has never seen. This tells us how well it generalizes, not\r\n just memorizes.\r\n
\r\n Prediction phase: We deploy the model in the real\r\n world, where it gets new X\u2019s and must predict Y\u2019s it never saw during training.\r\n

\r\n\r\n

\r\n Example \u2013 Regression: X = apartment size, location, number of rooms;\r\n Y = rent price.
\r\n Example \u2013 Classification: X = age, GPA, attendance;\r\n Y = final grade (A\u2013F).
\r\n Example \u2013 Probability: X = recent weather patterns;\r\n Y = P(rain tomorrow) = 0.85.\r\n

\r\n\r\n

Different Families of Models

\r\n\r\n

\r\n Different models have different personalities:\r\n

\r\n\r\n

Linear / Logistic Regression \u2013 simple, fast, and interpretable.
Decision Trees / Random Forests \u2013 flexible, can capture interactions and non-linearities, and are easy to explain visually.
Neural Networks \u2013 very powerful for images, text, and huge datasets, but often less interpretable.
Naive Bayes \u2013 simple and fast for text or categorical data.
Support Vector Machines (SVMs) \u2013 good at finding separating boundaries between classes.

\r\n\r\n

\r\n In Data 101, one of the workhorses is the decision tree,\r\n implemented in R with the function rpart().\r\n

\r\n\r\n

Decision Trees and rpart

\r\n\r\n

\r\n A decision tree (recursive partitioning) repeatedly splits the dataset into smaller\r\n and smaller groups that are more and more homogeneous in terms of the target Y.\r\n Each split is based on a simple rule like Score >= 84.5 or\r\n Major = \"Economics\". Eventually we reach leaves that correspond to\r\n predictions such as \u201cmostly A\u201d or \u201cmostly F\u201d.\r\n

\r\n\r\n

\r\n In R, a basic tree for the Moody dataset might be built with just one line:\r\n

\r\n\r\n

\r\n library(rpart)
\r\n model <- rpart(Grade ~ Score + Attendance + Major, data = moody)\r\n

\r\n\r\n

\r\n The tree\u2019s text output is an indented structure showing how the data is split:\r\n the root node, splits on Score, further splits on Major,\r\n and leaf nodes marked with *. Each node shows how many observations it\r\n contains and the class distribution inside it. You can read it like a flowchart:\r\n follow the conditions down from the root and you reach a grade prediction.\r\n

\r\n\r\n

\r\n What rpart does:
\r\n 1) Grows a tree by recursively splitting the data to make leaves purer in terms of the target.
\r\n 2) Chooses splits that give the best improvement (e.g., reduce Gini impurity or variance).
\r\n 3) Produces a model object you can inspect with print(),\r\n summary(), plot(), or rpart.plot().\r\n

\r\n\r\n

Controlling Tree Growth: rpart.control()

\r\n\r\n

\r\n Left unchecked, trees love to grow too large and overfit. The\r\n rpart.control() function sets \u201cguardrails\u201d that shape the tree:\r\n

\r\n\r\n

\r\n minsplit: the minimum number of observations that\r\n must be in a node before we even consider splitting it.\r\n Smaller minsplit \u2192 deeper, more complex trees.\r\n
\r\n minbucket: the minimum number of observations\r\n allowed in any terminal (leaf) node. Often about minsplit/3.\r\n
\r\n cp (complexity parameter): the minimum required\r\n improvement in model fit to justify a split. Larger cp shrinks the\r\n tree; smaller cp lets it grow more.\r\n

\r\n\r\n

\r\n A typical usage might look like:\r\n

\r\n\r\n

\r\n ctrl <- rpart.control(minsplit = 20, cp = 0.01, maxdepth = 5)
\r\n model <- rpart(Grade ~ Score + Attendance + Major, data = moody, control = ctrl)\r\n

\r\n\r\n

\r\n First, rpart() grows a tree; then these controls plus cross-validation\r\n help decide how much to prune it back to avoid overfitting.\r\n

\r\n\r\n

From Model to Predictions: predict()

\r\n\r\n

\r\n Once the model is trained, we want to use it. In both R and Python, the function\r\n predict() is the standard way to apply a model to new data:\r\n

\r\n\r\n

For classification: it returns predicted class labels (e.g., A, B, C, D, F).
Optionally: it can return class probabilities (e.g., 0.7 for A, 0.2 for B, 0.1 for C).
For regression: it returns predicted numeric values (e.g., price, score).

\r\n\r\n

\r\n In a typical workflow, you train your tree on Train, evaluate on\r\n Test using predict(), measure accuracy (or RMSE for\r\n regression), and then deploy the model to make predictions on real-world cases.\r\n

\r\n\r\n

Big Picture

\r\n\r\n

\r\n Prediction models are the engines of modern data science: they turn raw features into\r\n informed guesses about outcomes we care about. Decision trees and\r\n rpart give us a friendly, visual, and interpretable way to do this in\r\n Data 101. Once you understand the flow \u2013 train \u2192 control complexity \u2192 predict \u2013 you\r\n can plug in different models, compare their performance, and choose the one that best\r\n balances accuracy and interpretability.\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "11.11", "Name": "Quiz Example", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Prediction Models and rpart\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Prediction Models and Decision Trees (rpart)

\r\n \r\n\r\n

\r\n

1. What is a prediction model in machine learning?

\r\n

A) A random number generator
B) A model that memorizes training data but cannot handle new data
C) A mathematical/statistical model trained on data to predict unseen cases
D) A visualization tool for plotting data

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

2. In the slides, the input variables and output variable are also called:

\r\n

A) Input = Y, Output = X
B) Input = features/X, Output = target/Y
C) Input = labels, Output = noise
D) Input = outcomes, Output = predictors

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

3. Which of the following is a regression problem according to the slides?

\r\n

A) Predicting whether an email is spam or not
B) Predicting the probability of rain tomorrow
C) Predicting a student\u2019s GPA
D) Predicting whether a student will get an A or not

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

4. During which phase is the model evaluated on data it has not seen before to check accuracy?

\r\n

A) Training phase
B) Evaluation phase
C) Prediction (deployment) phase
D) Feature engineering phase

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

5. Which of the following is correctly matched with its type from the slides?

\r\n

A) Logistic Regression \u2013 Regression model
B) Naive Bayes \u2013 Classification model
C) Random Forest \u2013 Only regression, never classification
D) Linear Regression \u2013 Probability model

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

6. In the rpart example, what is being predicted?

\r\n

A) Score from Grade, Attendance, and Major
B) Grade from Score, Attendance, and Major
C) Major from Score, Attendance, and Grade
D) Attendance from Score and Grade

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

7. What does the minsplit parameter in rpart.control() control?

\r\n

A) Minimum tree depth allowed
B) Minimum number of observations required in a node before trying a split
C) Minimum number of leaves in the whole tree
D) Minimum improvement in fit required for any split

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

8. What is the role of the complexity parameter cp in rpart.control()?

\r\n

A) It sets the maximum number of trees in a forest
B) It controls the learning rate of the tree
C) It is the minimum required improvement in model fit to allow a split
D) It forces all leaves to have equal size

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

9. Which of the following best describes what predict() does in R for models like rpart or lm?

\r\n

A) It retrains the model on new data
B) It computes feature importance scores only
C) It applies a trained model to new data to produce predicted classes, values, or probabilities
D) It randomly shuffles the dataset

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

10. Which model family is described in the slides as \u201csimple, interpretable\u201d compared to others?

\r\n

A) Neural Networks
B) Random Forests
C) Linear Regression / Logistic Regression
D) Support Vector Machines

\r\n

Answer: C

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "12", "Name": "Linear Regression", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "12.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Linear Regression, MSE, and R\u00b2\r\n \r\n\r\n\r\n\r\n

Linear Regression, Mean Squared Error, and R\u00b2

\r\n\r\n

\r\n Linear regression is about capturing a linear relationship\r\n between one or more input attributes and a numeric outcome. We assume that the target\r\n variable can be approximated by a straight-line combination of features. For a single\r\n predictor, this looks like an ordinary line on a plot: Y on the vertical axis and X\r\n on the horizontal axis. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Linear Modeling: From Line to Equation

\r\n\r\n

\r\n Imagine that the X-axis represents SCORE and the\r\n Y-axis represents GRADE. A blue regression line\r\n drawn through the cloud of points shows how grades tend to change as scores increase.\r\n The model has the familiar form:\r\n

\r\n\r\n

\r\n GRADE = slope \u00d7 SCORE + intercept\r\n

\r\n\r\n

\r\n In a multivariate setting, we extend this idea to several predictors. For the Moody\r\n dataset with numeric components of the course, we can model\r\n ClassScore using midterm, project, and final exam.\r\n

\r\n\r\n

Linear Modeling in R

\r\n\r\n

\r\n In R, a linear model is fit with lm(). For example:\r\n

\r\n\r\n

\r\n lm(formula = ClassScore ~ ., data = moodyNUMTr)\r\n

\r\n\r\n

\r\n Here, ClassScore ~ . means \u201cpredict ClassScore from all other numeric\r\n columns in moodyNUMTr.\u201d The model might output coefficients like:\r\n

\r\n\r\n

\r\n

\r\n ClassScore = 0.08079 + 0.20036 \u00d7 Midterm + 0.49707 \u00d7 Project + 0.29977 \u00d7 FinalExam\r\n

\r\n

\r\n\r\n

\r\n Each coefficient tells us how much the predicted score changes when that component\r\n increases by one point, holding the others fixed. The intercept is the baseline when\r\n Midterm, Project, and FinalExam are zero.\r\n

\r\n\r\n

Mean Squared Error (MSE)

\r\n\r\n

\r\n To judge how good the model is, we compare its predictions to actual outcomes on a\r\n test set. One natural measure is the\r\n Mean Squared Error (MSE):\r\n

\r\n\r\n

\r\n MSE = mean( (predicted \u2212 actual)\u00b2 )\r\n

\r\n\r\n

\r\n In R, once we have a trained model, we can do:\r\n

\r\n\r\n

\r\n pred <- predict(train, newdata = moodyNUMTs)
\r\n mean((pred - moodyNUMTs$ClassScore)^2)\r\n

\r\n\r\n

\r\n This gives an MSE value, for example 1.53. MSE is always non-negative, and\r\n values closer to zero are better. Because we square\r\n errors, large mistakes are punished heavily: a few very bad predictions can dominate\r\n the average. :contentReference[oaicite:1]{index=1}\r\n

\r\n\r\n

Computing MSE Manually

\r\n\r\n

\r\n The steps for computing MSE are simple:\r\n

\r\n\r\n

For each data point, compute the error: error = predicted \u2212 actual.
Square the error: error\u00b2.
Take the mean of all squared errors.

\r\n\r\n

\r\n Because of the squaring, MSE can look \u201cdepressing\u201d when the target variable is on a\r\n large scale: for annual salary in dollars, squared errors can be in the millions.\r\n Taking the square root gives RMSE, which is on the\r\n same scale as the target and feels more intuitive.\r\n

\r\n\r\n

MSE in a Prediction Challenge

\r\n\r\n

\r\n Consider a prediction challenge where students model\r\n Salary using university features:\r\n <University, GPA, GraduationYear, Private, Location, Tuition, Salary>.\r\n Several submissions show how MSE behaves:\r\n

\r\n\r\n

\r\n Submission 1: MSE \u2248 4,282,868. One worst-case tuple has a huge squared error,\r\n yet there are 5,000+ perfect predictions with zero error.\r\n
\r\n Submission 2: MSE \u2248 52,026,524. Again, one extremely bad prediction dominates\r\n the squared error distribution.\r\n
\r\n Submission 4: MSE \u2248 18,210,198,576 when the model predicts salary = 0 for some\r\n cases, creating catastrophic squared errors, despite 7,000 perfect predictions.\r\n

\r\n\r\n

\r\n These examples show that MSE is a \u201cpunishing\u201d measure: it exposes even a small number\r\n of disastrous predictions. The distribution of squared errors often looks like a\r\n power law: many tiny errors and a few huge ones that\r\n dominate the average. :contentReference[oaicite:2]{index=2}\r\n

\r\n\r\n

R\u00b2: Coefficient of Determination

\r\n\r\n

\r\n While MSE measures the size of errors, R\u00b2 measures how much of the variation\r\n in the target the model explains. It is defined as:\r\n

\r\n\r\n

\r\n R\u00b2 = 1 \u2212 RSS / TSS\r\n

\r\n\r\n

\r\n where:\r\n

\r\n\r\n

TSS (Total Sum of Squares) = total variation in actual values.
RSS (Residual Sum of Squares) = unexplained variation (sum of squared residuals).

\r\n\r\n

\r\n Some intuition:\r\n

\r\n\r\n

R\u00b2 = 1: perfect model, RSS = 0, explains all variation.
R\u00b2 near 0: model is no better than predicting the mean every time.
R\u00b2 < 0: model is actually worse than the baseline mean predictor.

\r\n\r\n

\r\n \r\n \r\n \r\n \r\n \r\n \r\n

Model	RSS	TSS	R\u00b2	Fit Quality
Perfect	0	100	1.00	Perfect fit
Good	20	100	0.80	Explains 80% of variation
Weak	90	100	0.10	Explains 10% of variation
Bad	110	100	\u22120.10	Worse than baseline

\r\n

\r\n\r\n

Scale and the Need to Report Both

\r\n\r\n

\r\n A crucial point is that R\u00b2 is relative. It measures\r\n the proportion of variance explained, not the absolute size of errors. You can have:\r\n

\r\n\r\n

Very large MSE (because salaries are in hundreds of thousands), yet R\u00b2 close to 1.
Small MSE (for example, IMDB scores between 1 and 10), but also R\u00b2 close to 1.

\r\n\r\n

\r\n In both cases, the model explains most of the variation, but the numerical size of the\r\n errors is very different because the scales are different.\r\n

\r\n\r\n

\r\n Rule of thumb:
\r\n Use R\u00b2 to understand how well patterns are captured,\r\n and MSE or RMSE to understand how big the errors are\r\n in the original units. Always report both for a complete picture of model performance.\r\n :contentReference[oaicite:3]{index=3}\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "12.2", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Linear Regression, MSE, and R\u00b2\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Linear Regression, MSE, and R\u00b2

\r\n\r\n

\r\n

1. In the slides, \u201clinear modeling\u201d primarily means:

\r\n

A) Fitting any non-linear function to the data
B) Fitting a linear relationship between one or more attributes and a numeric target
C) Using decision trees to predict categorical variables
D) Clustering data points into groups

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. If the Y-axis is GRADE and the X-axis is SCORE, the regression line shown in the slides has the general form:

\r\n

A) GRADE = SCORE \u00d7 SCORE + intercept
B) SCORE = slope \u00d7 GRADE + intercept
C) GRADE = slope \u00d7 SCORE + intercept
D) GRADE = intercept \u2212 SCORE

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. For the Moody dataset, the linear model reported in R is:

\r\n

A) ClassScore = Midterm + Project + FinalExam
B) ClassScore = 0.08079 + 0.20036\u00b7Midterm + 0.49707\u00b7Project + 0.29977\u00b7FinalExam
C) ClassScore = Midterm \u2212 Project + FinalExam
D) ClassScore = 0.08079\u00b7Midterm + Project + FinalExam

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. Which of the following is the correct R code (from the slides) to compute MSE on the test set?

\r\n

A) mean(pred - moodyNUMTs$ClassScore)
B) mean((pred - moodyNUMTs$ClassScore)^2)
C) sum(pred - moodyNUMTs$ClassScore)
D) sum((pred - moodyNUMTs$ClassScore)^2)

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

5. Which statement about Mean Squared Error (MSE) is correct according to the slides?

\r\n

A) MSE can be negative for a very bad model
B) MSE is always non-negative and values closer to zero are better
C) MSE must always be less than 1
D) MSE ignores large errors and focuses on small ones

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

6. What are the correct steps to compute MSE manually, as listed in the slides?

\r\n

A) error = actual \u2212 predicted; sum errors; divide by N
B) error = predicted \u2212 actual; square error; sum squares
C) error = predicted \u2212 actual; square error; take mean of all squared errors
D) error = actual \u2212 predicted; take absolute value; take mean

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

7. In the prediction challenge examples, why did some submissions have extremely large MSE despite thousands of perfect predictions?

\r\n

A) Because MSE counts only the number of correct predictions
B) Because MSE is dominated by a few huge squared errors from very bad predictions
C) Because R miscalculated the MSE
D) Because perfect predictions increase MSE

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

8. R\u00b2 (coefficient of determination) is defined in the slides as:

\r\n

A) R\u00b2 = RSS / TSS
B) R\u00b2 = TSS / RSS
C) R\u00b2 = 1 \u2212 RSS / TSS
D) R\u00b2 = 1 \u2212 TSS / RSS

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

9. What does it mean if a regression model has R\u00b2 < 0, according to the slides?

\r\n

A) The model is perfect
B) The model is as good as predicting the mean
C) The model is worse than always predicting the mean
D) R\u00b2 cannot be less than 0

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

10. Which combination best summarizes the roles of R\u00b2 and MSE in the slides?

\r\n

A) R\u00b2 measures error size; MSE measures explained variance
B) R\u00b2 and MSE are identical, just scaled differently
C) R\u00b2 shows how much variation is explained; MSE shows how big the prediction errors are in the original units
D) R\u00b2 is only for classification; MSE is only for clustering

\r\n

Answer: C

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "12.3", "Name": "Crossvalidation", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "12.4", "Name": "Example - former prediction challenge", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "13", "Name": "Combing Multiple Prediction Models: Boosting, Stacking and Bagging", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "13.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Combining Prediction Models: Boosting, Bagging, Bootstrapping, Stacking\r\n \r\n\r\n\r\n\r\n

Combining Prediction Models: Boosting, Bagging, Bootstrapping, Stacking

\r\n\r\n

\r\n Modern prediction systems rarely rely on a single model. Instead, they often use\r\n ensembles \u2013 clever combinations of many simple models\r\n that, together, perform much better than any one of them alone. This chapter walks\r\n through four key ideas: boosting,\r\n bootstrapping, bagging\r\n (including Random Forests), and model stacking. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Boosting: Fixing Yesterday\u2019s Mistakes

\r\n\r\n

\r\n Boosting starts with a weak learner, often a small\r\n decision tree that is only slightly better than random guessing. We train it on the\r\n data, then look at what it gets wrong \u2013 the residuals in regression, or the\r\n misclassified examples in classification.\r\n

\r\n\r\n

\r\n The key idea is to focus on the hard cases:\r\n

\r\n\r\n

Make an initial, simple prediction (for example, \u201cevery house costs $300,000\u201d).
Compute the errors: residual = actual \u2212 predicted.
Train a shallow tree to predict these residuals \u2013 this tree learns where we are under- or over-predicting.
Update the prediction by adding a fraction of the tree\u2019s output:\r\n
\r\n new_pred = old_pred + learning_rate \u00d7 tree_prediction.\r\n
Repeat: recompute residuals, fit a new tree, update predictions.

\r\n\r\n

\r\n After many rounds, we sum all the tiny corrections from each tree to get a strong\r\n model. Boosting trains models sequentially, each one\r\n concentrating on the remaining errors. It can dramatically reduce both bias and\r\n variance, but without regularization (learning rate, tree depth limits, early\r\n stopping) it may overfit. Famous implementations include AdaBoost, Gradient Boosting,\r\n XGBoost, LightGBM, and CatBoost.\r\n

\r\n\r\n

Bootstrapping: Making New Data from Old Data

\r\n\r\n

\r\n Bootstrapping is a resampling technique. Given a dataset of size n, we create\r\n a bootstrap sample by drawing n rows at random\r\n with replacement. Some rows appear multiple times, others are left out.\r\n

\r\n\r\n

\r\n If our original data is x = [2, 4, 6, 8, 10], a bootstrap sample might\r\n look like [4, 4, 6, 10, 2] or [6, 8, 8, 10, 10]. Every time\r\n we resample, we get a slightly different pseudo-dataset.\r\n

\r\n\r\n

\r\n Bootstrapping is used to:\r\n

Estimate uncertainty (e.g., confidence intervals) by looking at the variability of a statistic across many bootstrap samples.
Study the stability of estimators like means or regression coefficients.
Power ensemble models such as Random Forests, where each tree is trained on a different bootstrap sample.

\r\n

\r\n\r\n

\r\n In R, a simple bootstrap for the mean looks like:\r\n

\r\n\r\n

\r\n x <- c(2,4,6,8,10)
\r\n means <- replicate(1000, mean(sample(x, replace = TRUE)))
\r\n hist(means)\r\n

\r\n\r\n

Bagging: Bootstrap + Aggregation

\r\n\r\n

\r\n Bagging (Bootstrap AGGregatING) takes bootstrapping\r\n one step further. Instead of computing a single statistic on each bootstrap sample,\r\n we train a model on each sample and then average their predictions.\r\n

\r\n\r\n

\r\n The workflow is:\r\n

\r\n\r\n

Draw B bootstrap samples from the training data.
Train a model (often a decision tree) on each sample.
For regression, average the B predictions; for classification, use majority vote.

\r\n\r\n

\r\n Bagging mainly reduces variance. Each model sees a\r\n slightly different version of the data, so their errors are less correlated. When we\r\n average them, the random noise tends to cancel out, making the ensemble more stable\r\n than any single tree.\r\n

\r\n\r\n

Random Forests: Decorrelated Bagged Trees

\r\n\r\n

\r\n Random Forest improves on bagging by decorrelating the\r\n trees even more. It does two things for each tree:\r\n

\r\n\r\n

Uses a bootstrap sample of the rows (like bagging).
At each split, considers only a random subset of features (for example, \u221ap out of p).

\r\n\r\n

\r\n This prevents a single strong predictor from dominating all trees, encourages diverse\r\n split structures, and produces a more powerful ensemble.\r\n

\r\n\r\n

\r\n library(randomForest)
\r\n rf_model <- randomForest(Y ~ ., data = df, ntree = 500, mtry = sqrt(ncol(df)))\r\n

\r\n\r\n

\r\n This code trains a Random Forest that automatically bootstraps the data for each tree,\r\n samples features at each split, and aggregates predictions over hundreds of trees.\r\n

\r\n\r\n

Model Stacking: A Meta-Learner on Top

\r\n\r\n

\r\n While boosting is sequential and bagging/Random Forests are parallel ensembles of\r\n similar models, model stacking blends potentially\r\n very different models using a meta-model on top.\r\n

\r\n\r\n

\r\n The idea:\r\n

\r\n\r\n

Train several base models on the same dataset (e.g., linear regression, random forest, gradient boosting).
For each case, collect their predictions: Pred1, Pred2, \u2026
Add these predictions as new columns to your dataset, alongside the original features.
Train a meta-learner (often an rpart tree) on this expanded dataset to predict Y.

\r\n\r\n

\r\n In R, this might look like:\r\n

\r\n\r\n

\r\n df$Pred1 <- predict(model1, df)
\r\n df$Pred2 <- predict(model2, df)
\r\n df$Pred3 <- predict(model3, df)
\r\n meta_tree <- rpart(Y ~ ., data = df)\r\n

\r\n\r\n

\r\n Each Pred_i acts as a meta-feature\r\n summarizing what one model \u201cbelieves.\u201d The meta-tree learns where each model is\r\n trustworthy. For example, it might learn that Model 1 works best for low-income\r\n customers, while Model 2 is better for high-income ones. Stacking is parallel (all\r\n base models are trained independently) and does not use residuals or\r\n re-weighting like boosting; instead, it focuses on combining existing opinions.\r\n

\r\n\r\n

Why Ensembles Work

\r\n\r\n

\r\n All these methods share a core philosophy: instead of betting everything on one\r\n model, we let many models contribute and then combine them. Boosting fixes\r\n yesterday\u2019s mistakes, bagging and Random Forests average out noise via resampling,\r\n and stacking lets a meta-learner orchestrate a \u201cbest-of\u201d combination. When used\r\n carefully, ensembles can deliver dramatic gains in accuracy and stability over any\r\n single model on its own.\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "13.2", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Boosting, Bagging, Bootstrapping, Stacking\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Boosting, Bagging, Bootstrapping, and Stacking

\r\n\r\n

\r\n

1. In boosting, how are the models trained?

\r\n

A) All at once on the same dataset
B) Sequentially, each new model focusing on the errors of the previous one
C) Sequentially, but each model ignores the previous models
D) In random order, each on different labels

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. In the boosting workflow, what does the weak learner (e.g., a small tree) fit at each iteration in regression?

\r\n

A) The original target values
B) Random noise
C) The residuals (Actual \u2013 Predicted)
D) The squared errors

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. What is the role of the learning rate in boosting?

\r\n

A) It controls how many trees are grown in the forest
B) It scales how much of each weak learner\u2019s prediction is added to the overall model
C) It decides which features are used in each split
D) It determines the train\u2013test split ratio

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. A bootstrap sample is best described as:

\r\n

A) A sample without replacement, smaller than the original dataset
B) A random permutation of the original data
C) A sample of the same size, drawn with replacement from the original dataset
D) The subset of points with the largest residuals

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

5. Which of the following is a main purpose of bootstrapping mentioned in the slides?

\r\n

A) To reduce the dimensionality of the feature space
B) To simulate new datasets and estimate uncertainty of statistics
C) To remove outliers permanently from the data
D) To find the optimal number of clusters

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

6. Bagging (Bootstrap Aggregating) is defined as:

\r\n

A) Training one model on the original data and one on a bootstrap sample
B) Drawing many bootstrap samples, training a model on each, and aggregating their predictions
C) Sequentially updating a single model on different bootstrap samples
D) Selecting only the best-performing bootstrap model

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

7. What additional randomness does a Random Forest introduce beyond bagging?

\r\n

A) Randomly dropping training rows at prediction time
B) Randomly changing target values for some trees
C) Randomly selecting a subset of features at each split
D) Randomly reordering the trees at the end

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

8. In the R code shown in the slides, which line trains a Random Forest model?

\r\n

A) rf_model <- rpart(Y ~ ., data = df)
B) rf_model <- randomForest(Y ~ ., data = df, ntree = 500, mtry = sqrt(ncol(df)))
C) rf_model <- glm(Y ~ ., data = df)
D) rf_model <- naiveBayes(Y ~ ., data = df)

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

9. In model stacking as described in the slides, what are Pred1, Pred2, Pred3, etc.?

\r\n

A) The original raw features
B) Random noise features added to stabilize the model
C) Predictions from several base models, added as new columns (meta-features)
D) Residuals used by boosting to fit new trees

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

10. Which statement correctly contrasts boosting with stacking, based on the slides?

\r\n

A) Both are sequential; the difference is only in the choice of weak learner
B) Boosting is parallel, stacking is sequential
C) Boosting is sequential and uses residuals/errors; stacking is parallel and combines model outputs as meta-features
D) Stacking never uses decision trees, boosting always does

\r\n

Answer: C

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "14", "Name": "Naive Bayes and Logiistic Regression", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "14.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Other Prediction Models: Logistic Regression and Naive Bayes\r\n \r\n\r\n\r\n\r\n

Other Prediction Models: Logistic Regression and Naive Bayes

\r\n\r\n

\r\n Decision trees (rpart) and random forests are not the only tools for\r\n prediction. Two classically important models in data science are\r\n logistic regression and\r\n Naive Bayes. Both are used mainly for\r\n classification — predicting categories such as\r\n Hired / Not Hired, Spam / Not Spam, or Admitted / Rejected. They answer questions of\r\n the form: \u201cGiven these features, how likely is the outcome to be \u2018Yes\u2019?\u201d :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Logistic Regression: Predicting Probabilities

\r\n\r\n

\r\n Logistic regression is the go-to model when the target is a binary outcome\r\n (0/1, Yes/No). Instead of predicting a raw number like linear regression, it predicts a\r\n probability between 0 and 1. Under the hood, it still\r\n uses a linear combination of inputs, but then passes that through an S-shaped\r\n logistic (sigmoid) function that squashes any real\r\n value into the [0, 1] range.\r\n

\r\n\r\n

\r\n Why not just use linear regression?
\r\n Linear regression can output values less than 0 or greater than 1, which makes no\r\n sense for probabilities. Logistic regression wraps the linear score inside a\r\n logistic curve so outputs stay between 0 and 1.\r\n

\r\n\r\n

\r\n In R, the syntax looks very similar to rpart or random forest, but we\r\n use glm() with a binomial family:\r\n

\r\n\r\n

\r\n model <- glm(Hired ~ ., data = hire_train, family = binomial)
\r\n summary(model)\r\n

\r\n\r\n

\r\n For prediction, we often want probabilities first, then convert them to\r\n Yes/No labels:\r\n

\r\n\r\n

\r\n pred_prob <- predict(model, newdata = hire_test, type = \"response\")
\r\n pred_class <- ifelse(pred_prob > 0.5, \"Yes\", \"No\")\r\n

\r\n\r\n

\r\n A confusion matrix and accuracy are then easy:\r\n

\r\n\r\n

\r\n table(Predicted = pred_class, Actual = hire_test$Hired)
\r\n mean(pred_class == hire_test$Hired)\r\n

\r\n\r\n

Naive Bayes: Fast Probabilistic Classification

\r\n\r\n

\r\n Naive Bayes is a simple but powerful probabilistic\r\n classifier based on Bayes\u2019 theorem. Given the features\r\n of a case (Major, Coding skill, Impression, etc.), it computes the probability of\r\n each class (Hired / Not Hired) and chooses the most likely one.\r\n

\r\n\r\n

\r\n The \u201cnaive\u201d part is the strong assumption that all features are\r\n independent given the class. In reality, they are\r\n rarely truly independent, but this assumption makes the math and computation very\r\n easy. Surprisingly, Naive Bayes often works very well in practice, especially when\r\n there are many features.\r\n

\r\n\r\n

\r\n In R, using the e1071 package:\r\n

\r\n\r\n

\r\n library(e1071)
\r\n model <- naiveBayes(Hired ~ ., data = hire_train)
\r\n pred <- predict(model, newdata = hire_test)
\r\n table(pred, hire_test$Hired)
\r\n mean(pred == hire_test$Hired)\r\n

\r\n\r\n

When Is Naive Bayes Better Than rpart?

\r\n\r\n

\r\n Naive Bayes can outperform decision trees in several common situations:\r\n

\r\n\r\n

\r\n Features are roughly independent given the class.\r\n If Major, Coding, and Impression each provide separate evidence for being hired,\r\n Naive Bayes simply multiplies their contributions. A tree has to choose one split\r\n at a time and may overfit to early splits.\r\n
\r\n Data is small.\r\n Naive Bayes only needs enough data to estimate a few probabilities for each\r\n feature level. Trees need more data to find reliable splits. With a few hundred\r\n rows, Naive Bayes can be more stable than rpart.\r\n
\r\n You want a fast, probabilistic baseline.\r\n Naive Bayes trains almost instantly and yields smooth probabilities like P(Hired =\r\n Yes). Trees can give very sharp jumps and may require pruning and tuning.\r\n
\r\n High-dimensional, text-like data.\r\n For problems like spam detection or sentiment analysis, you often have thousands\r\n of binary \u201cword present / not present\u201d features. Naive Bayes handles this naturally;\r\n trees struggle in such wide spaces.\r\n

\r\n\r\n

\r\n Summary: Naive Bayes is a fast, probabilistic, independent-features\r\n model — a great first model for simple tables or text data. Trees are more\r\n flexible when interactions and thresholds matter.\r\n

\r\n\r\n

When Is Logistic Regression Superior to rpart?

\r\n\r\n

\r\n Logistic regression shines in a different set of situations:\r\n

\r\n\r\n

\r\n Relationship is roughly linear/additive in the log-odds.\r\n If each extra unit of a predictor increases the log-odds of \u201cYes\u201d by a fixed\r\n amount (e.g., every 1000 followers adds 5% to the odds of being hired), logistic\r\n regression captures this smoothly. Trees approximate this with step functions.\r\n
\r\n You need interpretable coefficients.\r\n Logistic regression produces clear numbers: \u201cHolding other variables fixed, a one\r\n point increase in GPA multiplies the odds of admission by X.\u201d Trees give rules,\r\n but coefficients are easier to compare and report.\r\n
\r\n You want stable, smooth probabilities.\r\n Tree probabilities jump abruptly at split thresholds (e.g., from 0.2 to 0.9).\r\n Logistic regression probabilities change gradually with the inputs, which is\r\n better for risk scores, credit, medical decisions, or any domain where calibrated\r\n probabilities matter.\r\n
\r\n Predictors are mostly numeric.\r\n With numeric features (GPA, Scores, Followers), logistic regression uses the full\r\n numeric scale. Trees discretize them into bins, which can lose information.\r\n

\r\n\r\n

Big Picture

\r\n\r\n

\r\n Logistic regression, Naive Bayes, and decision trees are three different ways of\r\n thinking about classification:\r\n

\r\n\r\n

Logistic regression — linear, smooth, interpretable, probability-focused.
Naive Bayes — probabilistic, fast, great for many weak, independent features.
rpart — rule-based, flexible, good for interactions and thresholds.

\r\n\r\n

\r\n In practice, you don\u2019t choose just one forever. You try several, compare their\r\n accuracy and interpretability, and often combine them in ensembles. But understanding\r\n when each model has a natural advantage helps you make smarter choices and build\r\n better prediction systems.\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "14.2", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Logistic Regression and Naive Bayes\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Logistic Regression and Naive Bayes

\r\n\r\n

\r\n

1. Logistic regression is primarily used when the target we want to predict is:

\r\n

A) A continuous number like salary or price
B) A category such as Yes/No or Hired/Not Hired
C) A time series value like stock price tomorrow
D) A cluster label with no probabilities

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. Why is ordinary linear regression not suitable for predicting probabilities directly?

\r\n

A) It can only work with categorical predictors
B) It forces predictions to be exactly 0 or 1
C) It can output values less than 0 or greater than 1
D) It does not allow more than one predictor

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. In R, which command from the slides fits a logistic regression model for Hired using all other columns in hire_train?

\r\n

A) lm(Hired ~ ., data = hire_train)
B) glm(Hired ~ ., data = hire_train, family = binomial)
C) naiveBayes(Hired ~ ., data = hire_train)
D) rpart(Hired ~ ., data = hire_train)

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. After fitting a logistic regression in R, which predict() call from the slides returns predicted probabilities?

\r\n

A) predict(model, hire_test)
B) predict(model, newdata = hire_test, type = \"response\")
C) predict(model, newdata = hire_test, type = \"class\")
D) predict(model, newdata = hire_test, type = \"prob\")

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

5. In the slides, how are class labels (Yes/No) obtained from logistic regression probabilities?

\r\n

A) Using ifelse(pred_prob > 0.5, \"Yes\", \"No\")
B) Using round(pred_prob)
C) Using cut(pred_prob, breaks = 2)
D) By comparing to the mean of pred_prob

\r\n

Answer: A

\r\n

\r\n\r\n

\r\n

6. What does \u201cNaive\u201d mean in Naive Bayes, according to the slides?

\r\n

A) It assumes all classes have equal probability
B) It assumes all features are independent given the class
C) It ignores some of the features randomly
D) It uses only numeric features

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

7. Which R code from the slides trains a Naive Bayes classifier on the hiring data?

\r\n

A) model <- glm(Hired ~ ., data = hire_train, family = binomial)
B) model <- naiveBayes(Hired ~ ., data = hire_train)
C) model <- rpart(Hired ~ ., data = hire_train)
D) model <- randomForest(Hired ~ ., data = hire_train)

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

8. According to the slides, in which situation is Naive Bayes especially likely to outperform rpart?

\r\n

A) When features are strongly dependent and interact in complex ways
B) When there are only two numeric features
C) When features are approximately independent given the class and the dataset is small
D) When the target is a continuous number

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

9. The slides state that logistic regression is superior to rpart when:

\r\n

A) The relationship between predictors and log-odds is roughly linear/additive
B) All predictors are purely categorical with many levels
C) We only care about hard Yes/No predictions, never probabilities
D) The data set is extremely wide, like raw text

\r\n

Answer: A

\r\n

\r\n\r\n

\r\n

10. Which of the following is listed as an advantage of logistic regression over rpart in the slides?

\r\n

A) It produces sharp jumps in probability at split points
B) It cannot handle numeric predictors
C) It provides smooth, stable probabilities and interpretable coefficients
D) It always trains faster than Naive Bayes

\r\n

Answer: C

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "15", "Name": "Association Rules ", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "15.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Association Rules, Apriori, and Their Meaning\r\n \r\n\r\n\r\n\r\n

Association Rules, Frequent Itemsets, and the Apriori Idea

\r\n\r\n

\r\n Association rules were introduced by Agrawal, Imielinski, and Swami in 1993 and\r\n quickly became a core tool for market basket analysis\r\n and other forms of unsupervised learning. The classic picture is a supermarket:\r\n each transaction is a basket of items, and we want to discover patterns like\r\n \u201ccustomers who buy milk and bread often also buy butter.\u201d\r\n :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Frequent Itemsets and the Apriori Principle

\r\n\r\n

\r\n A frequent itemset is simply a set of items that\r\n appears together in many transactions. For example, if {milk, bread} shows up in\r\n 60% of all baskets, it is a frequent itemset.\r\n

\r\n\r\n

\r\n The Apriori algorithm is the classic method for\r\n finding such itemsets. It relies on one key insight:\r\n

\r\n\r\n

\r\n Apriori principle: if an itemset is frequent, then all of its\r\n subsets must also be frequent.\r\n

\r\n\r\n

\r\n The algorithm works level by level:\r\n

\r\n\r\n

Find all frequent 1-itemsets (e.g., {milk}, {bread}, {butter}).
Use them to generate candidate 2-itemsets (e.g., {milk, bread}).
Prune any candidate whose subsets are not all frequent.
Repeat for 3-itemsets, 4-itemsets, and so on, until no more frequent itemsets appear.

\r\n\r\n

\r\n A minimum support threshold (for example, 1% or 5%)\r\n decides which itemsets are kept as \u201cfrequent\u201d and which are discarded as noise.\r\n

\r\n\r\n

Association Rules, Support, Confidence, and Lift

\r\n\r\n

\r\n Once we have frequent itemsets, we turn them into\r\n association rules of the form:\r\n

\r\n\r\n

\r\n X \u21d2 Y\r\n

\r\n\r\n

\r\n where X and Y are disjoint sets of items. Intuitively: \u201cif X is in the basket,\r\n Y tends to be there too.\u201d\r\n

\r\n\r\n

\r\n Three key measures describe such a rule:\r\n

\r\n\r\n

\r\n Support: how often X and Y occur together.\r\n
\r\n Example: support({milk, bread, butter}) = 3/5 if they co-occur in 3 out of 5 transactions.\r\n
\r\n Confidence: probability of Y given X.\r\n
\r\n For {Milk, Bread} \u21d2 {Butter}, confidence = P(butter | milk & bread).\r\n
\r\n Lift: how much X increases the probability of Y\r\n compared to Y\u2019s baseline frequency:\r\n
\r\n lift = confidence / P(Y).\r\n

\r\n\r\n

\r\n If lift > 1, X and Y are positively associated; X makes Y more likely. If lift = 1,\r\n X does not change the chance of Y; there is no real association beyond randomness.\r\n Lift < 1 suggests a negative association.\r\n

\r\n\r\n

A Small Example

\r\n\r\n

\r\n Consider 5 transactions:\r\n

\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n

Transaction	Items
1	Milk, Bread, Butter
2	Milk, Bread
3	Milk, Butter
4	Bread, Butter
5	Milk, Bread, Butter

\r\n\r\n

\r\n For the rule {Milk, Bread} \u21d2 {Butter}:\r\n

\r\n\r\n

Support: {Milk, Bread, Butter} occurs in 3 out of 5 transactions \u2192 3/5 = 0.6.
Milk & Bread alone occur in 4 transactions \u2192 confidence = 3/4 = 0.75.
Butter appears in 4 out of 5 \u2192 P(butter) = 0.8, so lift = 0.75 / 0.8 = 0.9375.

\r\n\r\n

\r\n Lift < 1 here means that, in this toy example, knowing that milk and bread are\r\n present actually slightly decreases the chance of butter compared to the\r\n overall base rate.\r\n

\r\n\r\n

Using the `arules` Package in R

\r\n\r\n

\r\n The arules package in R makes association rule mining\r\n straightforward:\r\n

\r\n\r\n

\r\n install.packages(\"arules\") # once
\r\n library(arules)\r\n

\r\n\r\n

\r\n For the small example above:\r\n

\r\n\r\n

\r\n transactions_list <- list(
\r\n c(\"milk\",\"bread\",\"butter\"),
\r\n c(\"milk\",\"bread\"),
\r\n c(\"milk\",\"butter\"),
\r\n c(\"bread\",\"butter\"),
\r\n c(\"milk\",\"bread\",\"butter\"))
\r\n txn <- as(transactions_list, \"transactions\")
\r\n rules <- apriori(txn, parameter = list(supp = 0.4, conf = 0.6))
\r\n inspect(sort(rules, by = \"confidence\"))\r\n

\r\n\r\n

\r\n On larger data, we read a transaction file (e.g., MarketWide.csv),\r\n convert it to \"transactions\", and run apriori with a\r\n minimum support (say 1%) and confidence:\r\n

\r\n\r\n

\r\n trans <- as(df, \"transactions\")
\r\n rules <- apriori(trans, parameter = list(supp = 0.01, conf = 0.5))\r\n

\r\n\r\n

\r\n We can then sort and inspect the top 10 rules by confidence, or filter for particular\r\n consequents (for example, rules that predict \u201cbutter\u201d with lift > 1).\r\n

\r\n\r\n

Applications Beyond Supermarkets

\r\n\r\n

\r\n Association rules are used far beyond \u201cpeople who bought diapers also bought beer\u201d:\r\n

\r\n\r\n

Market basket analysis: product placement, bundles, cross-selling.
Healthcare: co-occurring symptoms, diagnoses, treatments; drug interaction alerts.
Recommendation systems: \u201cpeople who did X also did Y\u201d.
Customer segmentation: typical patterns of behavior within specific groups.
Fraud detection: unusual combinations that break normal patterns.
Educational data mining, web usage mining, bioinformatics, manufacturing, targeted advertising, and more.

\r\n\r\n

Association Rules and Bayesian Reasoning

\r\n\r\n

\r\n Interestingly, association rules can be viewed through a Bayesian lens. For a rule\r\n {milk} \u21d2 {butter}:\r\n

\r\n\r\n

P(milk) is like a prior on the condition.
P(butter | milk) = confidence is like a posterior probability.
Lift = P(butter | milk) / P(butter) resembles a likelihood ratio:\r\n how much the evidence (milk) shifts our belief about butter.

\r\n\r\n

\r\n When lift > 1, the presence of milk increases the odds of butter, just as evidence\r\n with a likelihood ratio > 1 increases posterior odds in Bayesian updating. When\r\n lift = 1, milk doesn\u2019t change our belief about butter at all.\r\n

\r\n\r\n

Big Picture

\r\n\r\n

\r\n Association rule mining is a way to let the data speak in the form of \u201cif X, then Y\u201d\r\n patterns. The Apriori algorithm efficiently finds frequent itemsets, support and\r\n confidence measure how strong the rules are, and lift tells us whether they represent\r\n genuine associations or just reflect the base rates of items. Combined with tools like\r\n arules, these ideas give us a powerful lens on transactional and\r\n high-dimensional data.\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "15.2", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Association Rules and Apriori\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Association Rules, Frequent Itemsets, and Apriori

\r\n\r\n

\r\n

1. What is the main goal of the Apriori algorithm?

\r\n

A) To train decision trees on labeled data
B) To find frequent itemsets in transactional data
C) To compute Bayesian posterior probabilities
D) To cluster customers into k groups

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. The Apriori principle states that:

\r\n

A) If an itemset is infrequent, all of its supersets are frequent.
B) If an itemset is frequent, all of its subsets must also be frequent.
C) If an itemset is frequent, all of its supersets are frequent.
D) All itemsets with at least one frequent item are frequent.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

3. In association rule mining, the support of a rule\r\n {X} \u2192 {Y} is:

\r\n

A) P(Y | X)
B) P(X | Y)
C) P(X \u2229 Y)
D) P(X) + P(Y)

\r\n

Answer: C (Support is the fraction of transactions containing both X and Y.)

\r\n

\r\n\r\n

\r\n

4. For the rule {Milk, Bread} \u2192 {Butter} in the small example\r\n with 5 transactions, support({Milk, Bread, Butter}) = 3/5 and\r\n support({Milk, Bread}) = 4/5. What is the confidence of the rule?

\r\n

A) 0.60
B) 0.75
C) 0.80
D) 0.50

\r\n

Answer: B (Confidence = 0.6 / 0.8 = 0.75.)

\r\n

\r\n\r\n

\r\n

5. Lift for the rule X \u2192 Y is defined as:

\r\n

A) P(X \u2229 Y) / P(X)
B) P(Y | X) / P(Y)
C) P(X | Y) / P(X)
D) P(Y) / P(Y | X)

\r\n

Answer: B (Lift = confidence / P(Y).)

\r\n

\r\n\r\n

\r\n

6. What does a lift value greater than 1 indicate for a rule\r\n {X} \u2192 {Y}?

\r\n

A) X and Y are negatively associated.
B) X and Y are independent.
C) X and Y are positively associated; X increases the chance of Y.
D) The rule has zero support.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

7. In R, which package and function are used in the slides to\r\n generate association rules from a transactions object?

\r\n

A) rpart package and tree()
B) arules package and apriori()
C) e1071 package and naiveBayes()
D) stats package and lm()

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

8. In the provided R code, what is the role of the line

\r\n

txn <- as(transactions_list, \"transactions\")?

\r\n

A) It converts a data frame of numeric values into factors.
B) It fits a decision tree model on transactions_list.
C) It converts a list of character vectors into a transactions object for arules.
D) It computes the support and confidence automatically.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

9. The slides discuss the connection between association rules\r\n and Bayesian reasoning. In that mapping, confidence({milk} \u2192 {butter}) is treated as:

\r\n

A) P(milk)
B) P(butter)
C) P(milk \u2229 butter)
D) P(butter | milk)

\r\n

Answer: D

\r\n

\r\n\r\n

\r\n

10. Which of the following is not listed in the slides\r\n as an application area for association rules and frequent itemsets?

\r\n

A) Market basket analysis
B) Healthcare (co-occurring symptoms/diagnoses)
C) Educational analysis and web mining
D) Training linear regression models for stock prices

\r\n

Answer: D

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "16", "Name": "Aggreate Data Mysteries", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "16.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n Mysteries of Aggregated Data: Simpson, Prosecutors, and Ecologists\r\n \r\n\r\n\r\n\r\n

Mysteries of Aggregated Data

\r\n\r\n

\r\n Aggregated data can be both powerful and deceptive. As soon as we start averaging,\r\n summing, or combining groups, we risk creating patterns that do not reflect what is\r\n happening at the individual level. This chapter explores three classic traps:\r\n Simpson\u2019s paradox,\r\n the prosecutor\u2019s fallacy, and\r\n the ecological fallacy. Each of them shows how easy it\r\n is to be fooled when we trust aggregates too much or reason carelessly with\r\n probabilities. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Simpson\u2019s Paradox: When Totals Reverse the Story

\r\n\r\n

\r\n Simpson\u2019s paradox is a situation where a trend that appears in several groups\r\n disappears or reverses when the groups are combined. The paradox is not a\r\n bug in math; it is a warning about how aggregation can hide important structure.\r\n

\r\n\r\n

\r\n A classic multiple-choice definition asks:\r\n

\r\n\r\n

\r\n Simpson\u2019s paradox:
\r\n A) A trend appears in different groups of data but disappears or reverses when the groups are combined.
\r\n B) A trend is present in a group but disappears after the group is split.
\r\n C) None of the above.\r\n

\r\n\r\n

\r\n The correct answer is A. The paradox often appears in admission statistics, medical\r\n trials, or sports data. For example, one player might have a higher free-throw\r\n percentage than another in 2015 and again in 2016, yet when you combine both seasons,\r\n the ranking flips. One player took more shots in the \u201cbad\u201d year and fewer in the\r\n \u201cgood\u201d year, so the overall proportion changes.\r\n

\r\n\r\n

\r\n Consider the Tony vs. Anand homework question: Tony scores higher than Anand on\r\n each individual homework. Can the total score over the whole semester still\r\n be higher for Anand? Surprisingly, yes. If Tony tends to do better on low-point\r\n assignments and Anand pulls ahead on high-point assignments, then summing can reverse\r\n the comparison. Simpson\u2019s paradox reminds us that \u201cbetter in every category\u201d does not\r\n always mean \u201cbetter overall\u201d when the categories have different weights.\r\n

\r\n\r\n

The Prosecutor\u2019s Fallacy: Misreading Rare Evidence

\r\n\r\n

\r\n The prosecutor\u2019s fallacy is a mistake in interpreting probability, especially in\r\n legal settings. It often appears with DNA evidence or lottery cases and confuses\r\n \u201cthe probability of the evidence given innocence\u201d with \u201cthe probability of innocence\r\n given the evidence.\u201d\r\n

\r\n\r\n

\r\n Imagine a DNA test where a random innocent person has only a 1 in 1,000,000 chance of\r\n matching the crime-scene sample. That sounds incredibly incriminating. The prosecutor\r\n might say, \u201cThere is only a one in a million chance this is an innocent match.\u201d But\r\n that is incorrect.\r\n

\r\n\r\n

\r\n If we test 10 million people, we expect about 10 matches just by chance, even if\r\n everyone is innocent. The correct question is: given that we have one match\r\n (the defendant), what is the chance they are guilty compared to the other possible\r\n matches and the prior odds of guilt? The fallacy is treating \u201cP(match | innocent) is\r\n small\u201d as if it were \u201cP(innocent | match) is small.\u201d\r\n

\r\n\r\n

\r\n A related example is the lottery cheating story. The prosecutor points out that the\r\n chance a particular person wins the lottery is tiny. True — but millions of\r\n people play. The probability that someone wins is actually quite high. Using\r\n the tiny probability of an individual win as evidence of cheating ignores how many\r\n tickets were bought.\r\n

\r\n\r\n

\r\n Prosecutor\u2019s fallacy (simplified):
\r\n Using the rarity of the evidence to claim that the defendant must be guilty, without\r\n accounting for how many opportunities there were for that rare event to occur.\r\n

\r\n\r\n

\r\n There is also a \u201cdefense fallacy\u201d in which the defense argues that because many\r\n matches were expected, the evidence is completely irrelevant. Both extremes are\r\n wrong; the correct Bayesian reasoning lies somewhere in between and must account\r\n for base rates and the size of the population tested.\r\n

\r\n\r\n

Ecological Fallacy: Confusing Group Patterns with Individual Patterns

\r\n\r\n

\r\n The ecological fallacy arises when we infer individual behavior from aggregate data.\r\n A numerical example makes this especially clear.\r\n

\r\n\r\n

\r\n Group A: 80% of people score 40 points and 20% score 95 points. The mean is 51.
\r\n Group B: 50% of people score 45 and 50% score 55. The mean is 50.\r\n

\r\n\r\n

\r\n At the group level, A has a higher mean than B (51 vs 50). But if we pick one person\r\n at random from A and one from B, the probability that the B person wins is 80%.\r\n Most of the time, we draw a 40 from A and a 45 or 55 from B. The aggregate mean of A\r\n is higher because of a small fraction of very high scores (95), but a typical\r\n individual from A still loses to a typical individual from B.\r\n

\r\n\r\n

\r\n In real life, ecological fallacy appears in political data. Wealthier states in the\r\n U.S. tend to vote for Democratic candidates, while poorer states often lean\r\n Republican. However, at the individual level, richer people are more likely\r\n to vote Republican than poorer people. The correlation between wealth and voting is\r\n positive at the individual level but looks negative at the state level.\r\n

\r\n\r\n

\r\n Several mechanisms can produce this reversal: wealthy states may contain a few very\r\n rich individuals who raise the average; people compare themselves to neighbors and\r\n vote based on relative, not absolute, wealth; or other state-level traits (like\r\n urbanization) drive voting patterns apart from individual income.\r\n

\r\n\r\n

\r\n Core message:
\r\n The correlation of aggregate quantities (ecological correlation) is not the same as\r\n the correlation of individual quantities. We must not assume that \u201cwhat is true for\r\n groups is true for individuals.\u201d\r\n

\r\n\r\n

\r\n This also applies to classroom data: if the average prediction score of non-CS majors\r\n is lower than CS majors, we cannot immediately conclude that any random CS student is\r\n more accurate than any random non-CS student. Overlaps, outliers, and uneven group\r\n sizes can make the individual-level story very different from the group-level story.\r\n

\r\n\r\n

Aggregates vs. Individuals: Handle with Care

\r\n\r\n

\r\n All three phenomena share a common theme: aggregates can be\r\n misleading if we forget how they were formed. Simpson\u2019s paradox warns that\r\n combining groups can reverse trends. The prosecutor\u2019s fallacy warns that rare\r\n evidence must be interpreted in the context of how many chances there were for it to\r\n occur. The ecological fallacy warns against reading individual behavior directly off\r\n group averages.\r\n

\r\n\r\n

\r\n Aggregated data are essential for summarizing and communicating complex realities.\r\n But every time we see an average, a percentage, or a correlation computed over\r\n groups, we should ask: What does this hide? What happens if we look inside the\r\n groups? How many people or events generated this pattern? Only then can we\r\n avoid being fooled by the mysteries of aggregated data.\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "16.2", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Mysteries of Aggregated Data\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Mysteries of Aggregated Data

\r\n \r\n\r\n

\r\n

1. Which statement best describes Simpson\u2019s paradox?

\r\n

A) A trend is present in a group but disappears after the group is split.
B) A trend appears in different groups of data but disappears or reverses when these groups are combined.
C) A trend is caused only by random noise.
D) None of the above.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. In the Tony vs Anand example, Tony scores higher than Anand on every individual homework. Can Simpson\u2019s paradox still occur so that Anand has a higher total score?

\r\n

A) Yes, this is possible.
B) No, it is impossible if Tony wins every homework.
C) Only if they have exactly the same grades on at least one homework.
D) Only if there is a grading error.

\r\n

Answer: A

\r\n

\r\n\r\n

\r\n

3. The prosecutor\u2019s fallacy often confuses which two probabilities?

\r\n

A) P(guilty) and P(innocent).
B) P(evidence | innocent) and P(innocent | evidence).
C) P(evidence | guilty) and P(guilty | evidence).
D) P(innocent | guilty) and P(guilty | innocent).

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. In the DNA example, a \u201c1 in a million chance of a match\u201d means:

\r\n

A) The defendant is definitely guilty.
B) Only one person on Earth can match that DNA.
C) A random innocent person has a 1 in a million chance of matching the DNA profile.
D) The probability that the defendant is innocent is 1 in a million.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

5. If 10 million people are tested and the random-match rate is 1 in a million, how many innocent matches do we expect on average?

\r\n

A) 1
B) 5
C) 10
D) 1,000

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

6. In the lottery cheating example, what is the main logical error made by the prosecutor?

\r\n

A) Assuming the lottery is fair.
B) Ignoring how many people play the lottery and focusing only on how unlikely it is for one specific person to win.
C) Forgetting that tickets cost money.
D) Using approximate probabilities instead of exact ones.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

7. Which of the following best describes the ecological fallacy?

\r\n

A) Assuming that individual-level trends always reverse at the group level.
B) Assuming that the correlation between two variables is always 0 at the individual level.
C) Inferring individual behavior or relationships directly from aggregate (group-level) statistics.
D) Using ecology examples to explain statistics.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

8. In the numerical example of ecological fallacy, Group A has mean score 51 and Group B has mean 50. Yet, when picking one person at random from each group:

\r\n

A) The A person wins about 80% of the time.
B) The B person wins about 80% of the time.
C) They are tied exactly 50% of the time.
D) There is not enough information to tell.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

9. Which statement about wealthy states and wealthy individuals illustrates the ecological fallacy?

\r\n

A) Wealthy states vote Democrat, so wealthy individuals must also vote Democrat.
B) Poor states vote Republican, so poor individuals never vote Democrat.
C) Wealthy individuals vote Republican, so wealthy states must always vote Republican.
D) If a state is wealthy, its average income is lower than that of poor states.

\r\n

Answer: A

\r\n

\r\n\r\n

\r\n

10. The slides state that \u201cthe correlation of aggregate quantities (ecological correlation) is not equal to the correlation of individual quantities.\u201d What is the main takeaway?

\r\n

A) We should always prefer aggregate data over individual-level data.
B) Individual-level correlations can be safely inferred from state-level or group-level correlations.
C) Group-level patterns may differ drastically from individual-level patterns, so we must be careful when interpreting aggregates.
D) Correlations computed at any level are always identical.

\r\n

Answer: C

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "17", "Name": "Power Law", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "17.1", "Name": "Summary", "Details": "\r\n\r\n\r\n \r\n From Mediocristan to Extremistan: Power Laws vs the Bell Curve\r\n \r\n\r\n\r\n\r\n

From Mediocristan to Extremistan: Power Laws vs the Bell Curve

\r\n\r\n

\r\n Not all randomness is created equal. Some worlds are ruled by the familiar\r\n Gaussian (normal) distribution, where most values hug\r\n the average and extreme events are vanishingly rare. Other worlds follow a\r\n power law, where extremes are not just possible but\r\n dominant. The slides describe these two realms as\r\n Mediocristan and Extremistan. :contentReference[oaicite:0]{index=0}\r\n

\r\n\r\n

Mediocristan: The World of the Bell Curve

\r\n\r\n

\r\n Many physical traits live in Mediocristan: height, weight, blood pressure, cholesterol.\r\n They are approximately normal. Most observations cluster around the mean, and as you\r\n move away from that center, the probability of seeing a value drops faster and faster,\r\n roughly at an exponential rate.\r\n

\r\n\r\n

\r\n The slides illustrate this with height. Assume the average height is 1.67 m\r\n (about 5\u20327\u2033). The odds of being:\r\n

\r\n\r\n

10 cm taller (1.77 m): about 1 in 6.3
20 cm taller (1.87 m): about 1 in 44
30 cm taller (1.97 m): about 1 in 740
40 cm taller (2.07 m): about 1 in 32,000
50 cm taller (2.17 m): about 1 in 3,500,000
60 cm taller (2.27 m): about 1 in 1,000,000,000
100 cm taller (2.67 m): essentially impossible

\r\n\r\n

\r\n The probability drops so quickly that \u201cmonsters\u201d simply do not exist. If the total\r\n height of two people is 14 feet, the overwhelmingly likely breakdown is something like\r\n 7 and 7 feet, maybe 6 and 8, but not 2 and 12. Extremely tall individuals are just too\r\n rare to dominate the total.\r\n

\r\n\r\n

\r\n Gaussian distributions are fragile in the tails: a small mis-estimation of the\r\n standard deviation can lead to massive underestimation of rare events. The odds of a\r\n 20-sigma event are a trillion times higher than those of a 21-sigma event, so tail\r\n probabilities are very sensitive to measurement error.\r\n

\r\n\r\n

Extremistan: The World of Power Laws

\r\n\r\n

\r\n The \u201cvirtual world\u201d of money, sales, web traffic, and popularity does not behave like\r\n height. It belongs to Extremistan, where power laws\r\n rule. Here, a few observations can be larger than all the others combined. Wealth,\r\n income, book sales, views, and clicks often follow such power-law patterns.\r\n

\r\n\r\n

\r\n In a power-law distribution, the probability that a value exceeds a threshold V*\r\n falls as a fixed power of V*:\r\n

\r\n\r\n

\r\n If VALUE increases by a factor of k, then Count(V > V*) decreases by k^e.\r\n

\r\n\r\n

\r\n Crucially, the rate of decline is steady. For example, suppose net worth\r\n follows a power law where each doubling of wealth divides the number of people by 4:\r\n

\r\n\r\n

Over \u20ac1 million: 1 in 63
Over \u20ac2 million: 1 in 125
Over \u20ac4 million: 1 in 250
Over \u20ac8 million: 1 in 500
Over \u20ac16 million: 1 in 1,000
Over \u20ac32 million: 1 in 2,000
Over \u20ac320 million: 1 in 20,000
Over \u20ac640 million: 1 in 40,000

\r\n\r\n

\r\n When you double the wealth level, the incidence always shrinks by the same factor.\r\n There is no accelerating drop-off like in the Gaussian world. This is why massive\r\n fortunes and viral hits are not impossible outliers; they are baked into the\r\n structure of Extremistan.\r\n

\r\n\r\n

What If Wealth Were Gaussian?

\r\n\r\n

\r\n The slides ask a provocative question: what if wealth followed a normal distribution?\r\n

\r\n\r\n

\r\n Under a hypothetical Gaussian law, we might have:\r\n

\r\n\r\n

Over \u20ac1 million: 1 in 63 (same as observed)
Over \u20ac2 million: 1 in 127,000
Over \u20ac3 million: 1 in 14,000,000,000
Over \u20ac4 million: 1 in 886,000,000,000,000,000
Over \u20ac8 million: astronomically rare

\r\n\r\n

\r\n In such a world, billionaires would essentially not exist. Extreme wealth would be as\r\n rare as 8-foot-tall people. But reality does not look like that: net worth follows a\r\n Pareto distribution with heavy tails, where the mean\r\n may exist but the variance can be infinite. This is why finance and wealth are prone\r\n to black swan events: huge, surprising deviations that\r\n dominate the averages.\r\n

\r\n\r\n

Mediocristan vs Extremistan: Splitting a Total

\r\n\r\n

\r\n One way to feel the difference is to split a fixed total between two people.\r\n

\r\n\r\n

\r\n Income: if two people together earn $1 million a year, in Mediocristan the most\r\n likely split is $500,000 and $500,000. In Extremistan, a more realistic split\r\n might be $50,000 and $950,000.\r\n
\r\n Book sales: if two authors sold a total of one million copies, the most likely\r\n split in Extremistan is something like 993,000 vs 7,000, not 500,000 vs 500,000.\r\n For large totals, the breakdown becomes more and more asymmetric.\r\n
\r\n Height: if two people together are 14 feet tall, the most likely split is around\r\n 7 and 7 feet. A 12-foot vs 2-foot split is essentially impossible.\r\n

\r\n\r\n

\r\n Extremistan is also where the 80/20 rule lives: 80% of\r\n the work done by 20% of the people, 80% of sales from 20% of products, 80% of views\r\n from a small fraction of videos. Such extreme concentration is natural for\r\n power-law distributions.\r\n

\r\n\r\n

A Thought Experiment with Weight

\r\n\r\n

\r\n To further highlight the absurdity of applying power laws to physical traits, the\r\n slides imagine weight following a power law. Suppose median weight is 160 pounds and\r\n 150 million people weigh more than that. With an exponent of 4, if we increase the\r\n threshold tenfold to 1,600 pounds, we divide the count by 10,000:\r\n

\r\n\r\n

\r\n 150,000,000 / 10,000 = 15,000 people over 1,600 pounds\r\n

\r\n\r\n

\r\n Clearly this is nonsense in the real world: such bodies cannot exist in those\r\n numbers. Power laws are for wealth, views, connections, not for bones and flesh.\r\n

\r\n\r\n

Black Swans and Infinite Variance

\r\n\r\n

\r\n Mathematically, a power-law distribution has a finite mean only if the exponent is\r\n large enough, and a finite variance only if it is larger still. Many real-world\r\n power laws have exponents where the mean exists but the variance does not. This\r\n means that a single extreme observation can overwhelm all others.\r\n

\r\n\r\n

\r\n Think of estimating average income in a room of friends. Add the world\u2019s richest\r\n person. The average explodes. In a Gaussian world that would never happen; in a\r\n power-law world it is expected.\r\n

\r\n\r\n

Conclusion: Know Which World You Are In

\r\n\r\n

\r\n The key lesson of the slides is simple but profound: before you analyze data or\r\n reason about risk, ask yourself whether you are in Mediocristan or Extremistan.\r\n Gaussian tools and intuitions work well for heights, test scores, and biological\r\n measurements. They fail disastrously for wealth, traffic, and network effects, where\r\n power laws, black swans, and the 80/20 rule dominate.\r\n

\r\n\r\n

\r\n Misclassifying the world leads to mismeasured risk. Treat a power law as a bell\r\n curve, and you will be \u201ca trillion times wrong\u201d about the odds of extreme events.\r\n Treat a bell curve as a power law, and you will imagine monsters that never appear.\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "17.2", "Name": "Quiz", "Details": "\r\n\r\n\r\n \r\n MCQ Quiz \u2013 Power Laws vs the Bell Curve\r\n \r\n\r\n\r\n\r\n

MCQ Quiz \u2013 Mediocristan, Extremistan, and Power Laws

\r\n\r\n

\r\n

1. In the slides, \u201cMediocristan\u201d refers to a world where:

\r\n

A) Power-law distributions dominate and a few values are larger than all others combined.
B) Most variables follow approximately Gaussian distributions with rare extreme values.
C) All outcomes are equally likely (uniform distribution).
D) The mean and variance do not exist.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

2. Which of the following is most likely to live in Mediocristan?

\r\n

A) Individual wealth
B) Number of views per YouTube video
C) Human height
D) Company market capitalization

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

3. In a power-law (Extremistan) world, increasing the value threshold by a factor of k typically:

\r\n

A) Reduces the count by an exponential of k\u00b2.
B) Reduces the count by a fixed power of k, like k^e.
C) Leaves the count unchanged.
D) Reduces the count linearly by k \u2212 1.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

4. Why are \u201cmonsters\u201d (e.g., a 12-foot human) essentially impossible in a Gaussian world?

\r\n

A) The mean of the distribution is always zero.
B) The probability of very large deviations from the mean drops extremely fast.
C) Gaussian distributions have infinite support only on the left side.
D) Gaussian distributions do not allow any values above the mean.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

5. If total height of two randomly chosen people is 14 feet, the slides argue that the most likely split in Mediocristan is approximately:

\r\n

A) 2 feet and 12 feet
B) 4 feet and 10 feet
C) 7 feet and 7 feet
D) 0 feet and 14 feet

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

6. Which of the following phenomena is a hallmark of Extremistan and power-law behavior?

\r\n

A) The central limit theorem always holds exactly.
B) No single observation can significantly change the mean.
C) A small number of observations contributes a large fraction of the total (e.g., 80/20 rule).
D) The distribution is always symmetric around the mean.

\r\n

Answer: C

\r\n

\r\n\r\n

\r\n

7. If wealth actually followed a Gaussian distribution with the same average as observed data, what would the slides predict about billionaires?

\r\n

A) They would be very common, similar to people of average height.
B) They would not exist or be astronomically unlikely.
C) There would be exactly one billionaire per country.
D) Their number would be independent of the distribution shape.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

8. In a power-law wealth distribution where each doubling of wealth cuts the number of people by 4, going from a threshold of \u20ac1M to \u20ac16M changes the count by what factor?

\r\n

A) Divides by 2
B) Divides by 4
C) Divides by 8
D) Divides by 16

\r\n

Answer: D
\r\n (Each doubling: \u00f74. From 1 \u2192 2 \u2192 4 \u2192 8 \u2192 16 is 4 doublings, so 4\u2074 = 256; that\u2019s a divide-by-256 overall, i.e., a factor of 16 per doubling in wealth level four times.)

\r\n

\r\n\r\n

\r\n

9. Why are Gaussian-based risk estimates fragile in the extreme tails, according to the slides?

\r\n

A) The mean and variance are undefined for Gaussian distributions.
B) Tail probabilities depend extremely sensitively on the estimated standard deviation.
C) Gaussian distributions assume all events are equally likely.
D) The Gaussian has no tails at all.

\r\n

Answer: B

\r\n

\r\n\r\n

\r\n

10. Which of the following is the most important practical lesson from the Mediocristan vs Extremistan discussion?

\r\n

A) Always model everything with a power law.
B) Always model everything with a Gaussian.
C) First identify whether your data lives in a Gaussian-like or power-law world before choosing tools.
D) Averages are meaningless in any distribution.

\r\n

Answer: C

\r\n

\r\n\r\n\r\n\r\n", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "18", "Name": "Notes on R101", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "19", "Name": "Short Intro to AI", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "20", "Name": "Symbolic AI", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "21", "Name": "Large Language Models - Core Concepts", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "22", "Name": "Red Teaming Chatbots", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "23", "Name": "Verification - Canary in Coal Mine", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}, {"Section": "24", "Name": "Agentic Architecture", "Details": "", "Slides": "", "BoundlessDataset": 0.0, "Ignore": 0.0, "Video": 0.0}]