Maintenance Phase - School Lunches, P-Hacking and the Original "Pizzagate"

Episode Date: July 20, 2021

This week we're diving into one of the biggest-ever scandals in nutrition research. For nearly two decades, Brian Wansink's Food and Brand Lab told Americans that lower weights, healthier workplaces and better school lunches were just a few small tweaks away. Then, in 2015, he wrote a blog post and it all came crashing down. Support us:Hear bonus episodes on PatreonDonate on PayPalGet Maintenance Phase T-shirts, stickers, face masks and moreLinks!A Credibility Crisis in Food ScienceStatistical heartburn: An attempt to digest four pizza publications from the Cornell Food and Brand LabBrian Wansink: Data Masseur, Media Villain, Emblem of a Thornier ProblemThe Wansink Dossier: An OverviewMore Evidence That Nutrition Studies Don’t Always Add UpHere’s How Cornell Scientist Brian Wansink Turned Shoddy Data Into Viral Studies About How We EatFalse-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as SignificantDeath of a Veggie SalesmanThe Strange, Uplifting Tale of “Joy of Cooking” Versus the Food ScientistMoms, “Food Fears” and the Power of the InternetThe science behind Smarter LunchroomsYou Can’t Trust What You Read About NutritionEnergy balance measurement: when something is not better than nothingScientific method: Statistical errorsEffect of the Healthy Hunger-Free Kids Act on the Nutritional Quality of Meals Selected by Students and School Lunch Participation Rates9.67 Degrees of DeceptionSupport the show

Hi everybody and welcome to Maintenance Phase, the podcast that will gladly trade you two regular milk tickets for one chocolate milk ticket. That's good. I knew we were talking about school lunches and that was my one very vivid memory. The podcast that serves you turkey tetrazini. I'm Aubrey Gordon. I am Michael Hobbs. If you'd like to support the show,

we are on Patreon at patreon.com slash maintenance phase. We're now releasing bonus episodes so you can enjoy some bonus content, support the show if you want to and don't, if you do. Keep listening and never give us a dime, it's chill. That's right. And today, we are talking about school lunches, I think. Aubrey, I'm so excited.

Oh, I can't wait. This is our first, I think, straight forward clickbait episode. Ha ha ha. I am going to title this episode, school lunches, pee hacking, and the original pizza gate. The original pizza gate. That's like how we're drawing people in. But the actual story that we're gonna talk about today

is basically the rise and fall of a single food and nutrition researcher who was one of the most prominent people in this field for more than a decade. His name is Brian Wonsink, and I think it's a really good story. It's like by, our most methodology, Queenie episode.

But also, if we called it Brian Wonsink, nobody would listen to it. So we've gotten you here with a catchy title, and now we're gonna pump statistics into you. I can't wait. Also, like, my knowledge of this topic runs like an eighth of an inch deep. Excellent.

The only reason the name rings a bell is listener sent in an email being like, I think you should do an episode about this guy. And I told you about it when you were like, I don't know what I'm gonna do for my next episode. And then my eyes got as big as the dinner plates in the research we're about to cover. Well, the great and hilarious thing is that I told you about it. I was like, apparently there's this whole nutrition research

scandal and you were like, oh, Brian launching? Yeah. Pete, Michael Hobbs response. So I have actually, I've been following this for a while because full disclosure, I was like one of the people who totally fell for this guy. I'm not going to pretend to be above any of the biases that we're going to talk about or any of, really, think, the structural problems in media and in academic research that this episode is like an entry point into. You know, he spent more than a decade being like one of the most prominent sort of brand name researchers in this field. He wrote two best-selling books. He was on the sort of the TED Talk circuit when this entire downfall happened. The New York Times writes an article about it, this kind of perfunctory article, and at the end, they note that he had been quoted in 60 New York Times articles over the course of almost 20 years. Good Lord! So, one of the frustrating things about this honestly

is that for our main protagonist, Brian Wonsink, there's actually very little information available about his early life and sort of how he got into the field of food research. What we do know about Brian is that he starts as a marketing professor, which is that we're already in the foreshadowing section. I was going to say this does not bode well. This is an excerpt from his book Mindless Eating, which comes out in 2007. I'm never sure what to say when someone asked so I first became interested in food, psychology, and marketing. I usually say I really liked Vance Packard's 1957 book The Hidden Persuaders, because he tried to show how advertising

unconsciously affects us. I think this also happens when we eat, except the hidden Persuaders are the way we set up our tables, our kitchens, and our routines. I'm gonna go out on a limb and also assume that Brian wants to get's not a fat dude. Oh, absolutely not. He's a skinny white guy, he's blonde, he looks around like six-one, something like that. I'm looking up a picture of him, just to, yeah, there you go. He's got kind of the Ed Begley Jr. kind of look about him.

Yeah, like suburban dad, like do you guys want some nachos, like calling in from the kitchen? Yes. Guy who owns a recumbent bike. Absolutely. It is the vibe from the kitchen? Yes. Guy who owns a recumbent bike. Absolutely. Yes, the vibe with this guy. Yes. I mean, the only thing that I think

that we can sort of pull out of these origin stories is really that he's fascinated by the idea that people, especially consumers, make choices without really knowing why they're doing it. Right. So after he gets his PhD from Stanford, he's basically a kind of normal marketing professor at various business schools. He works at Dartmouth. He goes to the Wharton

Graduate School of Business at the University of Pennsylvania. And eventually he sort of lands at the University of Cornell in 2005. Because there's kind of no biographical details in his books, I kind of had to piece together his career basically from his like Google Scholar citations. So I just like organized all of his research in chronological order and just started looking at the kinds of studies that he was publishing over the course of his career. He publishes over 480 academic studies. Man, oh man.

So what, where does that productivity come from? I mean, man, I'm gonna go a little gutter. Fuck, okay, shit. Huge, I can't tell you without a huge boiler. No, okay. But the first 10, 15 years of his career, the research that he's producing is like very straightforward marketing research.

So one thing that he's really obsessed with is this idea of unit size. That, you know, if people buy a large bag of chips, they'll eat the whole bag. And if they buy a small bag of chips, they'll eat the whole bag. And if they buy a small bag of chips, they'll eat the whole bag. But he's doing all of this research to give advice to companies on how big their unit sizes should be. It's very clear that what he's doing is like he's helping companies sell more products.

That's the way that all of his work is framed throughout the 1990s. Yeah, this is the Halo Top approach, right? Yes. Don't stop till you hit the bottom. Let's just assume that you're going to eat. Don't stop till you hit the bottom. Like, blah, blah, blah, blah, blah, but like, let's just assume that you're gonna eat the whole package of whatever you buy. Exactly.

So his early work shows that like, what you name food is actually really important for whether or not people buy it, and it even affects their taste. So people will actually rate like wine that is from California, that they think is from California as more tasty than wine that they think is from California, as more tasty than wine that they think is from North Dakota. People rated freedom fries as tastings superior to french fries. This is one of those places where it's like it's very tempting to believe that we're all

much more sophisticated than we are and our decision-making is really different than it is, but we're all kind of on autopilot and we're all way more predictable than we would like to think that we are. Completely. We all are profoundly affected by marketing and we all think that we're all way more predictable than we would like to think that we are. Completely. We all are profoundly affected by marketing and we all think that we're not. Yeah. Like, I bought a Casper mattress, man. Oh, did you? Oh, yeah. It's not because I like did a literature deep dive.

It's like, no, they mentioned it on like five different podcasts. Truly two days ago, I bought a Helix mattress for exactly what? Yeah, what? A fun fact, Aubrey, when your Helix mattress comes, you're more likely to eat more of it if it comes in a large package. In a small package. So, I mean, a lot of his early work

is sort of around these same kinds of ideas, is basically trying to figure out what makes people purchase products. He finds that you eat more fat if you put olive oil on your bread than butter, but you also eat less bread. His studies get a little bit of play in the media, but like he's not really really a name in nutrition research.

That all changes in 2005. He publishes two studies that are explosive in the press. You could not avoid these stories at this time. The first, I'm sure you've heard of this one. Do you know the bottomless bowl study? I don't know the bottomless. Is this an all you can eat kind of thing? Yeah, so they did this thing.

He has this like lab now that has like hidden cameras and two way mirrors and it's all these ways of like surveilling the way that people eat and why they eat differently. And the study is exactly what it sounds like. They build a bowl that has like a little tube underneath it where it's actually feeding more soup into the bowl as you're eating it.

And so he finds that people eat, I think it's actually feeding more soup into the bowl as you're eating it. And so he finds that people eat, I think it's 53% more of this bowl that is refilling itself. It's this idea that you just eat until the bowl is empty. Like none of us are eating based on any satiety cues. We're just like, there's more left in the bowl, so I better keep eating. This lab of his was just the Olive Garden?

Yeah. There actually is a test restaurant. I don't know why they don't call it the test-a-ront at this university, where people, diners can come and they know that they're participating in studies. It's like marketing studies. And it'll be the menu will be like, have French names one night, and then English names

the other night, and they'll test, like does this affect your purchasing decisions. So he's also doing these lab studies in this tester-on constantly. That's fascinating. And also, like, I feel, as you're just talking about the experimental designs, I feel myself going, like, oh!

Oh, yeah. Right, like, it feels like an interesting, like, dinner party conversation, sort of topic. So, I'm like, there's a lot of curb appeal to these studies. I mean, this is, catnip for journalists. Sure. The other big study that comes out in 2005 is the, do you know this one, the popcorn study?

I don't know the popcorn study, at least not by name. This is one where people are going to a movie and sort of at the door, he tells them like, you've been entered into a drawing something, something, we're giving everybody free popcorn tonight. And so they do this in like a bunch of different conditions, right? So in one night they give everybody a medium-sized bucket of popcorn. And then the next night they give them large buckets of popcorn.

Sure. So the sort of the twist of the study is that half of the people in all of these conditions get like shitty popcorn. They said it was like squeak when you eat it. Like it's old, it's stale, it's gross, and two people apparently asked for their money back, and they're like, the popcorn's free. Like, just don't eat it. It's a business, of course, makes a big splash, because it turns out, you know, first of all, people eat way more when they are given

this large bucket of popcorn, and secondly, they eat more of even like shitty food. So, this is what he says in mindless eating about this study. Did people eat because they liked the popcorn? No. Did they eat because they were hungry? No. They ate because of all the cues around them. Not only the size of the popcorn bucket, but also the distracting movie, the sound of people eating popcorn around them, and the eating scripts we take to movie theaters with us. All of these were cues that signaled it was okay to keep on eating and eating. So like, I understand that people are eating this sort of mindlessly

and without really any connection to the quality of the food or how fresh it was or anything like that, but like, how does he get from there to, it's all these other, like, I figured out what it is. Like, I know it's not this thing, therefore it's this other thing. Exactly. Like, how does he get from point A to point E? These questions, Aubrey, are exactly the questions

that nobody fucking asked at the time. These two studies, like, they're written up in the Atlantic, they show up in the New York Times. I mean, there's just a huge frenzy of media activity around these two stories. And thus begins Brian Wonseng's like long career of just like publishing blockbuster study after blockbuster study.

This is an excerpt from a Vox article. His experiments have found, for example, that women who put cereal on their kitchen counters way more than those who don't, and that people will pour more wine if they're holding the glass than if it's sitting on the table. I hate this shit so much, Mike. I know, dude.

I remember years ago, do you remember that show of the Doctors? It was like a daytime TV show where they had like a few doctors on. At the end of every show, they would read out shit like this that is like completely decontextualized, completely like nonsense, right? Yeah. And I absolutely remember being at a nail place one time, getting my nails done. The doctors was on in the background and they were like, here's an interesting finding. Women who have fresh cut flowers at home report being happier than those that don't. So if you wanna make your wife happy, bring her some flowers.

And I was like, what the fuck is this? You're not gonna talk about like, who has 20 bucks to blow every week on fresh flowers? Like flowers are expensive. It's like a very broad statement, but because it comes from a sort of a sciencey source, totally. It feels more legit when you're like, it's still weird bullshit garbage.

It's perfectly structured to be in that form of like, hey, did you know women who do this also have this as if it's some kind of rule? It's essentially like a snaple cap fact. Yes, exactly. Were you like, no, no, no, no, no, no, no, no, no, no, no, listen, the great thing about the snaple cap fact is that I find out that fish take naps. Yeah, yeah, yeah, yeah, it, no. Listen, the great thing about the Snapple Cap fact is that I find out that fish take naps.

Yeah, yeah, yeah. It doesn't make me be a fish who takes it out. I mean, the best example of that is one of his studies that gets like an amazing amount of play in the mainstream media is one that shows men eat 93% more pizza and 86% more salad in the presence of women. What? It is one of those things where like people try to translate this into a weight loss rule.

They're like, donate with women. I think something else is going on. Ha-ha-ha-ha. There's also an infamous study where Brian goes through old editions of The Joy of Cooking. You know, The Joy of Cooking has been around since like 1936. He publishes a study where he says only, there's only about 18 recipes that

have like endured from 1936 through all the additions to, I think the most recent at the time was the 2006 edition. And it turns out the calories in those recipes have increased significantly over time. So one of his sort of explanations for the obesity epidemic is this idea that like portions, unit sizes, the food environment has completely changed over time. And this is like a perfect little encapsulation of that. That like a sort of a normal dinner time recipe

is just like 30% larger now than it was in 1936. That feels observably true, right? Like in my lifetime alone, right? Like the largest drink size that you can get has like doubled, maybe true, right? Like in my lifetime alone, right? Like the largest drink size that you can get has like doubled, maybe tripled, right? Another thing that he mentions in the study is that muffin tins. If you look at old muffin tins,

the muffins were like half as big as they are now. Yeah, there you go. Another infamous one is he measures the eye angle of cartoon characters on cereal boxes. What? And he finds that brands aimed at children, the cartoon characters are looking downward at an angle of 9.67 degrees

so that it looks to children in the grocery store as if they are looking at them. This is another of his like very famous studies that if you Google it, you can still find like 50 references to it. Man, when you started talking about cartoon eye angles, I was like, are we gonna get into real racist territory? Anyway, back to this Dr. Seuth book from the 40s.

Like, oh no! For once, that isn't where that was going. Oh, thank God. First time on the show. So by 2007, he's basically translating all of these findings into weight loss advice. Uh-oh. So his book Mindless Eating comes out in 2007, becomes a massive bestseller. It's, you know, reviewed in the New York Times and the New Republic. It's like this huge deal. What his research implies is that like there are all of these sort of small forces on our

behavior. And if you can change those small forces, you can actually lose weight, sort of without really knowing it, right? So that, you know, one of the main pieces of advice that goes around from his book is like, you can just switch to smaller plates in your house, because people tend to eat less from smaller plates. This explains why so many co-workers around that time were like, hmm, get a smaller plate. Right. This is absolutely what people want to hear. You can become thin without having to really think about it. Exactly. And without anything else in society changing either.

So we are going to watch a clip. Oh, this is a presentation of his work from a Michael Pauline documentary in the early two founders. Also, at some point, we're going to use him talking about Michael Pauline, man. I know, dude. That's, I know. So, um, grab a plate up there, the pasta is right in the stove, serve yourself up. Brian Wonsink is an expert on eating behavior. He's discovered we're often not aware of why we eat as much as we do. Sometimes it's because of something we don't give the slightest thought to, like the size of our plate. We'll bring people in, we'll give them a large plate to serve themselves,

but what they don't realize is that the pasta is cold. One thing concocks an excuse, so that everyone has to get a different plate, which is slightly smaller. These things weren't the right temperature, because I think you'd come back and just grab another plate out of the cupboard there. One of the things we find is that they'll serve themselves a second time. They won't believe they serve an amount any different than they did the first time. Did you guys notice any difference between the first time you served yourself and the second time you served yourself? Play feels like small it looks smaller. Oh. So here's one thing we found, the size of a plate tremendously biases us in terms of how much we serve.

The smaller the plate, the less food people take. Serving you serve four ounces on a 90's, but you go holy cow, I'll never be able to eat that. So let's take a look at what happened to you guys. Now I've some big plate, 207 calories, smaller plate, it dropped down to 162 calories. Whoa, that's about 40 calories or 40? If this happened three times a day,

over the course of a year, we used a smaller plate, you'd end up between nine pounds less than you would if you had a bigger plate. It's just really small things, make this really huge difference. I hate this shit. I know me too. I hate it so hard.

I knew you would. It really feels like it plays into this sort of life hack kind of approach to there are scientific reasons that are beyond your control. And if you just fix those scientific reasons, you will become thin. And again, like any fucking fat person can tell you, having less pasta on your plate does not make you nine pounds later at the end of the year. Right, like it's just, it's such weird, facile logic

that I think because it's coming from a researcher and because it's coming from a researcher at an Ivy League university, that it feels fancier, it feels more legit. I also love the fact that it's like this wildly artificial scenario, right? It's like it's only what appears to be college undergrads. They are all white.

They are in this weird situation where they serve themselves food. And then he says, no, no, no, we have to serve you again out of this like microwave to bowl and new plates. And then they take the plates, but it's not clear like if they're gonna eat the first thing they serve themselves,

so it kind of makes sense that like the second time you would take less, because you're like, well, do I have to eat all of this? Yeah. The whole scenario is just so fucking weird and artificial that it's like, it's not clear to me that you can actually extrapolate from this. Also, I'll tell you what, that pasta looked overcooked

before it went back in the microwave. I know, it looked bad. It looked mushy by the time it came out. So that part's also, like, I would take less of the, like, it was already overcooked, and now it's like pasta mush. I mean, do you remember this context? Do you remember the book Nudge?

Yes, also, this was around the time that like, Malcolm Gladwell sort of burst onto the scene, and we all started talking about 10,000 hours of whatever for mastery, and it was a whole wave of again, this sort of like life hacking kind of stuff. Totally. That was like, you just need to know these little, like, they seem small, but they're really important scientific findings that will impact every other thing about your life. Yeah, I mean, this was like one of the most important prominent ideas at the time was this idea of behavioral economics

that sort of the way that people behave is one of the books that came out at the time was called predictably irrational. So the canonical example that showed up in every single article about this was organ donations. Some countries have 17% of people volunteer to be organ donors. And then if you look at, I believe it was the Netherlands,

it's like 60% of people. And you know, it seems like, oh my God, there's so much more virtuous, like what's going on in the Netherlands? They're also much nicer about their organs. And it turns out that it's just the default on the form. In America, you have to take a box to say, yes, I will donate my organs.

And in the Netherlands, you have to take a box that says, nope, I don't really want to donate my organs. It's the same thing with automatic voter registration. Like when people have to opt out of being a registered voter, more people are registered and more people turn out to vote. Yes. And Brian Wonsink was actually a huge part of that. So, the guys that wrote the Nudge book, like the book that basically began this entire trend in 2008, they wrote a really long review of Brian Wonsink's book in New Republic.

Like this idea of sort of art-eating behavior, being a metaphor for all of these other behaviors, like whether we pay our taxes, which schools we send our kids to. This feels like an extremely dude way of approaching the world, which is like, Oh, I know. We just need to listen to the data and do what the data says. That's it. It just, it was this very sort of quote unquote rationalist approach. Of that, you know, life can be broken down into these sort of inputs and outputs, like these little flowcharts, right? And we know that people will do X if we give them Y, right? Like we can predict the ways that people are going to behave,

and all we need to do is follow the science and we'll be able to solve all of these social problems. Yeah. So before we get to the downfall, I just want to talk a little bit more about the kind of work that he was doing. He did a lot of workplace wellness consulting. Oh, no. I know. This is the thing. I don't want to get into it because we need to do a whole episode on like the unbelievable

trash fire that is this field. But I just wanted you to read like one very brief excerpt. Like this was the kind of advice that was going around to workplaces at the time. This is an excerpt from his book Slim by Design, and he's talking about how he's doing a consultation with Google to prevent what's called the Google 15. Fuck off. That when people start working at Google

because there's like these canteens everywhere and the food is really good and it's free, everybody gains weight when they start working at Google. This is one of the ideas that he comes up with. I'm gonna send this to you because I cannot get through this without tittering. Okay, to tackle the, I gained weight before I knew it problem.

Bob Evans, one of their software engineers, had an idea. Have you ever seen those iPhone or Android apps that let you upload a photo of yourself and it shows you what you would look like if you were 20 or 40 pounds skinnier or a fatter. Oh my god, I hate this already. I know it only gets worse. John figured out there might be a way to have a quote unquote food scanner set up that could scan someone's tray and a camera screen in front of them would take their photo and instantly display what they would look like in a year if they ate this much food every day for lunch

way cool Is the end of that quote it's one of those Demented fucking things I've ever heard I hate every So they have no concept of people with eating disorders or body dysmorphia They have no concept of like fat people and increasing bias. I know. They have no concept of like a lot of the things that I care the most about. Imagine being a fat person at Google and somebody gets their tray out and they hold it under this fucking miserable like minority report scanner and it shows them a body that looks like you.

Like what are we fucking doing here, Brian? It really feels like there are hops skipping a jump away from just adding like an oinking sound effect or something. You know what I mean? This is like school bully shit. Oh yeah. Woof.

He also at one point in the same chapter suggests that employees should have to sign health declarations. Fuck off. Were they promised their employers that they're gonna like exercise two days a week? And that like if their BMI goes above 30, they have to be like mandatory attendance at waitwatchers. Fuck off.

I know. Truly magly deeply, fuck off. So again, we're gonna save like like, most of our fire emojis for our eventual workplace wellness episode, but like, that is a main thread of his work. The other main thread of his work is this Smarter Lunchroom's program. Have you heard of this?

I remember this because, again, like, I have a mom who's an early childhood ed person who never stops yelling about this study, which is the kids were offered. I think it was like fruit and candy. Yeah, it was apples and cookies. Yeah. Apples and cookies, and they studied essentially like,

if you just offer kids an apple or a cookie, which one do they pick? Unsurprisingly, a lot of kids picked cookies. And then they put stickers of Elmo. Yes, it was Elmo. Is it Elmo on the apples? And then they were like of Elmo? Yes, it was Elmo. Is it Elmo on the apples? And then they were like more kids, shows the apples. But then, and this is the thing that my mom never stops yelling about.

She's like, look at the ages of the kids that they're putting fucking Elmo stickers on apples for. They're like 10. 8 to 11, yeah. You know, doesn't care about Elmo? It's like a fourth grader. Man, you are in danger. Gra grave danger of spoiling this episode. Oh, really?

We will come back to this. Really, really? Yes. I mean, this is like one of the central studies that becomes the basis of this entire program. So this is by Nick Brown, a researcher who looks into this later. Here's how the study worked.

Researchers recruited 208 students at seven elementary schools. As part of their regular lunch menu, these students were already allowed to take an apple, a cookie, or both, in addition to their main dish. Before the study period, about 20% of the children chose an apple, and 80% chose the cookie. But when researchers put an Elmo sticker on the apple, more than a third chose it.

So it's like perfect Brian bait, right? It's cheap, it's easy, it doesn't require like taking away the cookie, it's just like this little tiny thing and you got more kids eating fruits and veg. It does feel very odd to be like what if fruits and vegetables were branded? Yeah. What if these were Star Wars grapes or something where you're like, well, sort of, that's fine. I mean, a lot of this actually is sort of trying to use traditional marketing techniques

for, you know, relatively unsexy fruits and vegetables. So another one of the canonical studies that's part of this program is renaming vegetables. What? Yeah, so they try to sort of brand vegetables and cafeterias to like make them cool. So there's like x-ray vision carrots. It's one of them because carrots have beta-carotene

and that helps your eyeballs. So some of the other ones, this is a list of the brands that they use for fruits and vegetables and lunchrooms. Orange squeezers, monkey phones, that's bananas, snappy apples, cool as a cucumber slices, sweetie pie, sweet potatoes, and they renamed healthy bean burritos as big,

bad bean burritos. And so, according to the studies, this actually increases consumption as much as 30%. So this basically becomes like a massive, like a massive sort of long-running program. And in 2007, he's appointed to the USDA and he starts helping them design this Smarter Lunchrooms program, which is like exactly what you would expect from his kind of work. There's a checklist of 15 different changes. And they're all sort of along these lines. So it's, you know, you add a salad bar,

but like you move the salad bar sort of in the middle of the cafeteria so kids kind of have to walk around it, right, it's not like in a corner where they can ignore it. He suggests things like, you know, you put fruit in a bowl next to the cash register rather than like this special place where kids have to go search for it. He moves the chocolate milk to the back of sort of the rack

so you have to like reach a little farther for it. It's all of these like little tweaks. It's the school lunchroom equivalent of putting tabloids next to the cash register. 100% yes, that's a very good metaphor. So I did not know this when I started researching this, but this was used in 30,000 schools. Whoa!

So these are the three main threads of his work. There's the weight loss stuff, there's the workplace wellness stuff, and there's the school lunchroom stuff. You know, he's giving talks on every continent and he's quoted in the newspaper a billion times and he's this massively famous researcher, like one of the few sort of brand name researchers in this field.

And on December 25th, 2016, the whole thing comes crashing down. Is this where we get to the scandal part? Super duper scandal part, yes. Excellent. Give me some scandal. I fucking love this. This is like one of my favorite downfalls. Really? I feel bad about celebrating this, but like,

this is just one of those delicious downfalls I've ever seen. Okay, so the entire thing, the dominoes start to fall with a blog post. Wait, Brian, wanting? Right to a blog post? Yeah, like he has a blog at the time that's quite well known. And you know, he talks about like his research and sort of their findings and just, you know, in the ways that like academics have blogs, they'll just have sort of musings on various things. Sure. And so in late 2016, he writes a blog post that

begins with three paragraphs that I am going to make you read. Oh, is it going to be better or worse than the fucking Google scanner? Oh, wait a minute. Okay, good, good, good. By the standards of our show, like this is, this is weak shit. This is fine. Okay. Like as far as the trauma meter that is always like bouncing in the red at the bottom of our show at all times, like this is green to yellow.

Okay, good to know. Okay. A PhD student from a Turkish university called to interview to be a visiting scholar for six months. When she arrived, I gave her a data set of a self-funded failed study which had no results. It was a one-month study and in all you can eat Italian restaurant buffet where we had charged some people half as much as others. I said, this cost us a lot of time and our own money to collect. There's

got to be something here we can salvage because it's a cool, rich, and unique data set. I had three ideas for potential plan B, C, and D directions since plan A had failed. Every day she came back with puzzling new results, and every day we would scratch our heads and ask why and come up with another way to re-analys the data with yet another set of plausible hypotheses. Eventually, we started discovering solutions that held up regardless of how we pressure tested them. I outlined the first paper and she wrote it up, this happened with a second paper, and then a third paper, which was one that was based on her own discovery while digging through the data. What do you think? So basically he talks about sort of bringing in this PhD student to help out at his lab.

He's essentially asking her to keep reinterpreting the data basically until she finds something. And every day she goes and reinterprets the data and brings it back to him and he goes, they're not quite it, reinterpret it again. That's not quite it, write this paper and write it differently. Yeah.

It seems real fucking wild to go back to the same data set again and again and again and go, what about this? What about this? What about this? What about this? Right. Of course, interpretation is always, always, always part of the deal when you're doing research. Right. Like, everything gets interpreted by humans. There is nothing that is like fully, fully, fully objective as we want to think there is. And there's a point at which either there are conclusions to draw or there aren't. And when you start to force it, you start to change the shape of the data itself, right?

Yes. It feels almost like photoshopping. Oh yeah. There's a point at which you're changing the contrast and the brightness, and then there's a point at which you're actually just manipulating what's in the photo. Yes.

Do you want to hear the titles of the papers that came out of all of this data digging? Oh god. Low prices and high regret. How pricing influences regret at an all-you-can-eat buffet. Lower buffet prices lead to less taste satisfaction. How traumatic violence permanently changes shopping behavior. And also, remember the study we mentioned earlier that men eat more in the company of women? That's one of the five studies that they get from just scraping this data

basically to death to find any associations in it. Right, so they didn't set out to be like, hmm, let's take a pool of people who've experienced traumatic violence and see what happens to them in a grocery store. They were like, we are doing a study in a grocery store. What happens when we look at just the people who've had experiences with traumatic violence? Basically, right? Yes. That's a big oops.

Yeah, it's bad. And it's more than a noops, right? And so the reason why I find this so delicious, this is, you know, the cute opening anecdote to a blog post that like isn't about this. The blog post is basically this like rise and grind bullshit where he's comparing this like hard working unpaid Turkish researcher to a paid grad student in his lab who refused to do this.

The whole thing is like this fucking sub-tweet of this poor woman who left his lab and didn't want to do this like wildly unethical research. So he says six months after arriving, the Turkish woman had five papers accepted or submitted. In comparison, the postdoc left after a year and also left academia with one quarter as much published as the Turkish woman. I think the person was also resentful of the Turkish woman. God damn it! It's also I will say in this blog post, it feels extremely wild to watch someone commit career suicide without knowing that that's what they're doing. Right, I have no idea.

It's really something. But so, are you familiar with this term, Peahacking? It's one that I've heard. I don't totally, like, my understanding is that it is sort of this general practice of like, you interpret the data so much that you start to manipulate it. Basically, yeah. I actually think that a better term for this is

Harking, which stands for Hypothesis after results are known. Oh, that's a great acronym. It's good and it's a good word too. You can say like Harking at the moon and stuff. This is basically what Brian is describing here. Where it's like, you've gathered all of this data, the central question that you're trying to answer, you didn't get the result that you wanted or it's inconclusive or whatever. And so basically,

you just start like systematically going through your data and being like, well, what about, you know, men eating with women, what about like older people and pizza, what about salad? You just start going through it and being like, well, is there anything else here? This is a problem in science generally, but it's especially a problem in nutrition research. I think that's something that people don't really know or like haven't really internalized, is that there's essentially no way to research nutrition.

Because you can't really induce diet changes in people in any sort of scientifically robust way, like this is why essentially every study that compares like the Atkins diet to the Ornish diet, none of them actually find interesting results because nobody can stay on these diets very long. Right, almost all of those studies are like, there's like a little section

where they just sort of mentioned briefly, like 70% of the people on this diet dropped off. Anyway, the results are blubblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleblebleble giant cohorts, and they'll ask them a bunch of questions. Do you eat blueberries? Do you eat apples? Do you have cancer? Are you tall? Are you short? Are you red-headed? And then you can publish the associations that you find. People who eat breakfast every day, like way less than people who don't eat breakfast every day,

like every day you can find a study coming out that is like along these lines. Right. People who ate full fat dairy as a kid are more likely to be thin in adulthood. Exactly. That is like blows people's brains out of their heads A great full-fat dairy as a kid are more likely to be thin in adulthood. Exactly. That is like blows, people's brains out of their heads and is one of these sorts of associations

where you're like, okay, but what else does that mean? Exactly. There's a really good series of articles by Christi Ashwandan at Buzzfeed, who sort of does a deep dive into like the way that these large scale studies are done, and the sort of the main thing to know about these huge survey studies, where they're asking people about their health conditions and about their weight, is that the data is total trash, because there's really only two ways that you can get information from people about what they're eating. The first way is you do these 24 hour recall studies. You keep a diary for a day, and then

you write down, like, today I had a sandwich for lunch and then like I went to McDonald's for dinner or whatever. But of course the problem with that is that first of all the minute you start keeping a food diary you start eating differently. Yeah that's right. Like if somebody tells you to write down everything you're eating you're probably gonna eat better that day. And even that Christy and her article talks about like she goes to an Indian place and eats like a curry for dinner and she's like well how many calories was that? How many grams was that?

What were the ingredients in that curry? None of us know like the weight of what we're eating or whatever. Totally. And even when you do have sort of like, straight ahead kinds of foods that you're eating, even if you're like eating a stock of celery, the difference between a small stock of celery

and a large stock of celery is subjective, right? So, you know, 24 hour food diaries are problematic in their own way. And so a lot of studies will do food questionnaires. They're called food frequency questionnaires, which is like a pretty standard methodology for these kinds of studies that include all kinds of, you know, a huge battery of questions

about like how often in general you eat various foods. Uh-oh. As you are probably guessing, this is also super problematic. So this isn't excerpt from Christie's article in Buzzfeed where she actually took one of these questionnaires. Some questions, how often do you drink coffee, were straightforward. Others confounded us, take tomatoes.

How often do I eat those in a six-month period? In September, when my garden is overflowing with them, I eat cherry tomatoes like a child of ours candy. But I can go November until July without eating a single fresh tomato. So how do I answer the question? Questions about serving sizes perplexed us all. In some cases, the survey provided weird but helpful guides. For example, it depicted what a half cup, one cup, or two cups of yogurt looked like with photographs of bowls filled with various amounts of wood chips. I don't know why they didn't just use bowls filled with yogurt, but whatever.

It seems really odd to choose wood chips. It seems like an easy one. You know what it is? It's the commercials for tampons and pads that are like, it's just mid-s free blue liquid. Yeah, you know that type of one? When you, like, just, there's a bunch of windex that shows up. You're underbast? This is that. Other questions seemed absurd.

Who on this planet knows what a cup of salmon or two cups of ribs looks like? I noticed that when I was offered three choices of serving sizes, my inclination was to pick the middle one regardless of what my actual portion might be. There's quite a few studies of, like, how bad these are. Like, in one of these large cohort studies, they found that people were underestimating

their calorie counts every day by as much as 800 calories. Whoa! And so it basically makes like all of these comparisons are completely invalid because you can't say that you know, blueberries prevent glaucoma or something if like you don't actually know how much blueberries people are eating.

This feels like an inroad to the studies that we're sort of constantly getting on foods that are sort of like controversial nutritionally, right? Like cranberry juice is really good for you. Know what's really bad for you. Draft chocolate, have it every day, never have it. Oh my god. Red wine, drink it for your heart health, but not after age 70 or whatever the things are, right? That we're constantly getting conflicting information about a handful of things like eggs were this way for a long time. Fuckin' eggs, man.

Again, it makes it feel like it is impossible to know, you know, as a consumer, what you should and shouldn't be eating. And actually, the answer here is to be more transparent about like, it's really hard to find conclusive findings. Yeah, we just don't know. We're not good at that. We don't have media systems that are especially good at

sort of display things as take it with a grain of salt. Yeah. They're just like a bunch of systemic and individual problems. Also, I would say like, we're asking people to do this thing, which is estimate the size and amount and weight and caloric value of things without really having any training and how people should do that. Generally speaking, we're all pretty bad at that and we shouldn't actually get good at

it because when we get good at it, a lot of us develop eating the same. Yeah, no kidding. She has a really interesting section in her article where she talks about, you know, because this questionnaire includes 54 questions. This is where P hacking comes in, right? Because you're getting 54 variables. And she says, the food frequency questionnaire we use produced 1,066 variables, and the additional questions we asked, sorted survey takers according to 26 possible characteristics. This vast data set allowed us to do 27,716 regressions.

Holy shit. So according to the data that they collected, people who eat cabbage are more likely to have any belly buttons. People who eat shellfish are more likely to be right-handed. People who eat more fried fish are more likely to be Democrats. And people who eat bananas have higher scores on the SAT verbal. Yeah, I ate a lot of bananas. These are statistically significant results, by the way.

Like these are all technically publishable. I also think like part of the backdrop of findings like these and interpreting findings like these and people just sort of running with them, is this desire to believe that scientifically we have arrived? Yes. Right, that we sort of like know everything that is knowable.

We have reached the pinnacle and we are there now. And now we're looking out onto all of the world as it is, right? Rather than going, wait a minute. Sometimes people are not as meticulous as we want them to be. Or sometimes we put wishful thinking into our science. Or sometimes their ship we don't know. And techniques that are developing now that will help us in the future.

Instead, what we get is a conversation about science that is like, the science says this, therefore it's true, therefore you gotta get smaller plates or something where you're like, well, that's not the whole picture. I also think that one of the fundamental misunderstandings here and this comes up so much is that when you hear that something is a significant result,

that makes you think that it's big. Like if I say, watching legally blonde had a significant effect on my life, you'd be like, okay, it's a big effect. But the terms statistically significant, all that means is that it's unlikely to be due to chance. This is from a nature article about this. Critics bemoan the way that p-values can encourage

muddled thinking. Last year, for example, a study of more than 19,000 people showed that those who meet their spouses online are less likely to divorce than those who meet offline. That might have sounded impressive, but the effects were actually tiny. Meeting online nudged the divorce rate from 7.7% down to 6%. So those are statistically significant results, but people who met online have a 1.5% lower divorce rate is not that interesting. And also, if you're using a site like, say, e-harmony, where they're like, it's science.

We're matching you based on science, that that also like increases your buy-in to the method of meeting and makes you feel like it's somehow more legit. Yeah. Again, there are so many variables here that could account for these differences. And when you just say people who met online have, you know, or less likely to get divorced, people are like, oh, I should be looking online, right? Rather than going, well, wait, what does that mean? Yeah. I mean, this also comes up a lot in sort of anything involving mortality that you

always hear these things. Like eating nuts reduces your risk of prostate cancer by 40%. Yeah. And then you look at the actual numbers, and it's like, you know, you have a three and a hundred thousand chance, and that goes down to like a two and a hundred thousand chance. It's not clear to me that I need to change my dietary habits to make this extremely rare thing, like slightly rarer. Totally. And I think in our brains, right? Because most of us are sort of accepting this news pretty passively and again, pretty uncritically. We hear that as eat nuts three times a week and

you definitely won't get prostate cancer. Exactly. Part of this sort of breakdown happens in the research itself. Part of it happens in how the research is presented in the paper. Part of it happens in how the research is interpreted and reported in media. And part of it happens, sort of at the point of consumption, which is the point that we hear it and sort of translate it into what we're going to do in our daily lives. There are breakdowns at every step along the way in this process. I also think, you know, the fundamental point about all of this research, too, especially these large, you know, surveys, is that they don't show causation, right? That they're very limited in what they can show. All they can show is associations, you know, there's ways that you can control for poverty,

you can control for education. I'm a little skeptical of that sort of how much statistical controlling you really can do. But the fundamental fact is that all you can find is associations and oftentimes those are measuring a third thing. There's probably something independent that is affecting how many bananas you eat

and your score on the SAT verbal. Right, bananas as a snack are also sort of like speaking to a really specific sort of like racial class and cultural background, right? That like if you're in an immigrant family, your after-school snacks might be a different thing. That doesn't make you less likely to do well when you're SAT verbals, right?

Like there are other sort of factors at play here. Yeah. It feels really challenging. I find myself getting really angry. It's we're talking about all that dude. Because it's like a huge weird like not intentionally. Well, I don't even know maybe intentionally like it's a grift economy. I mean, this is this is what's so hard about this is because I think for people of good faith, if you're actually trying

to find out sort of which nutritional habits are the best for promoting health, there's no perfect way to gather that information. There's no good way to answer that question. Most of the people in this field are trying to kind of triangulate zigzag their way to real answers, but the problem is that these methodologies, the gaping holes in these methodologies, leave them vulnerable to grifters, and also vulnerable to the incentives of science. So one of the things that is really important in Brian Wonsink's story is this idea that you have to publish, right?

If you want to get tenure, if you want to get noticed in your field, you have to publish as much as possible. And for a lot of people, if you've gone to all of this work to gather this data, you spent months watching what people are doing at a pizza buffet. It's like, well, fuck, I can't just get rid of this. I spent a ton of money. There's grant money

on the line. Yeah, I mean, I also feel like because of the ways in which sort of capital S science, right, has been used as a political football in recent years, particularly around climate change, particularly around, I mean, COVID is a great example. Do you believe the science or do you not believe the science? So there has become this sort of reaction on the left to be like, we believe in science, which means that we sort of like accept a lot of this stuff uncritically, right? That we sort of slip slide into this mode of just like whatever science says is

the truth without really a recognition of, you know, what most researchers and most scientists will tell you, which is that science is like a series of very active participatory conversations. Right. Like, the point of science is to figure out things we don't know. And it's a process. And it's a process. And so this actually brings us to the original pizza gate. This is, this is not comment pizza. No. This is not Q pizza gate. This is not comment pizza.

No, this is not QAnon. This is a different pizza-related scandal. This is just a clickbait title that I'm giving this episode. Let's be clear. So after this blog post comes out, the comment section is an absolute red wedding. Everybody in the comment is like, this is why I left science.

What you're describing is like, exactly the problem with science. So there are four grad students, kind of random people, they're not like official investigators. They're basically just people who read this blog post and they're like, this dude sucks. Their names are Tim Van Z, James Heathers, Nick Brown, and Jordan Anaya. And these four dudes dive deep into Brian Wonsink's work. And the first thing that they

dismantle are these pizza buffet studies that he was talking about that he's sent to this Turkish researcher. So one of the first things that they find is in these four studies that are all based on the same data, they have a total of 95 references to Brian Wonsink's other work. You know, in the literature review, they'll say, like, blah, blah, blah, blah, we know larger plates, blah, blah, blah, blah, right? And yet, none of them have any links to each other. And none of them even mention

that there's been any other publications with this same data. Ah! So that's already like, kind of a statement that like we know what we're doing here. This is like an active choice to kind of bury the lead. Yes. So the other main thing that they find in these pizza studies is weird statistical irregularities. Uh oh.

We have to do some math to understand this. Okay. I'm going to try to make this as simple as possible partly because I'm not sure that I understand all of this, but I'm going to try to present it as well as I can. So basically, imagine if you had like a sample of 100 people and you're trying to figure out, like do people like broccoli, right? Yes or no?

And you're surveying them and the only two options are yes or no, they can't say I don't know, right? They have to say yes or no. If I did that study and I came to you and I said 32.5% of people like broccoli. That's an impossible number, right? If I'm serving exactly 100 people, there's a finite number of sort of results

that I can get, right? Right. So if I say 32.5% of people, that would mean that 0.5% of a person is in that day. Yeah, that's right. Does that make sense? Right, if you're doing a study of three people, which you wouldn't, because that's too small, your options are 33.333%, 66.66% or 100%.

Exactly. If it's two people, your options are 50% or 100%. Yes. That's better than my broccoli thing. Exactly. And also, even if you have like 672 participants, there's going to be a finite number of options that you can have. Even though it's going to be a much larger number of options, it is also going to be finite.

So one of the things that they find when they start going through these pizza studies is that a lot of the numbers are impossible. For the sort of the regret data, people are rating their regret on a one to seven scale. Right? I don't regret it or I regret it super duper much. And one of the samples, like I think it's people who ate more than three pizza slices, there's only 10 people in that tranche. So if all 10 people save seven,

I regret it the maximum amount, right? To get the average, you divide it by 10, right? Because that's the number of people. A total of 70, a score of a total of 70, divide that by 10, you'd get seven. If everybody says 7, but one person says 6, you'd get 69, and then you divide that by 10, and that'll be 6.9. And you can keep going all the way down to 6.8, 6.7, right? And it's basically like, you'll get a two-way one-tenth of one-percentage point. Exactly. No matter what. Yeah. You can't get pi out of that. Yeah, that's right. That's right.

So what they notice in the tables of like the published tables of these pizza studies, is that one of the averages is 2.25. What? And another one is 3.92. Are they then just fully making up numbers? This is a thing. To this day, we don't know exactly what happened. The only thing that makes any sense is that they're doing this from different sample sizes. Like maybe they were using 11 people for these calculations

and forgot to replace them. But what this indicates is that it's like, p-hacked to fucking death. It is fascinating to me that all of this past peer review with like, dude, very simple arithmetic issues. I know. Right, like this isn't even like, how do you interpret data blah blah blah blah blah? This is just like, do all the numbers

in the first four columns add up to the number in the fifth column? No? They eventually find 150 mathematical impossibilities. Another really weird thing is that the sample sizes don't add up. So, you know, if you have like a hundred people in your study and then it's like,

people hate no pizza, people hate one slice, people who ate two or more slices. That's your whole study, that should add up to 100, right? But it doesn't add up to 100. Right, that happens when like, I'm thinking about this sort of like the data analysis that they're doing being a little bit like Hansel and Gretel style that like,

yes, they didn't quite leave the trail of breadcrumbs to get them back out. Yes. So basically after Pizza Gate, after they look at these four papers, everybody starts looking at this guy. Yeah, for sure. So James Heathers, one of the guys that's actually

looking into this, designs a, it's called Sprite, I forget what it stands for, but it's this statistical tool that can actually reconstruct results. Whoa. And so they start going back through his papers. Remember the popcorn study? Yeah, yeah, yeah, the bad tasting popcorn. That one falls apart.

There's like numbers in there that should be impossible. To be fair, that one is like, they say that that's not as bad as some of the other ones. There's one where it's one of the workplace wellness ones where if you're sitting near a candy dish, you eat more than if it's like placed far away from you, that one falls apart. Another one that gets debunked is the bottomless bowls study. It just has a lot of data in it that like literally is impossible. Like there's no way to get those averages and standard deviations from real data. It's so tricky because none of this means

that any of this is definitely true or definitely untrue. Yes. There's a bunch of stuff we thought we knew and we don't actually know. And it's gotten so far out into sort of the public, you know, imagination and into our sort of collective bloodstream that you kind of can't unring that bell, right?

By the time people sort of just sort of start to intuitively believe and understand that men eat more pizza in front of women, you can't be like, well, actually, mathematically, those findings were blood. Right. You know what I mean? Like, that doesn't mean anything to anybody. Yeah, and also, I would not be remotely surprised if it turned out that if a candy dish is near you, you end up eating more candy during the day.

Like, I think that that's extremely plausible. I also think that like, this is the whole point of science. Is to actually confirm things that seem plausible. Like, it seems super plausible that the sun rotates around the earth. But then, you look into it and you're like, whoops, it seems like this very common sense plausible thing turns out not to be true. This is the episode when Mike comes out as a flat-erather. I know.

Listen, there is a glass dome. There are flat edges. Why is there a horizon? I'm just asking questions. Did you not know that I was a tolamie stan? I like other lutey loops. And then it's like from this mathematical stuff, it's just like a non-stock avalanche of like

ethical weird stuff. Like there's a huge amount of self plagiarism, like entire articles that he's basically repurposing for book chapters and vice versa. Like tons of that stuff. There's my favorite one is apparently he wrote some weird article about World War II veterans and sort of like how trauma affects their eating habits, something, something. And in his sample, this is like right there in black and white, in his sample, 20% of the participants are women. The whole thing is based on combat veterans and like trauma among combat veterans. And it's like, well, women didn't fight in World War II. We're doing a study on the Tuskegee Airmen, and in this sample, we're gonna talk through

the 50% of them that were white women, and you're like, what? No! No! And so, we also get in this like wave of downfall stuff. We also get the complete collapse of all of this school lunches shit. What happens with the school lunch?

Oh my fucking god. Is it still in play? Like what? The school lunch stuff, the program is now defunct. I mean, I think that like whatever, there's probably schools that have like bowls of fruit next to the cash register. Like I think that some of the principles

are probably still in play. Sure. But as a sort of USDA program, it's gone like their website, you can only look at it on archive.org. Like the thing is toast. There's a researcher named Eric Robinson who starts looking into this stuff and The first thing that he finds it's actually like pretty bad and pretty bad that nobody noticed this before But they started implementing this program in 2014 the first randomized control trials of these principles

Didn't start until 2014. So basically when they implemented this program, they had no data. One of the things that Eric Robinson also notes in his paper is that, you know, there's these 15 strategies that schools were supposed to be implementing. It appears to this day only six of them have ever actually been studied. Yeah, that just seems wild. And also, I will say, like, there is already so much that we expect out of schools, right? Like, we are expecting teachers and school administrators

and school staffers to carry the weight of so many of our social anxieties. Oh my God. And this also becomes another one that they're now holding. Yeah. That But like all of our weird shit around health and weight and whatever that kids generally don't have in the same way. This is like part of how we ensure that kids pick that stuff up, right? And that we sort of projected onto them. And also, I mean, this was the thing that I was biting my tongue to keep from saying earlier

when you were talking about the Elmo stuff. So the Elmo study, choose an apple with an Elmo sticker or a cookie, right? Yeah. It turns out the study wasn't on 8 to 11 year olds. It was on 3 to 5 year olds. Well, that makes more sense. Right. You sniffed it out.

Millions of researchers did not sniff this out, Aubrey. You knew. What? I mean, truly, like anyone who has spent any amount of time around a nine or 10-year-old can tell you that even if they still like Elmo, they sort of know that they aren't expected to. So, like, this is a fucking wild error to make.

Like, I thought it was among eight to 11-year-olds, it was actually three to five-year-olds in daycares. So this has no application to elementary schools. Well and also like again, children age of 3 to 5 are in a completely different state of brain development. Exactly. And impulse control and like, it's not quite like you should be studying another species, but it is like so far off the mark.

Well yeah, I mean the whole concept of choosing food is like very different for a four-year-old versus a 10-year-old. Yes, there's also another fucked up thing. This actually shows up in a lot of lunchroom studies, I've been reading studies all week, that in this sort of this study where they made the chocolate milk harder

for the kids to grab, what happened is they took more milk. So it's like, yay, they're taking milk. It's not flavored, it's not sugared. But then they're not actually drinking it. No! Getting kids to take vegetables is a totally different thing from getting them to eat vegetables. Like it is very easy to get my nephew

to put vegetables on his plate. Yes. That's fine. He will do that without objection. When you're like, hey, you got to have a couple more bites of vegetables. He'll be like, how big of a bite? Let's fine. He will do that without objection when you're like, hey, you got to have a couple more bites of vegetables So be like how big of a bite? Let's negotiate. There's also this is one of the fucking pettiest things I've ever seen in an academic paper

You know, there are like randomized control trials that show that you know smarter lunchrooms do actually improve You know the number of fruits and vegetables that kids are eating and you know This is sort of evidence that Brian Wonsink was using for years and he would talk about it in his TED talks and like, we're doing this and it really improves things for kids. And then when Eric Robinson goes back to the data, the studies show that after all of these interventions in schools, like the biggest sort of study that has been done on this shows that kids are eating point one unit of fruit more And kids who aren't getting this intervention. And so, he has a whole page of his paper dedicated to photos of this. So he goes figure one, a small apple.

Figure two, 10% of an apple. And it shows like this lonely little slice of an apple on a plate. And he like clearly took the photos at his office. Like they're in the shitty break room. I love everything about this. That's fucking brutal, dude. You could have just said it. There is a real special place in my heart

for the extreme pattingness that is reserved for very high-minded feels. Love it. So basically it appears that all of these interventions increased the amount of fruit that kids ate by one tenth of an apple. We're not talking about like a revolution in children's consumption here. We're talking about extremely modest, to fact.

Yeah. And that was never how he was describing them publicly. It's so hard because this feels like a real like Embror's new clothes, kind of an episode. Yeah. I mean, this is honestly like my biggest revelation from this is that, you know, there's the statistical stuff There's the p-hacking stuff, but the worst thing that he did was like out in public Yeah, like one of the things Eric Robinson mentions is that, you know, he would have these studies

Where you know, we find kids are eating point one unit of apple more and then in the conclusion of the article He'd be like this is an effective intervention for childhood obesity. God, fucking dammit. And you're like, well, it's very evidently not. You have to line up so many dominoes before you can sort of flick one and have them all fall the way that you think they should,

right? And this guy essentially set up two dominoes and they didn't make it past sort of the finish line, right? And he was like, did we do it? There's another example in here is that there's articles where in the abstract, he says that, you know, giving kids pre-sliced fruit, increased fruit consumption by 71%. And then you read through the paper and it actually increased it by 4%. If you and I as people who are not, we don't have math degrees, like if this stuff jumps out to you and I

just at face value, right? Like that feels like, uh oh. It's not good. It's not great. It's not great. So this is actually sort of like the next stage of the downfall. There's sort of like this year long period

where like the pizza gate stuff is happening and there's statistical analyses and there's a lot of the sort of behind the scenes are like intra-academic debates about this Brian Wonsent guy, but it hasn't really bubbled up to the surface. Like, sort of normies were not really noticing this. And the next stage, this is so weird. The next stage of this downfall basically happens with a tweet thread by the joy of cooking

Twitter account. fall, basically happens with a tweet thread by the joy of cooking. Oh God, I really love the idea of a nutrition researcher getting owned by the joy of cooking. Fucking owned by my grandma's cookbook. So do you remember the joy of cooking, like, study that he did? Sure. They had, of course, seen his study. They'd seen it like, it's been cited more than 30 times. It shows up in media reports.

It's something that like everyone just sort of mentions and like cute little parentheses whenever joy of cooking comes up. They're like, lol, the portion sizes are so much bigger. Mh. There's this great New Yorker article by friend of the show Helen Rosner,

who interviewed people at joy of cooking about like, what it felt like to be the target of this study and sort of trying to debunk it for years, and the world wasn't ready. Finally, after things start to domino out of place for Brian Wonsink, they put out this thread saying that first of all, the entire idea of his original study was that, you know, there were all these different recipes throughout time, he could only find 18 that appeared in every single joy of cooking to compare to each other.

The joy of cooking is like, we found 275 recipes that were in every single edition. And then they do this, like, this whole thing about, it's kind of absurd to say that portion sizes have increased when most recipes in the book aren't meant to be eaten all at once like one of the recipes that He sort of says like has gotten bigger over there's this gumbo It's like a big-ass pot of gumbo. Yeah, I mean it feels a little bit like look my reference point for this whole gumbo. Yeah, I mean, it feels a little bit like, look, my reference point for this whole section is just going to be the New York Times cooking section and the comments on those recipes. Have

you ever looked at the comments in the New York Times cooking section? Oh, aren't they like, I replaced the chicken with beef and the rice with styrofoam chips. Like, it's always like, I modified it. And it was not very good. One star. You're like, whoa, come on, dude. And that feels like the part of the challenge here, right? It's like, you can't and don't know. Like also, hey, Brian Wonsink, you love your plate sizes so much. How big are the plates that people are eating the gumbo off of?

Yeah, good lord. That study was basically trash. Yeah. This is sort of the overall frustration. There were dozens of articles written about this. They're like, joy of cooking portions are getting bigger. Like, what? It's just not a story that means anything,

and it never was. Whoa. So, the final chapter of this downfall, and I think probably the most important one, is a reporter named Stephanie Lee at Buzzfeed, does a public records request. You know, a lot of the people that Brian Wonsink has been corresponding with are employees of universities, and universities are public institutions, and you can foyer them.

And so she gets this huge trove of emails from New Mexico State University where one of his collaborators works. And she basically finds that all of this was totally deliberate. What? He was basically running this lab as like a publication factory in a very explicit way. There's all these emails about sort of the same way that it was with the pizza study

where it's like they've gathered all this data and then whatever grad student gathered it has moved on and then he'll assign like another graduate student to sort of mess with the data until they can get something publishable out of it. And so he says in one of the emails, a lot of these papers are laying around on our desktops and they're like inventory that isn't working for us.

We've got so much huge momentum going, this could make our productivity legendary. In one case, there's this data that's like how people shop in grocery stores where they don't speak the language. And it's just like, isn't all that interesting? And they didn't really find like nothing jumped out of the data.

And he's like, oh, we'll keep looking at lower and lower tier journals until you can get it published. This is such a weird capitalism forward approach to like scientific findings. It's really odd. That feels really odd to me. And also media forward. Yeah.

Because in a lot of these emails, he's also, you know, people will come to him with ideas and he'll be like, ooh, that'll definitely go viral. Oh. So this is a very explicit goal of his. He's even at one point training his grad students how to pitch their findings to the media. Like they'll have little sessions where they'll practice

their like elevator speeches to media sources. Which isn't in and of itself nefarious, right? Like you practice elevator speeches for all kinds of stuff But if again if that is the focus you're not gonna have findings that are as useful or as solid as you would want them to be and Eventually the Cornell Sun the newspaper of the university Interviews his former grad students and they say that like they felt really uncomfortable with the way that they were pressured To manipulate data to frame things for the media to try to get these things to go viral. It seems like this was like a very well-known problem among people who worked with him.

Yeah, and there's like this intense power dynamic there, right? Yes. That he is like a nationally renowned researcher. You are a grad student or an employee who's working for him. This is the person you would go to for a reference. Exactly. There is not a neutral power relationship here. I had a bunch of other studies that I was going to debunk, but like I think you get the idea. Yeah. Most of the stuff, most of the stuff doesn't hold up. And I feel like there's the sort of,

there's the debunkable articles, but then I think the much bigger problem with his work is this thing of him deliberately framing articles to get media play. And a lot of these articles, a lot of them are just fucking dumb. There's a really interesting article by this child marketing researcher guy. This is way before this sort of the scandal happened. This is in 2014 who looks into Brian Wantson. Remember the study on cartoon eyes looking downward at children?

This actual marketing researcher writes this long essay who's like, this is dumb. Like this is not a real study. You can't measure the angle of eyeballs in cartoon characters. He includes photos of the trick's rabbit. The trick's rabbit is looking upwards, but his eyes are tilted down.

And he's like, well, does this count as down or straight on? You can see him sort of sputtering in the text. He's like, this is dumb. I'm like, why was this published? And of course, the reason why it was published was because you get media out of it. Yeah, and I could totally see as a marketing person that he'd be like, we're not doing that.

We're doing other things, but we're not doing that, we're doing other things, but we're not. Exactly. We're manipulating children in other ways. You're making us look evil in the wrong ways. So, in 2018, he's pushed out of Cornell. In September of 2018, the Journal of the American Medical Association retracts six of his papers.

These grad students, there's like a running tally of all of the articles with plagiarism and conclusions that aren't supported by the data, et cetera. And it's up to 52 publications. Whoa! Some of them are self-plagiarism, which I honestly don't put in the same category as faking your data kind of stuff. But also, they don't include the papers

where they just shouldn't exist, like the cartoon study. There's been 13 articles have been officially retracted and 15 of his articles have been officially corrected. Out of how many articles altogether? So he actually says this on his website. He has a BrianWonsink.com. He says 7% of my research articles were retracted. Oh, brother.

So that's his little way of like downplaying, like how severe this is, but it's also, it's not clear how many articles people have looked into, right, the denominator is not his total body of work, the denominator is how many did people investigate. Mm-hmm. Because I noticed that none of these debunkings related to his work on workplace wellness.

So I was like, okay, Brian Wonson, workplace wellness, just Googled it, found a random paper, and then the sample sizes didn't add up. In the first paper that I looked at, and it also had conclusions that were not remotely supported by the data at all. And again, the first one that I looked at.

So I think that like, I think people just got sick of debunking this guy's papers after a certain point. But I think that like a much larger number of them also wouldn't hold up to scrutiny if you took a look. Yeah, I mean, I think that makes sense. Also, I would not be surprised if he was like, 7% of my papers have been questioned or discredited or whatever.

And then you like look at the data set and you're like, it can't be 7%. It's not not exactly impossible, Brian. Have you learned nothing? I think it's worth looking at him and his work very closely. And I also think in the same way that his work encourages us to look at folks sort of on an individual level. And there's an impulse to resist there. I think there's an impulse to resist here, which is only looking at him and not also looking at like, hey, all of this stuff passed

muster for all the systems that we have. Oh, yeah. This went through the entire peer-review process. This got published in the Journal of the American Medical Association, the New England Journal of Medicine, like whatever the biggest journals are, right? So that also feels like a really challenging layer to add on to all of this, is that this isn't an exception, this isn't someone who skirted the system, this is someone who went through the system, and this is what came out. Exactly. I mean, I think there's a huge media story here too, in that the New York Times should not be quoting anybody 60 times. That, just to me, is just a huge red flag

when you're going back to the same source over and over again, because then you have kind of like this mutualistic relationship between the journalist and the researcher. As a really interesting podcast series on the Dr. John Barardi show about sort of the, you know, scientific debunking as an institution, they said something really interesting. They said that in fields of science, where not very much is known, there's just a lot of mysteries still to solve.

It's oftentimes the most confident people, not necessarily the most knowledgeable people or the most careful people who get the most attention. And I think that Brian Wonseng really took advantage of that that he's like a good public speaker, he's handsome, he's straight and white and says, he's just kind of out in public giving good TED talks, you know, writing very pop, easy to read books. Yeah. And I still cannot get over the fact that, you know, all this stuff about like weight-laught, you know,

use smaller plates, eat one fewer candy bar a day, you lose 27 pounds. He never, it appears, even attempted to test any of this. And that's really incredible to me because his entire thesis, his entire career was dedicated to this idea that we can make inadvertent small changes to our food environments and lose a bunch of weight. Well, why not take 100 families and swap out 50 of them with smaller plates and see what happens? Yeah. You know, people can't stay on the Atkins diet for six months. People can eat off of smaller plates for six months. That's actually a pretty easy test

to do. Yeah. And yet he never even tried. You know, we talk in sort of food world and nutrition world about health halos, right? Things that seem to take on more value and almost more moral value because they appear to be healthy. There's like a little bit of a health halo effect with nutrition research. Oh, yeah. We are all in this sort of constant state of desperation for more concrete answers than the various

and sundry diet marketing that we're exposed to. And I think when someone comes along who is from the academy, who has trained in the scientific method and does something that seems official and more concrete, we sort of put folks who do that research up on a pedestal and put that research up on a pedestal in a very uncritical way.

Yes, I mean, this is the story of how far you can go if you are telling people things they want to hear. Ooh. I should also mention, I can't believe you just brought up the term health halo. Do you know who coined the term health halo? Brian Wonsink. Nah, fuck, come on, really?

Yeah, dude. Michael. This brings us to our happy epilogue. You want me to hear the happy epilogue? Let's hear it. I was totally surprised by this, actually. So in the research for this episode, I did a lot of reading on school lunches generally. Did you know that school lunches have gotten like way better? Really? Yes. So in 2010, the Obama administration passed the Healthy Hunger-Free Kids Act. And like, there's all these studies showing

that like kids are eating more fruits and vegetables, kids are eating more whole grains. They took, a lot of schools took like chocolate milk and strawberry milk out of schools. A lot of them took soda out. There's all this data now showing that the average school lunch is significantly more healthy than the average bagged lunch.

That's, I'm so glad to hear that it sounds like more nutrient dense, like foods. There are more foods with like fiber in them. There are fewer foods that are just high way high sugar. Yeah, a lot more food is made from scratch now. I mean, there's still, you know, it's obviously not perfect. It's not as good as other countries. There's still huge inequalities.

Like, it's not perfect. But I do think that it's worth noting sort of these kinds of improvements. And I also think that it's worth noting like what improves kids' health? And it's like laws and fucking money. Like one of the main things that went along with this act was way more money for school.

Yeah. I think it's like $3.38 per meal, and it used to be like $1.30 per meal. So it's like, yeah, when you give schools more money to feed kids, they feed kids better. Yeah, that's right. You know, there's all the hidden stuff with Brian Wonseng's work,

but then the more visible stuff is like, this is something he was not interested in at all. He goes out of his way throughout both of his books to be like, oh, we shouldn't take chocolate milk away from kids, because then the Levoid School lunches all together. Like, we don't want to change anything. That's too paternalistic. And it's like, no, Brian, we should take the chocolate milk away.

Like, I feel fine about there not being chocolate milk in schools, dude. Also, we live in a world of forced choices, right? It's happening all around us all the time. That is part of the central conceit of his work. It seems really weird to say we're going to stop short of forcing choices, even though all of his research is forcing choices. Exactly. The implication, the obvious implication of his work is that we should force choices, right? That these are designed environments. There's no such thing as a non-social engineered food environment.

Like, we are surrounded by social engineering at all times. So we might as well engineer good environments, especially for kids. But it's like, as soon as it came to any actual mandatory thing of like forcing kids to not have those choices, he would like completely freak out and be like, no, no, no, the corporations have to be our partners. Like, it's very clear what his actual project was the entire time. Ultimately, these are incredibly complex issues. And to focus just on these sort of like individual choice slash life hack stuff feels like very short-sighted to me.

So uh, in conclusion, don't life hack? Mmm. Write to your senator and, uh, make something from the 1936 joy of cooking tonight. Also, if there are any researchers or science reporters listening, put Elmo stickers on the good research. Thank you. Thank you.