Magic: The Gathering Drive to Work Podcast - #1129: The Question

Episode Date: April 19, 2024

Back in 2018, we ran the Great Designer Search 3 (GDS3). Part of that event was a multiple-choice test many contestants took to qualify for the next part of the qualification. One question from that test, question number 28, proved to be controversial. The topic recently returned to social media, so I decided to record a podcast about it. I talk about the history of the GDSs, the reasons why there's a multiple-choice test, and get into the specifics of question 28.

I'm pulling away from the curb because I dropped my son off at college. We all know what that means It's time for their drive to work Okay, so today I'm gonna be talking about the question I'll explain exactly what the question is and and it's relevant. So I'll give some history to it The short version of this is I do a thing called the great designer search and part of that is a multiple choice test people have to take. I will get into the details of this. On the third great designer search there was a test there

was a question on the test question number 28 that became a very controversial question so I'm going to talk I'm gonna sort of give some context for it then talk about the question and then explain my take on the question. So I'm gonna talk, I'm gonna sort of give some context for it, then talk about the question, and then explain my take on the question, of which I believe actually was a good question, and then we'll talk through it. So anyway, the reason I'm doing a podcast on this is it's come up again, it's one of these things

that just, it's this thing people like to discuss every once in a while. So I figure, since people are talking about it again again I will talk about it and give my thoughts on it in a more and I Like tweeted about it, but like here's a full half hour of my thoughts on the question Okay, so first let me give some context for those that don't know much about what I'm talking about Okay, so back in 2006 much about what I'm talking about. Okay, so back in 2006, my boss at the time was a guy named Randy Bueller. He's in the Pro Tour Hall of Fame. Anyway, he came to me, or we

have every week we have like what we call a one-on-one meeting where you meet with your boss once a week. And he said to me, he goes, look, we understand how to find developers, play designers, you know, we can go to the pro tour, that's a really good place to find people that are really good at understanding, you know, game balance and what's powerful, what's not. He goes, but I don't know how to find more designers, more, you know, people on the beginning end of the process that are making cards. How do we find designers? And so I said, okay, let me think about it because we need to find more designers.

How will we find more designers? So I came back, so my wife, Laura and I watch a lot of reality shows. And there are a bunch of different shows where the idea is, hey, I have a skill. We test people that have that skill. There's judges every week that weed it down. And then the final person often gets a job,

sometimes a project runway, probably a good example of this. These are like aspiring designers. They go through this whole process and then somebody wins a chance to sort of be a designer. So I said to him, I know this is a little out there, but what if we did the equivalent of a reality show? What if we opened it up to everybody that was allowed to, um, apply and then we, we

narrowed it down and then we had some number of people that we put through tests every week. We'd have judges, we eliminate the weakest person until we get down to three people and then we bring those three people in for an interview. Randy was like, sure. Now, we've done three of them so far. The thing about the Great Designer Search is it is a legit, like the winner of the Great Designer Search

gets a six month internship. That, at least in the day, that's how it used to work. For a long time, the way if you wanted to work in Wizards, when you got a job at Wizards, we would hire you for six months, and then see how it goes. And the end of the six months, you'd become full-time, or we'd go, thank you very much.

And so there always was this period of testing. So they go, okay, well, if you win the designer search, you essentially get, you get six months to have an seven internship show us what you can do Now it has turned out in retrospect that the great designer search has been an amazing way I would argue the best way so far we've had to find new designers the final three of each of the three Great designer searches all three people have been hired by wizards,

not all of them in R&D, but all been hired by wizards. All of them have worked on magic sets. Many of them have led magic sets. And many of the designers we have today, that's where they came from. You know, so the first one was won by Alexis Jansen. Ken Nega was there, Graham Hopkins was there. The second one was won by Ethan Fleischer, Sean Main and Scott Van Essen. And then the third one was won by Ari Knee with Chris Mooney and Jeremy Geist. All the people I've named have worked on magic.

Ethan and Jeremy and Chris are still there making magic sets today. So anyway it's been a very effective. Now the reason we don't do it all the time is A, it's a huge amount of work. B, there's a lot of you know HR and legal things. It's a tricky thing to do. It's not easy to do. But the idea was, OK, anybody who wants to apply. Now, given the prize you are winning, if you win, is a six month internship in R&D. So there were three things that had to be true in order for you to win that.

One was you had to be 18. Second was you had to be able to work in the United States, either be a US resident or have a green card. And number three is you had to be able to move to Seattle. I mean you had to come to Renton where we are. So assuming you can do those three things and I understand there's very qualified people like there are a lot of people for example that don't live in the United States that could be very good designers. Our problem is we have to follow US labor law. It is not something we have a lot of say in.

And the other thing about US labor law is hiring temporary people is practically impossible. So it wasn't something that was very viable. I do understand very talented people that just don't happen to live in the US. We have restrictions that we have, not the restrictions that we made for ourselves, but that actually exist that we have to follow. Anyway, so the idea is any number of people that wanted to apply could apply. So we needed a way to narrow down from some large number to a manageable number. So the way it works is there are three tests.

Test number one is an essay test. It's 10 questions. Test number two is a multiple choice test, which we'll get into in a second. And choice number three was a design test. I grade all the design tests, and then once I put them into three categories, what I consider to be good, okay, and not so good.

And the top category I didn't have other people look at. So I'm not making the top eight by myself, but I'm doing all the preliminary work by myself. And what we found is it's a lot of work to look through. So what we discovered is I can grade about 100 tests. So we needed to get down however many people applied because you have to write the essays, but anybody can turn in and write the essays.

And if you write the essays, usually you get it to do the multiple choice tests. We have to follow the instructions on the essays, but assuming you do that, you get it to do the multiple choice tests. So we were thinking there'd be hundreds of people that applied. GDS 1 had a thousand people apply.

So we had a 35 question multiple choice test. It turned out to not quite be big enough, meaning I was hoping that people could miss, I don't remember how many, I think you can miss three on the first one. I was hoping that people could miss a little bit more but still qualify. So in the second grade designer search we made a 50 multiple choice tests. I

don't remember exactly, there was like 1,200 people applied. So, for grade designer search three I said let's make even more. I really want to make sure we have enough questions. We went to 75 questions. Now it turns out that over 3000 people applied. I think 3056 took the test. So just so you're aware, the first great designer search set the record for Hasbro for the most people ever applying for a single position. Obviously GS2 broke that record, GS3 smashed that record.

So the idea of the multiple choice test is we don't have the ability to do something subjective. It has to be something people can take that's gradable by computer essentially. So we made a multiple choice test. And the idea of the multiple choice test is look, there are a lot of basic knowledge you have. We're just going to test a lot of basic knowledge. The. We're just gonna test a lot of basic knowledge. The goal is nothing subjective.

It's all supposed to be objective things. Do you understand? And in the very first one, very first GDS, Devin Lowe was one of the judges, made some, each time I get help with the questions, I make some of the questions, like help from other people. Devin made a few questions that were,

what I would now refer to as trick questions, which was it was less about do you know the answer and more about were you carefully reading the question. And I said, okay, none of that. We want our questions to be straightforward. We're not trying to trick people. So in the second test and third test, we made sure that there's no trick questions. So the question number, question number 28, the one, the question was written, I believe,

by Eric Lauer. So I'm gonna read the question now, and then we're gonna get into the nitty gritty of the question. So it was question number 28. Here's the question. We try to avoid making two color cards where the card could be done as a mono color card.

And only one, sorry. We try to avoid making two color cards where the card could be done as a mono color card. We try to avoid making two color cards where the card could be done as a mono color card in only one of the two colors. Given that, suppose you have a two color, 4-4 creature with flying and vigilance, no other abilities, just a French vanilla. What of the following color combinations would be the best choice for the card? A white blue, B white black, C green white, D blue black, E black green. Okay, before I go into this real quickly, when we did this question, so this question was, when was this question?

Back in 2018. When we did this question, blue did not yet have vigilance. Since this question, we've added vigilance to blue. Blue's not secondary in vigilance. Vigilance is primary white, secondary blue and green. Flying is primary white and blue, secondary black, tertiary red. So the idea was, it's important for this question.

So let me talk a little bit about the premise of the question, and then we'll get into the dynamics. So let's talk about the first part of the question, what we try to do. So we have a rule that says, if you're making a multicolor card, ideally it's good if that card couldn't just be done as in one of the colors, or in both of the colors,

although these days if you can be done in both of the colors or both of the colors although these days if you can be done in both of the colors will make you a hybrid card and not a multicolor card. Now why do I say we try to because I was trying to imply it was a soft limitation so the way it works is anytime we do a pass in the color consoles if we see a card there's a two-color card that could just be done for one of the colors in in the card, we will call it out. Now, we try to avoid it and the reason I say we try to avoid it is we don't. It's what we call a soft rule, meaning, you know, if there's a strong reason, like it turns out it is very hard making multi-color cards especially when you're trying

to make them kind of elegant and not wordy and a lot of times adding in you know well it could be done one of two colors trying to make that not true can make it can cause extra problems so the rule is when we see it we mark it and we try to have a good faith effort of people who are doing it these are are the set lead, of saying, hey, let's avoid this if we can, but we do let some through. And because I didn't want confusion, I didn't want to state in the question that we definitively do this when people can see that sometimes we don't. That's why I said we try to. I'm like, look, it's a soft rule. As a general rule, we do in fact try to do this rule, but it's not always carried out.

There are other things that are more important. So sometimes when we're making a card, we go, okay, yeah, this could be done in one of the colors, but it really makes a lot of sense for the faction we're doing. There's usually some reason why we go, okay, it's okay. But the rule I'm talking about is an actual rule. It is something we actually care about. And it's something we try to avoid when we can. We don't always avoid it. That's why I say we try to, I was trying to imply soft rule. This is something I thought if I applied it as a hard rule, it would confuse people because

you could see examples where that isn't true. And I didn't want people to feel like we're giving you a premise that is lying. That is why I talked about, we try to, this is something we, a goal we aim to achieve, but we don't always, there are reasons sometimes that we don't. And the idea behind the question was that I, I was trying, the question is testing two things.

One, a lot of, so let me get back into the multiple choice part of the test for a second. A lot of people apply, like I said, 3000 people applied. I can only grade 100 tests. So I have to get from 3000 people to 100 people. And what we found with a multiple choice test was a multiple choice test is pretty good at just testing, do you understand the basics? And a lot of the questions on the test are more like, what color can do this? You know, what rarity would this card be? I'm just saying, it's a lot of, hey, have you been paying attention? I read a lot of articles and do a lot of podcasts and answer a lot of blog questions. I put a lot of material out there explaining our ways and how we do

things. Are you paying attention to that? Look, we have 3000 people and our goal is, I have to get down to 100. Now, I should state this. I understand that anytime we do some objective setting, there are people that are gonna fall through the cracks. Are there people that don't do well in the multiple joint sets that might be amazing designers? There are. The problem is that in this particular process, we need some way to get there.

It is already a laborious process. The great designer search takes a lot of time and a lot of energy. And what we have found is the system we've created does a good job of finding great designers. Does it make sure that everybody's a great designer makes it through? No, it does not. And when 3,000 people apply, so it turned out in order to cut to 100, you had to get 73 of 75. That cut to 94. I think if you did 72 or 75, it was like 170, it was way, way over what I could grade. And so we ended up having to cut it 73. I feel bad.

73 or 75, like I would like to have a little bit more latitude than that. If I couldn't do it over again, I'd probably do 100 question tests, just so there's a little bit more room for people to miss things. 73 or 75 is very tight. And I do want to stress, one of the reasons I think that this issue becomes so large is for a lot of people, it's a dream job to work at Wizards in R&D. It's my dream job. I get it.

I understand why it's other people's dream job. And there's not a lot, if you're Joe, average Magic player, there's not a lot of ways to do it. I mean, the main entries in are like, you design elsewhere and do a good design, you know, like you work for other companies or You do well in the pro tour. They're not things that are just really easy to do This is the one thing we do we're like hey You could enter and you just you show your skills and you could you can get picked There are people that work now on on magic that before this thing were just random magic players

There's not a lot of opportunities for just random magic player to have that opportunity to become a, that's why the Grand Designer Search is really cool, why people ask all the time when we're doing the next one. And so I just wanna say, I get it. A multiple choice test is a brute system, but it's what we need to do to get there. And so a lot of what we're testing is,

I mean, I think the questions are trying to look at a couple different things. This question is looking at two different things. In the past, I've more talked about the surface thing. I'm gonna talk about both things today. So just to sort of get into the nitty gritty. So, okay, so let's first get into the color pie of it. So at the time, once again, this has changed, at the time vigilance was

primary in white, secondary in green. Flying was primary in white and blue, secondary in black. One of the general rules is usually when we make a multicolor card, we don't, if you're representing a color on a card, we normally don't do tertiary things. There's exceptions. We do make multicolor dragons. I mean, there are a few exceptions, but as a general rule of thumb, if you're contributing something, if you're one of the colors on card and you're contributing something, usually it's a primary or secondary ability. Okay, so given that, we're looking for a flying vigilance creature. Well, white is both primary in vigilance and primary in flying. In fact, the card we're talking about, a flying, a 4-4 flying vigilance creature, is quite famous. It was Sarah Angel.

It was in Alpha. It's a very famous card. Mono White can clearly do this. There's no question that White can do it. So the second thing, so I give you a soft rule. I say, here's the rule we try to do. I say we try to because we don't always do it, but it's a soft rule. It's something we do try to do.

It's a goal. We want to do it. There are reasons sometimes we don't always do it, but it's a soft rule. It's something we do try to do. It's a goal. We want to do it. There are reasons sometimes we don't follow it, but I say, given that, okay, I'm telling you it's a soft rule, but for this question, assume it's a hard rule. That's what the question is saying. You know, here's a parameter we have. Um, okay. Assume this parameter.

You have to follow this parameter. Now it becomes a hard rule. Given that, okay, I don't want it to show up, I don't want it to do in a combination that could be done in one color. Well white can be done in one color. You can make Sarah Angel. So white, blue, white, black, green, white can't be the right answer because white violates the essence of the question.

That leaves you blue, black, and black, green. Black at the time was the wrong answer because, well, blue could fly. Neither blue and black could both fly. Neither blue or black could do vigilance at the time. Black and green, black could fly. Green could get vigilance. Black couldn't get vigilance. Green couldn't fly. So the correct answer there was fly, green could get vigilance, black couldn't get vigilance, green couldn't fly.

So the correct answer there was black, green. That was the surface level question, was really, it's a color pie question. I'm just testing, do you know, do you know your abilities in your color pie? That's it. For a designer, that's super important. One of the things this test is trying to do is, look, do you just know the basic rules? I need, I need, you have to understand the color pie. You have to understand rarity. There's just a lot of basics. Now I spent a lot of time talking

about it if you read my articles you know I I've made an article two times on the color pie specifically they can look up stuff and when you take this test the other thing about this test is you can do whatever research you want. The way the test works is we give it to you you have the day to take the test but you're allowed to look at whatever material you want. The way the test works is we give it to you, you have the day to take the test, but you're allowed to look at whatever material you wanna look at. There's no, it's open book, I guess as they would say. You can look at whatever you wanna look at, you know.

The idea here isn't even, do you know it off the top of your head? Are you able to figure it out? That's fine. Because if I, I as your lead designer, am giving you a task, and you can go figure out what you need to that's fine You don't have to know everything on the top of your head a lot of people will but you don't have to

And that's why it's an open book test. So you have all day to take this to answer the questions and turn it in Okay, so let's get to the second layer of I don't talk about this part as much So once again, this question was written by Eric Lauer. I'm sure this was baked in by Eric Lauer into the question. So the next part of the question is one of the things that we do a lot in design is we explore new space, right? We're like, I talk about how magic's a hungry monster, we're constantly making new things, that one of the goal of magic designers is we are trying to make stuff you haven't seen before. We're trying to make new things. Now we will make old things and it's important understanding historical precedent is important, but it's probably

more important that you're not locked into historical precedents. Yeah, yeah, yeah. We've done it like the whole point of this question is if you look at multicolor cards that are flying in vigilance, the majority of them historically are white and blue. That is just what we happen to have done. But the point of the question was, of the deeper part of the question is there's no reason we can't make a black, green, flying vigilance creature. Nothing about the internal rules that we have say we can't. Now, does it feel a little weird? It might, but we need to give it flavor,

like we need to give a reason for it. But we make cards all the time that do things we haven't done before. Like, yes, we made a lot of blue, white flying vigilance creatures, but we could make a black green flying vigilance creature. It wasn't something we have made, but it's something we could make a black, green, flying digital creature. It wasn't something we have made, but it's something we could make. And the secondary skill we're looking for here is how beholden are you to historical

precedence versus making things that can be? How much, you know, if you're going to, every time you make something, go, well, this is how it's done. I'm just going to make it the way it's done that I you're not the best candidate for us We want someone to say well, I do understand that normally this is how you've made it But hey, it would work here, you know, if I give you constraints, I want you to work with the constraints and find stuff I don't want you to say like

the I the the other answers this question, where the controversy comes from is the idea that you read this question and go, I know you're giving me these constraints, but, aha, it's tricking me. The constraints are sending me in the wrong way. The correct answer is to look historical precedent and ignore what you could do and do what you have done. And the idea of the question, which is part of the way the question was built by Eric, is testing the idea of will you override what

has been done to look and see what could be done. And that is a super important design um... quality when i hire a new designer i mean obviously i want people to understand magic's path i want to know we haven't haven't done you know i want someone to say in the vacuum hate normally this is what we do but if i give you a constraint and they say hey follows constraint i want you to follow the constraint and see what could be done

now what has been see what could be done not what has been done What could be done because a lot of magic design is finding the new and different you think is finding the oh we haven't done this but and Yes to this day other than on up we did a play test card in the mystery boosters that made fun of this, Gogari Death Swarm. One day we will make a black, green, flying, visualized creature.

It'll happen. Like magic's a hungry monster. It'll happen one day. Okay, so now let's get into the depth of this. So what happened was this came out. Every time we've ever done a multiple choice test, there are a couple questions that, like for example, this is your dream job

and you took the test and you get 72 out of 75, meaning you missed one question. You missed one question and that was the difference between you advancing to the next part and not. That hurts. I get it. I get that hurts. I understand that. And what is normally the case because of a multiple choice is the audience will latch

on to the questions they feel are the most unfair questions. There actually was a question I think in the very first grade designer search where we decided we wrote the question wrong and gave credit for two different answers. So if after we give the test, we look back and say, oh, uh-oh, you know, like if we believe there's something like once the test got up there, the next day people were discussing it, right? We sort of said don't discuss it the day of the test. And then after the day of the test, okay, it's not public.

You can talk about it. We messed up in the first grade of the search, okay, it's not public, you can talk about it. We messed up in the first gridiron search. Well, we made a question where we thought it was ambiguous. We gave answers to both questions. We said, okay, either answer will give us a correct answer. So we were aware, I mean, every time we do this, there's people that are upset for different questions. I get it.

It's something that matters. You're so close. You missed by one question. This was the question that got a lot of attention around it. There are a few others, but this was the big one. And it's the one that's resurfaced. And once again, I spent a little bit of time on Twitter. I don't know.

I'm sure I know I know I talked about it on my blog too. But anyway, I hadn't gone in depth on it. Twitter is like so many characters you get on Twitter. But the thing is, so here's the interesting thing about this question. So we got this question, we talked about ourselves. Is this question a fair question? Is it, you know, asking? We thought it was.

So what I did is, okay, so when I look, when I judge that, so once you finish the multiple choice test, you do a design test. I put you in three categories, sort of the best, the medium, and the worst of the test takers. Now writing good answers and questions and answering the multiple choice doesn't inherently make you good at designing magic cards. That's why the third test is designing magic cards. Doing know, doing well the test means you understand what we've said. It means you understand color pie and stuff like that. It doesn't mean you know how to apply it. It doesn't mean you're good at design. So one of the things that was really interesting

to me was, okay, I now can look at 94 people who took the test. I put them into three different categories. I can go and look to see what questions they missed. Is there some corollary here? So what I did is I said, okay, I'm going to graph all the questions that people missed because in order to make it to this test, you missed either two, one, or zero. There were, I think, two people that got perfect scores. No one got perfect score on the first test. One person, Max McCall, who works at Wiz. No one got a perfect score on the first test.

One person, Max McCall, who works at Wizards now, got a perfect score on the second test. I think two or three people got a perfect score on the third test. So I went back and what I was looking for was were there any questions that were indicative of whether or not you had a higher chance of being a good designer or not. I looked at the top group, what questions did they miss? I looked at the bottom group, what questions did they miss? And was there anything correlative? Meaning were there certain questions that if you missed it meant you were more likely to be in the bottom group than in the top group? And I think there were seven questions, one of which was this question

Which meant it was a correlative question Just explain that means Is if you got that question wrong? It was one of seven questions that was the highest indicator that you would do poor on the design test Now once again, there's lots of factors there. I mean my sample is not huge. It's 94 people. But it was very interesting in that when I went back and looked that the people that got the question right were on average had a better chance of being better designers and the people that got it wrong had better sense of not being good designers, um,

so Which makes me believe that the thing the test was testing like I said, there's a surface thing and there's a secondary thing I think those both were important and I I actually think it was a good question Um, I know people will butt up, like be aware every time we've ever done a GDS and there's been a multiple, and all three had a multiple choice, there's always questions people yell me about. I think the first year they were correct. There was a question that was ambiguous and we corrected that. Ever since that on the second and third test, we now test the questions. We have multiple people take it. We tweak, you know, we get feedback

We tweak we tweak questions not to say we can't make bad questions not to say But I will defend this question. I do think this question was asking something important I do think that it was correlated to important things and I Think I think if your answer is the way you answer the question was, I'm ignoring the constraint and I'm going to follow the historical precedence, yeah, there was something we were testing there that you failed at. That question really was seeing a willingness to say, here's what could be done even though it hasn't been done before.

And that the resistance that Eric built into the question, I mean, the question really was designed to say wow this is fighting some basic sort of you know If I'm really familiar with magic, it's fighting something. That's not what magic is done But that is an important skill, you know I as the head designer or lead designer of a set if I'm asking for something I really need you to push past what we have done to find what we could do. So anyway, guys, that is my two cents on this question.

Like I said, you know, with 2020 hindsight, it is interesting, I think a lot about could I tweak this word or that word, but looking back and looking at the question, understanding what I'm saying, I think people can misinterpret things that they want, but I think if you got the question correct, you understood the essence of what, and if you got it wrong, you were prioritizing things you weren't supposed to prioritize. That missing the question meant, I think, that you were prioritizing Missing the question meant, I think, that you were prioritizing the historical context over what could be done, which is part of what the question was asking. So anyway, guys, I'd like doing podcasts where I get into nitty-bitty things, nitty-gritty, and I saw this opportunity when the question popped up again, people were talking about

it. I'm sure the fact that I'm doing a podcast will inspire more talk on it. And I saw this opportunity when the question popped up again people are talking about it I'm sure the fact that I'm doing a podcast will inspire more talk on it But I got podcast to do I got topics to do so I thought there's a fun topic Anyway guys, I hope you enjoyed this and for those that had no idea what this was Hopefully this was informative to you But anyway, I'm at work So we all know that means means is the end of my drive to work the set of talking magic It's time for me to be making magic. So see you guys next time. Bye. Bye