Cautionary Tales with Tim Harford - Laser Versus Parchment: Doomsday for the Disc

Episode Date: November 10, 2023

William the Conqueror undertook a remarkably modern project. In 1086, he began compiling and storing a detailed record of his realm: of where everyone lived, what they did and where they came from. 900 years later, the BBC began its own Domesday project, sending school children out to conduct a community survey and collect facts about Britain. This was a people’s database, two decades before Wikipedia. But just a few years later, that interactive digital database was totally unreadable, the information lost. We tend to take archives for granted — but preservation doesn't happen by accident; digitisation doesn’t mean that something will last forever. And the erasure of the historical record can have disastrous consequences for humanity... For a full list of sources, please see the show notes at timharford.com.See omnystudio.com/listener for privacy information.

Pushkin I'm Paul Muldoon, a poet who over the past several years has had the good fortune to record ours of conversations with one of the world's greatest songwriters, Sir Paul McCartney. The result is our new podcast, McCartney, A Life in lyrics. Listen to McCartney, A Life in lyrics, on the I Heart Radio app, Apple podcasts,000 men, lined up for battle near Hastings on the south coast of England, ranged along a hilltop where the English forces of King Harold, they faced an invader from France,

William Duke of Normandy with his force of archers Infantry and mounted knights. At nine in the morning, the battle began with a terrible sound of trumpets on both sides. The English formed a shield wall holding holding the high ground. The Norman cavalry and archers probed for an opportunity to shatter the English line. Then, words spread among the Norman army, that their Duke, William, had fallen. A left flank crumbled and fled down the hill, pursued by the English. But then, what's this? William himself helmet raised to reveal his face roadforth and proclaimed, look at me I live and with God's help I shall conquer. His forces rallied and counter-attacked slaughtering the English would surged forward.

The battle was brutal, and as the autumn sun began to set, the final Norman assault broke the English line. King Harold was slain. The next King of England would not be English, but Norman. I shall conquer," William had said, and on Christmas Day 1066, William the Conqueror was crowned William the First of England. For the French invader William faced struggles that have his right to rule accepted. Two decades later, he stumbled upon an idea to establish his legitimacy over his vast

fiefdom, an idea that still shapes the world today. He began a grand survey of what was in that fiefdom. Commissioners set up special sessions of county courts to hear testimony about who owned what and what there was to own, from land to mills to people. There was no single hide nor a yard of land nor indeed one ox nor one cow nor one pig which was left out. The result of this great survey was the Doomsday book. In fact there were two of them, great Doomsday and Little Doomsday.

Remarkably detailed snapshots of Williams' realm in 1086 AD. It was all a long time ago, and yet it's a very modern project. If the state wishes to govern, to tax, to help the deserving and punish the wicked, that state needs comprehensive detailed records of where everyone lives, what they do, and where they came from. I'm Tim Harford and you're listening to Corsion Retails. and retail. Where they came from. Michael Brathewate came from the sunny island of Barbados. In 1961, Little Michael, just nine years old and traveling with his brother, crossed all the way over the ocean to arrive under the grey skies of Britain.

Despite the weather and the risk of a hostile reception from a mostly white British population, it was a journey many other people had taken. At the time of Michael's journey, Barbados was still part of the British Empire, and the UK government had been welcoming migrants and offering them considerable legal protections. Michael's father already had a job with a royal male, his mother with a national health service. Michael and his brother were traveling to be reunited with their parents. He must have been a daunting journey, but it was a one-way trip. Michael started attending a British school, and from that moment he recalls, we regarded Britain as our home.

Michael and his brother entered the UK with a British Caribbean passport, stamped, indefinitely leave to remain. Michael grew up and started his own family. He went on to have three British children and five grandchildren. What he didn't have was a UK passport. Legally, he didn't need one and never applied for one and for five decades, there was no problem. But Michael Brathwaite was going to discover what William the Conqueror knew. Records matter. Something else matters too.

What you record them on. In 1983, a BBC television producer named Peter Armstrong is pondering the looming 900th anniversary of the Doomsday Survey. That anniversary will happen in 1986, just three years away. So what should the BBC do about it? Make a television series? And sure they could do that? But Peter Armstrong had a better idea.

I just thought, if we were to have a Doomsday book now, how would we do it? Oh, indeed. Peter Armstrong had two ideas, both of which were ahead of their time. The first was that the whole community would be involved in gathering information. All of us who, like me, were British school kids in the 1980s, knew about the BBC Doomsday project. The BBC will be writing to all 30,000 schools in Britain to ask them whether they want to participate in collecting facts during the summer term next year.

Explained a contemporary magazine, it is estimated that 10,000 schools will be needed to make the scheme work, which would involve about a million children. It was dubbed a People's Database, a kind of Wikipedia, almost two decades before Wikipedia. There were organized surveys and questionnaires, but also plenty of cub reporting, as children went out to photograph their local area. It was a massive grassroots effort. Peter Armstrong's second idea was just as visionary.

The BBC Doomsday project would manifest itself not in some way too encyclopedia, but in an interactive digital format. The project team decided that the best way to store this vast trove of information would be on something called a laser disk. Laser disks were silver disks of plastic, similar to the also newly invented CD, but much bigger. They were a foot across like a vinyl album. The Doomsday project was trying to use Laserdiscs in a groundbreaking way.

It combined digital text and analog video, with a clever, if slightly-rubed Goldberg Esk technique that produced revolutionary-looking results. A complete Doomsday system consisted of a laziness player, a monitor, a track ball device, and a beautiful boxy beige computer with black and red keys called a BBC Master. It was expensive. Each system cost half a year's salary at the typical wages of the day. But the BBC and its commercial partners hope that the system would be the foundation of a whole new market for business and education, with a vast range of other laser disks available. That never really happened alas.

The system was too pricey for most schools, and the technology moved on as technology does, the CD ROM replaced the laser disk as a more practical way to put multimedia projects on a silvery platter. Still, there was no shame in the low sales of the Doomsday multimedia system. Peter Armstrong's project team wore years ahead of their time and had a great deal to be proud of. They had created the mother of all time capsules, a remarkably rich and detailed picture of the UK in 1986, one which future generations could consult, learn from and enjoy. Or could they? Because by 2002, when the internet age had truly arrived,

the Doomsday project had almost been forgotten. The BBC department that had created it had been shut down for years. Nobody was in charge of keeping the project alive. And so it died. The British newspaper The Observer reported, It was meant to be a showcase for Britain's electronic prowess, a computer-based multimedia version of the Doomsday book. But 16 years after it was created, the 2.5 million pound BBC Doomsday project has achieved an unexpected and unwelcome status. It is now unreadable. Old documents have a habit of disappearing. Sometimes that's an unhappy accident. You have

a laser disk, but you can't find anybody with a laser disk player anymore. Sometimes it's all too deliberate. The year is 2009. In a basement, in a towering government office in Croydon, South East London, there are thousands upon thousands of aging cardboard documents. Each one recording the arrival in the UK of someone from the Caribbean or from other former coloners of the British Empire. These documents are hardly ever consulted, who needs old landing cards detailing the moment

someone stepped off a boat from Barbados or Jamaica in the 1950s or 1960s? Actually, every now and again, someone did need one. If you'd arrived in the UK on one of those boats, the 1971 Immigration Act gave you the right to live in the UK for the rest of your life. You hardly ever had to prove you had that right. In daily life, nobody asked. It was only if you were, say, applying for your first British passport that you might need to prove your data of arrival, and those landing cards provided that proof.

One of the clerks who worked at the croid and office were called. Sometimes the passport office would call up, and people would say, I'll look in the basement. Still, that didn't happen often anymore. After four or five decades, most people who'd ever want a passport would surely have applied for one. And so, when an office relocation beckoned, someone higher up decided, let's get rid of those old bits of cardboard that take up too much space. The clerks protested, couldn't we at least make digital copies? But the bosses decided there was no money for any of that, and late in 2010, the landing cards were destroyed.

A couple of years later, the government introduced a new law. They wanted to get tough on illegal immigration. The law required landlords to check for proof that their tenants had the right to live in the UK, and banks to check their customers, and employers to check their workers. In principle, these rules applied to everyone, although you employers to check their workers. In principle, these rules applied to everyone, although you have to wonder if they were more tightly enforced for people with foreign accents, or people who, like Michael Braithwaite, were black. More than half a century had passed since Michael Braithewade stepped off the boat from Barbados as a nine-year-old

boy and made Britain his home. He felt British, he had the permanent right to live and work in Britain, but he'd never had to prove it, nobody had ever asked. Now, they would. Corsionary tales will be back in a moment. to record hours of conversations with one of the world's greatest songwriters, Sir Paul McCartney. The result is our new podcast, McCartney, A Life in lyrics. Listen to McCartney, A Life in lyrics, on the iHeart Radio app, Apple Podcasts,

Amazon Music, or wherever you get your podcasts. music or wherever you get your podcasts. For more than two decades, Michael Brathwaite had been a teaching assistant in a school in North London, working particularly with young children who had special educational needs. He loved his job, and he loved the kids. We grew a great bond between us. He told one interviewer. One day, Michael was summoned to see the head teacher. The head looked scared, Michael remembers, his mannerisms were nervous. The head explained, he had been threatened with five years in prison. If he was found to be employing anyone he had reason to suspect, didn't have the right to work in the UK. As Michael had no passport,

he was going to need an official biometric ID card. He kept asking, what are you gonna do, Michael? Michael had felt like part of the school family. Now he realized he was on his own. I was totally confused, he said, because I never knew of a biometric card or what it meant to someone like me. It was a perplexing and bewildering situation. The government department in charge of handing out biometric cards

was called the Home Office. Michael Braithwaite, remember, absolutely had the right to live and work in the UK because he'd arrived before 1971. But the cardboard landing card that would prove that had recently been destroyed by the Home Office. Now they told him he needed to provide documentary evidence that he had been living in the UK since the 1970s, at least one official document for every single year for more than four decades. It was an impossible demand. Michael was refused his biometric card, and in 2017 he was summoned to another meeting at the school, and told that he couldn't keep his job. He'd have to leave that day.

You could have pulled my heart out and chucked it on the floor, he said. Michael walked out of the headmaster's office. The children had their weekly swimming lesson that afternoon. He helped a colleague walk them safely to the pool and then back again. Then he slipped into the school's door room, picked up a box of his belongings and left the building. Michael was not the only one treated in this way. A whistleblower who worked in the Home Office archives contacted a journalist, Amelia gentlemen, and told her what was happening.

Every week or so someone would say, I've got another one here. People were writing to say, I've been here 45 years. I've never had a passport. I've never needed a passport. Now I'm being told I'm not British. Previously, the Home Office worker would have said, I'll look in the basement. Now the archive of landing cards was gone. Instead, the officials would send a standard reply. We have searched our records. We can find no trace of you in our files. Which was perfectly true and perfectly disgraceful.

As Michael Braithwaite said, they took everything out of me, my confidence, my self-esteem, who I am, It tore me apart. It's tempting to think if only the archives had been digitized. And yes, that might have helped for a while. But as the BBC Doomsday fiasco tells us, just because something's digitized doesn't mean it lasts forever. BBC remember had embarked on a huge project in 1986 to crowdsource a unique historical record for the ages and a mere 16 years later it had become unreadable. Or had it. Let's meet Sally and Adrian Pearce, a couple of civically-minded nerds who lived in a small town to the south of London.

They were interested in community activities such as maintaining local footpaths, but also in computers and education. Sally had been one of the original community researchers for the BBC Doomsday project in the early 1980s, she had fond memories of gathering data. By the year 2000, she was working as a professor of education at the University of Brighton, teaching the next generation of teachers to use computers to access and analyze historical documents. Her husband Adrian, meanwhile, was an IT consultant. Both of them had heard about how difficult it was to access the Doomsday Laserdisc systems. Adrian was feeling burnt out from long hours patching old corporate computers. He needed a change of pace, and Sally suggested that helping to revive the BBC Doomsday system might be just the

project. Sally's vision was that young teachers in classrooms of the 21st century would be able to access Doomsday data from the 1980s, either using CD-ROMs or the internet, and Adrian had the technical expertise to try to make it happen. But a huge challenge faced Sally and Adrian Pierce because digital archives like chains, one weak link, can break them. Consider those laser discs. They had been touted as being almost indestructible, and that may be true, but without the player to read them, the laserdiscs were just glorified mirrors. I'm a child of computer engineers. I fondly remember from the 1980s the scream of the

Doc Matrix printer. My mother, with a screwdriver, attacking the weird innards of a beige computer with its lid off, fixing it so I could play classic games again. But I can tell you from experience, there are only so many times you can have hound those keys excitedly as you shoot at space bandits before the system will fall apart. before the system will fall apart. So, of course, most Lazardisk systems from 1986 had stopped working by the 21st century, or schools had thrown them out to make room for shining new PCs or Macs.

But Sally Pierce got lucky. She discovered an entire Doomsday system in storage at the University of Brighton. When they plugged it in, it still worked. Yet as Adrian explored the Doomsday system, he realised there were more weak links in the chain. He had absolutely no technical documentation. He didn't know how the data was being stored or organised by the Doomsday system. It was all in hexadecimal, picture the slow waterfall of green code in the movie,

the Matrix. How could it be translated into a format that the modern computer would understand? Adrian Pierce ended up decoding the hexadecimal structure the hard way, by hand. He later described it as like trying to solve a crossword puzzle. If the crossword puzzle was enormous and written entirely in an unknown language, after a while, like Neo in the Matrix, Pierce was seeing patterns in the data by I. Reading programming books from the mid-1980s and slowly groping his way through the rows and columns he began to understand how the data had been structured and to write new programs to search and interpret it. But Adrian also needed to solve a different problem. He needed to get all the data off the disc.

And the only way he could see of doing that was to pipe every bit of data through the old BBC computer and onto something modern. The old computer wasn't built for that. Some of the files took more than 50 hours to transfer. A few of those marathon transfers and what happened next was inevitable. The BBC Master's motherboard was fried. Adrian Pierce had started with a rare thing, a working doomsday system. But in trying to extract the data, he destroyed that system. Corsionary tales will be back in a moment. I'm Paul Maldon, a poet who over the past several years has had the good fortune to record hours of conversations with

one of the world's greatest songwriters, Sir Paul McCartney. The result is our new podcast McCartney, A Life in lyrics. Listen to McCartney, A Life in lyrics on the iHeart radio app, Apple podcasts, Amazon music, or wherever you get your podcasts. Michael Braithwite had lost the job that he loved. His own school had told him that he was an illegal immigrant. His own government had destroyed his only means of proving otherwise. Other people in his situation were being deported. It was just a matter of time then, until he was deported too. I was informed they turned up at 6 in the morning, he said.

I was already dressed and ready, waiting for that moment to come. Every morning, for 18 months, he got up early after a sleepless night, got dressed, and waited for the knock at the door. I talked about Michael Brathewade recently with my friend Patricia Sleeman. She's a digital archivist. In her early career, she worked on the kind of nerdy projects that occupied people like Adrian and Sally Pierce, trying to pull data out of unreadable old computer systems. Then about a decade ago, Patricia did something that at the time I couldn't really understand.

She went off to work for the UNHCR. The UN Agency responsible for protecting the rights and the welfare of refugees. I didn't get it. It seemed like my wonderful nerdy archivist friend was dashing off to join an emergency relief effort. What did the UNHCR need with an archivist friend was dashing off to join an emergency relief effort. What did the UNHCR need with an Archivist? When I realised what was happening to people such as Michael Braithwaite, I finally understood

that archives are a human rights issue. The UNHCR archive goes back to the 1950s, and it contains photographs, interviews and documents describing the lives and fraught journeys of some of the most vulnerable people in the world to prove where they came from and when and why the refugee needs that data or they risk disappearing into some nightmareish bureaucratic black hole. Just like Michael Braithwaite, condemned to stress, shame, unemployment, and the fear of a knock on the door at 6 o'clock in the morning. Digital archives feel better to us than paper archives. Digital files are searchable. You can back them up with a click.

You can store huge quantities of information in a tiny space. In contrast, paper archives are a hassle. They take up space in filing cabinets in rooms that could be used for something else. No wonder people are sometimes tempted to save money by throwing old paper archives away. But digital archives are also surprisingly vulnerable. While paper archives have to be deliberately discarded, digital archives can be lost without any effort at all. For example, I have Microsoft Word documents on my computer dating back to 1994, copied and recopied every time I bought a new computer.

I always assumed I could read them any time. But after I talked to my friend Patricia Sleeman, I wasn't so sure. So I clicked over to the old archive folder and double clicked on the first document I saw. Did it open? It did not. Instead, a pop-up informed me that... You are attempting to open a file type that has been blocked by your file block settings and the trust center.

It turns out that I can't open my own Microsoft Word documents. Well, what did you expect? This file is donkey's years old, and frankly, I'd carefully copied those old Word documents over and over again, but at some stage, my system stopped being able to read the files. And I never knew when that moment was, because I didn't check. Joni Mitchell saying, don't you always seem to go that you don't know what you got till it's gone? When it comes to digital archives, you might not know what you've got till it's been gone

for years, or decades. The psychologist James Reason, an expert in human error, calls this a latent condition. When the battery is in the fire alarm of run flat, you won't discover that unless you check, or until as a fire. When your digital documents are become unreadable, you won't discover that either until you check. Or since you probably won't check, you won't discover it until you desperately need them. And by then, it may be far, far too late. In 2007, three researchers studied how often links from online legal articles no longer

existed. On the website of the US Supreme Court, half of all the links cited in legal opinions didn't work. Half of them. In top legal journals, more than two thirds of the links were broken. This is the law of the land we're talking about, simply coming apart of the seams. And it isn't just legal opinions. Political statements are often deleted by the politicians who made them. Old tweets, old speeches, they all vanish, unless someone makes it a priority to capture and preserve them. That somebody is often the internet archive, most famous for operating the wayback machine,

which allows you to travel back in time to view earlier versions of any webpage. The Internet Archive is a private initiative run on a shoestring, and people who don't like what it preserves keep trying to shut it down. We tend to take archives for granted. We assume that when there's a digital record of something, if it matters, it'll get preserved. But this preservation doesn't happen by accident. It takes money, and organisation, and determination. George Allwell's 1984 painted a picture of life under a totalitarian state which continuously rewrote the historical record. The past was erased, the erasure was forgotten, the lie became truth. But we don't need a

totalitarian dictatorship to lose our grip on the past. We just need to stop paying attention. It's hard to imagine all the problems Adrian Pearce had to solve to reincarnate the Doomsday system. They went so much further than just finding an old system in working order. He had to find a supplier of long, obsolete parts because he kept frying them. To restore the video, he had to find the master tapes, and of course it was no good his job to keep the master tapes. In the end, cliche of cliches, the tapes were found gathering dust in the attic of the original BBC producer, Peter Armstrong.

Adrian worked on the Doomsday project for 16 months, unpaid, gazing so long at hexadecimal he could read it with his naked eye. Finally, he was able to realise his wife, Sally's dream. He produced a Windows compatible version of the Doomsday project, and uploaded to the web at doomsday1986.com. It all seems like a happy ending, right? Right? No.

Have we learnt nothing about taking digital archives for granted? It was with a sinking feeling that I looked for Doomsday 1986, Adrian and Sally's Labour of Love. It's offline. I discovered that Adrian Pearce died in 2008 and the site went dark shortly afterwards. That's what happens to old websites. Adrian's obituary in the local newspaper notes that one of his lasting legacies will be the Uckfield and District Preservation Society website, UDPS.co.uk. Some legacy. That website is gone too. There have been other Doomsday Preservation projects. Most of them are also gone.

The BBC itself launched a Doomsday Reloaded project. It went online in 2011, then disappeared. The BBC even made a radio program all about the preservation efforts. The program has a webpage, it says, sorry, this episode is not currently available. The UK's National Archives do have all the original text and video from those 1986 laser disks and it is online. But in archive form, it's basically useless. There's no way to navigate around and find what you're looking for. Or is there? In 2021, a software engineer named Daniel Irwicker hacked together an interactive interface that looks like Google Maps.

It can pull out a relevant page from the National Archives, turning an unusable heap of data into an interactive searchable resource again. Earwicker called his project, Doomsday 86 reloaded, reloaded. The multi-million-pound project built around the labour of hundreds of thousands of volunteers is legible today because one volunteer, Daniel Erwicka, thought it would be really cool if that happened. He was right. It is really cool cool if that happened. He was right. It is really cool, and it works. For now.

But if the experience of Adrian Pearce taught me anything, it's that one volunteer's digital preservation effort, no matter how heroic, is fragile. It may all be readable and searchable today. Tomorrow, it could be gone again, erased and forgotten, like all those old landing cards that nobody could be bothered to keep. Michael Brathwaite never did get the knock at the door. Thanks to the reporting of journalists such as Amelia Gentlemen, Michael Braithwaite never did get the knock at the door. Thanks to the reporting of journalists such as Amelia Gentlemen, the plight of the people

being deported came to public attention. It became a national scandal, known as the Windrush scandal, after a ship, the Empire Windrush, which brought more than 1,000 people from Jamaica to the UK in 1948. An inquiry was held, a cabinet minister resigned, a apologies were published, but not before the UK government to its shame had illegally deported more than 80 elderly people who'd come legally to the UK as children, but couldn't prove it. Michael eventually got his biometric ID card, but he couldn't face going back to his old

job. To work alongside the colleagues had been too frightened to support him. Instead, found a new calling, as a campaigner for the thousands of people affected by the windrush scandal. People who lost their jobs, their self-esteem, people who like Michael, had been torn apart. After the inquiry, the government set up a compensation scheme. It's being administered by the home office, the people who caused the problem in the first place. It isn't going well. You won't be surprised to hear that there's still plenty for justice campaigners such as Michael Braithwaite to do. And I wonder if we've fully appreciated all the right lessons from the Windrush scandal.

It had its roots in anti-immigrant rhetoric and unexamined racism, but also in bad archival practices, as William the Conqueror could have told us, record-keeping matters. And what about the original Doomsday books, nearly a millennium old, great Doomsday and little Doomsday and Little Doomsday. Today, they're regarded as the most complete surviving record of pre-industrial society anywhere in the world. Surviving? Yes, surviving. For centuries, William the conquerors' original manuscripts have been preserved for posterity. In 1666, for example,

they were saved from the great fire of London. In 1869, they were rebounded with an ornate new leather cover. During the Second World War, they were moved beyond the reach of Nazi bombs. Now, they kept in dry, cold storage at the National Archives in London. They're perfectly legible, 937 years old and still going strong. If you think that archives don't matter, tell that to Michael Breithwaite. If you think that digitization solves the problem, tell that to Michael Breithwaite. If you think that digitization solves the problem, tell that to Adrian Pierce. And if you think that Doomsday books survive through chance,

think again, they exist only because every generation since 1086 cared enough to make the effort. What archives today will we care about enough to preserve for tomorrow? For a full list of our sources, please see the show notes timhalford.com Corsion retails is written by me, Tim Harford, with Andrew Wright. It's produced by Alice Fine's with support from Marilyn Rust. The sound design and original music is the work of Pascal Lys. Sarah Nix edited the scripts. It features the voice talents of Ben Crow, Mel Negatridge, Stella Hartford, Jammiss Aunders, and Rufus Wright. The show also wouldn't have been possible without the work of Jacob Weisberg, Ryan Dilly,

Greta Cohn, Lytel Moulard, John Schnarrs, Eric Sandler, Carrie Brody, and Christina Sullivan. Corsinary Tales is a production of Pushkin Industries. It's recorded at Wardaw Studios in London by Tom Berry. If you like the show, please remember to share, rate, and review. Tell your friends. And if you want to hear the show add free, sign up for Pushkin Plus on the show page in Apple podcasts or at pushkin.fm-plus.

I'm Paul Muldone, a poet who over the past several years has had the good fortune to record hours of conversations with one of the world's greatest songwriters, Sir Paul McCartney. The result is our new podcast, McCartney, A Life in lyrics. Listen to McCartney, A Life in lyrics on the iHeart radio app, Apple podcasts, Amazon music, or wherever you get your podcasts. on the iHeart Radio app, Apple Podcasts, Amazon Music, or wherever you get your podcasts.