Is failure the new literary success?

From memoir and self-care books to comic novels, writing about our flaws and imperfections has never been so popular. But can failing ever be a success, asks Lindsay Baker?

“Fail again. Fail better,” wrote Samuel Beckett in what has become a familiar mantra in the world of business and tech start-ups – along with ‘Fail fast, fail better’ – where the notion of failure as a route to success has taken a firm hold. Recent years have seen a similar preoccupation seeping into literature, particularly in the memoir sector. Karl Ove Knausgård devoted several autobiographical volumes to everyday failures in My Struggle, and since then there has been a deluge of ‘fail-lit’, both in fiction and non-fiction. Could failure be the new literary success? And if so, doesn’t that mean it’s not really failure at all?

More like this:

-        Can fiction improve your mental health?

-        The 1968 sci-fi that spookily predicted today

-        Is it time to rewrite fairy tales?

In her memoir How to Fail: Everything I’ve Ever Learned from Things Going Wrong, Elizabeth Day explores with painful and often funny honesty what happens when things go wrong, and how failure can work in our favour. Paradoxically, the book and the podcast that inspired it – How to Fail with Elizabeth Day – have been hugely successful. So it’s perhaps no surprise that Day has been challenged about her failure credentials – particularly since, in addition, she is a successful novelist and newspaper columnist, has a double-first from Cambridge and willowy, Keira Knightly-ish looks.

“I have a wonderful life, and I’m incredibly privileged,” she counters. “But that doesn’t mean there weren’t times in my life that weren’t tough to live through. In my case it was personal failures – divorce, miscarriage, and I failed to have children,” says Day. The book is part memoir and part manifesto, and includes themes such as dating, friendship, work, sport, families and anger. “The premise of the book is that we can learn from our failures.” So is failure always a good thing in her view? “I don’t advocate being Pollyanna-ish about it, and my failure to have children is a sad one. But I’m at peace with it – you can choose to accept failure.”

“There’s something very levelling about ‘successful’ people itting that their career path has not been smooth,” says the author, who decided to start the podcast after various honest conversations with girlfriends. “I thought ‘wouldn’t it be great to open up the conversation about vulnerability’.”

The stories told in her book and podcast are compelling, hopeful, often empowering, and vary wildly. “Men and women do failure differently,” says Day. “There’s a kind of macho idea of failure in the bro culture of Silicone Valley and start-ups. But failure depends on how you react to it. The paradigm is shifting, and women are speaking their truth.”

Day says that “dark experiences” shouldn’t be sidelined. “Life is texture, sadness can be a beautiful thing, and you can’t appreciate one without the other. Life is a combination of things. I have evolved more as a result of things going wrong than when everything seemed to be going right. Out of crisis has come clarity, and sometimes even catharsis.”

Author and academic Paul Dolan views the failure “fad”, as he calls it, differently. “It’s successful failure, it’s failure as a key to success. I’m a bit sceptical about it,” Dolan tells BBC Culture. “We’ve seen an increasing trend for perfectionism, and the current interest in failure is a backlash to that. But actually it feels like it’s a reinforcement of perfectionism – failure is seen as a challenge to overcome while pursuing something. But sometimes we should just stop striving for the narratives that are expected of us. People want a lifetime narrative, but if you’ve been in a relationship with someone, for instance, and had 15 good years together, it doesn’t mean it’s a failure if it ends.”

In his latest book Happily Ever After: Escaping the Myth of the Perfect Life, Dolan explores some of the myths about our sources of happiness, and argues that there can be other, unexpected paths to lasting happiness. Marriage, university and having children are among the signposts to success that Dolan challenges. Statistically, unmarried and childless women are the happiest subgroup in the population, he points out. They also live longer, he says – traditional markers used to measure success no longer correlate to happiness: “By freeing ourselves from the idea of the ‘perfect life’, we might each find a life worth living.”

“The idea is, if you push through failure it becomes another hurdle to success. We love stories about overcoming adversity. And then we celebrate success even more. It’s about how hard-won the success was. So it’s not celebrating failure.”

The writers of the Midult and the book I’m Absolutely Fine: A Manual for Imperfect Women, Emilie McMeekan and Annabel Rivkin, not only accept failure but also celebrate it, and aim “to pop the perfection bubble”. Fed up with images of female perfection in advertising and on social media, they wanted to focus on the real lives of grown-up women without any sense of shame, women in all their flawed glory who, for instance, “ the lyrics to every 1990s boy-band song, but struggle to recall why you walked into the kitchen”. Writer and mental-health champion Bryony Gordon, meanwhile, writes about how she learnt to embrace her flaws and imperfections in her books. In You Got This, she encourages young people to accept themselves as they are, and not to endlessly compare themselves to others. “Comparison is the thief of joy,” she says.

Beauty in imperfection

Overcoming adversity – or not – has always fascinated us, as far back as the myths of Icarus and Sisyphus. Storytellers and writers have long found drama and tragedy in failure, from Greek tragedy to Shakespeare’s flawed, tragic protagonists to Tennessee Williams’ memorably fading, washed-up roles. There is something beautiful to be found in failure. Man Booker Prize winner László Krasznahorkai’s novel Seiobo There Below is based on a premise of Wabi-sabi– a phrase that stems from an ancient Japanese concept, meaning beauty in imperfection. In his novel, Krasznahorkai constructs failure as an aesthetically pleasing subject.

And failure can also be fertile ground for comedy. Tragicomic writing is a well-established genre, comedies of errors are universally popular, and bathos a familiar literary technique. Satire in general, it could be argued, is essentially about failure. US novelist Sam Lipsyte is known for his darkly funny depictions of human imperfection. His latest novel Hark tells the story of an unwitting mindfulness guru and his inner circle, chief among them the lamentably flawed Fraz Penzig.

So are abject failure and imperfection inherently funny? “Inherently funny, yes, but also inherently poignant,” Lipsyte tells BBC Culture. “We like to laugh and wince at reminders of the enormous gap between what we want to be and what we are. It's recognition of our humanity. Although maybe we are not alone in this. Maybe there are dolphins swimming around thinking ‘I'm not really the dolphin my mother hoped I'd be, am I?’”

Lipsyte points to numerous writers who have influenced him, including Grace Paley, Robert Musil, Geoff Dyer and Martin Amis. The character of Fraz is, he says, “an amalgam of several people, including me, my brother-in-law and the apostle Thomas”. Fraz and other characters in the book struggle with many different situations and circumstances. “Failure is part of life.”

Throughout the novel, there’s a sense of compromise, muddling through and expediency. “I’m fairly certain compromise has always been part of the human condition,” says the novelist. “But as far as modern life goes, I think there are categories of compromise. Some compromises… perhaps the kinds we make with our spouses and partners in the domestic sphere, help keep love alive.”

So can failure be helpful, in Lipsyte’s view? “I do think we learn more from mistakes than from triumphs. Maybe because we them more. But I think the key is to maybe stop lumping everything into one bin or the other. Most experiences contain both.”

Lipsyte’s favourite Greek myth about failure is Sisyphus, whose story is full of futility, failure and absurdity: “Sisyphus might be the Muhammed Ali of failure myths. The greatest of all time. Of course, it’s not his fault the rock keeps rolling down the hill. Except that it is, insofar as he brought the whole situation on himself with his deceitfulness.”

And he shares Paul Dolan’s scepticism around accepted notions of success. Is the idea of ‘making it’ a myth? “Yes, very much so,” says Lipsyte. “I do think working hard at something often pays off, sometimes in ways we might not even predict, but there are no guarantees. In America, the flipside of the American Dream has always been the notion that if you don't achieve this ideal you have somehow failed, probably due to a defect in your character. There is a good deal of shame and anxiety involved. We’re not enough, haven’t done enough. The idea of ‘making it’ is just a story. As we know, it’s largely nonsense. So much is determined by factors outside of one’s control.

“Also, there are many different measures of success. They all can’t be monetised. This is obvious stuff, but we often forget it and make ourselves miserable comparing ourselves to others and dwelling on what we haven’t achieved or acquired. I’m all for ambition when it’s related to the deeper satisfactions of using our bodies and minds to create art or make things better for people or engage in some meaningful way with the world, but I think one thing my books have always been about is how people deal with the bill of goods sold to them about what constitutes success and happiness.”

Why does he think we so much enjoy reading about miserable failure and imperfect characters? “Because it’s a twofer: a mixture of schadenfreude and self-reflection. Misery and imperfection belong to all of us.”

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us on Twitter.

And if you liked this story, sign up for the weekly bbc.com features newsletter, called The Essential List. A handpicked selection of stories from BBC Future, Culture, Capital and Travel, delivered to your inbox every Friday.