Catholic tradition holds that the Holy Spirit guides the cardinals who elect the pope, but there have been some dreadful men (and, according to legend, one woman) over the centuries: venal, corrupt, depraved. The seven men under scrutiny in Jesus Wept were none of these things, but they had many failings. Philip Shenon, once a journalist with The New York Times, employs all his reportorial skills, so that the book is highly detailed, engaging, often amusing, and always accessible.

Shenon describes his project as a ‘without fear-or-favour account of the battle for the soul of the church that began in earnest with the election in 1958 of Pope John XXIII’ and the Vatican II council (1962-65). That battle has raged ever since, right up to today. He categorises it as the attempt to dispense ‘the medicine of mercy instead of severity’, and the countervailing attempt to combat that very approach as heresy. This turmoil is widely interesting because the Roman Church has been, as Shenon says, easily the most important institution in the history of Western civilisation, controlling an outsized share of the world’s wealth and political and cultural influence.

The book covers Pius XII (1939-58), John XXIII (1958-63), Paul VI (1963-78), John Paul I (1978), John Paul II (1978-2005), Benedict XVI (2005-13), and Francis I (since 2013). Interwoven into the narrative are influential actors such as Hans Küng and Cardinal Leo Suenens, two of the most influential reform advocates at Vatican II, along with Joseph Ratzinger (later Benedict), plus Karol Wojtyla (later John Paul II), and Jorge Mario Bergoglio (later Francis), before their elevation.

Pius, a long-term Vatican diplomat to Germany, was tainted by his failure to denounce the Holocaust, which might have saved many lives, but he did shelter Jews in the Vatican late in World War II. Timid, apart from his fanatical hatred of communists, he wrote forty-one encyclicals and was the first pope to enjoy global celebrity.

The aristocratic Pius was succeeded by the roly-poly, warm-hearted peasant John XXIII, who loved a cigarette and a glass of champagne. On his accession, asked how many people worked at the Vatican, he replied ‘About half’. Another time he said, ‘I still don’t understand why I have become pope. I believe God is having some trouble explaining it to himself.’ But his goal was to call an international council, Vatican II. He wanted to reverse ‘the church of no’, to ‘dispense the medicine of mercy’, modernise and refresh the Church, and emphasise that it would become the loving mother of all. He met bitter opposition from the highly organised and dominant Curia (Vatican bureaucracy), and other conservatives, though Shenon notes that those most concerned to defend the use of Latin were those least able to speak it, employing ‘a laboured pidgin’. John set up two coffee bars in side chapels at St Peter’s, where Vatican II was held, so that bishops would not have to ‘hide their cigarettes under their mitres’.

Küng said that John changed the church more than any pope in five hundred years. John died in 1963 and was replaced by Paul VI, who spent much of his papacy vacillating – simultaneously indecisive and authoritarian – and who was dominated by the arch-conservative Cardinal Ottaviani, head of the Congregation for the Doctrine of the Faith (formerly the Holy Inquisition), who dedicated himself to blocking any reform.

Paul was succeeded in 1978 by ‘the smiling pope’, John Paul I. He survived only thirty-three days, dying of a heart attack, which led to apparently unfounded conspiracy theories that he had been murdered. Along came John Paul II (Poland’s Karol Wojtyla), who became the second-longest serving pope (twenty-seven years) and who, like Benedict, spent much of his energy trying to reverse the reforms of Vatican II. He had strong links to the ultra-reactionary Opus Dei movement, with its ancient disciplines, including self-mortification.

John Paul, who saw himself as the world’s bishop, centralising power more than ever in Rome, was a tireless global traveller, visiting 129 countries. He became the first superstar pope. His indefatigable courage and determination played a massive role in the collapse of communism in Eastern Europe, especially Poland. This stood in shocking contrast to his callous indifference to, and cover-ups of, sexual abuse of children by the clergy. First he denied, against mounting evidence, that it was a problem at all; then he claimed it was only an American issue; then an Anglophone one – all the while protecting notorious paedophiles such as Legion of Christ founder Marcial Maciel and Austria’s Cardinal Groër, (the latter abused at least 2,000 boys).

Celibacy has always been honoured as much in the breach as the observance among Catholic clergy. Research suggests a considerable majority fail their vows at some point, mostly with consenting adults, of whom about half are men. John Paul’s deputy was Joseph Ratzinger, who, as head of the Congregation for the Doctrine of the Faith, directed that all allegations of clergy abuse worldwide be sent to him (not local police), where they generally disappeared without action. (Token action against paedophiles is also the biggest stain on Francis’s legacy.)

Under John Paul, Ratzinger also returned his office to the pre-Vatican II role of McCarthy-style heresy hunting, whose victims usually didn’t even know they were under investigation.  Particularly incendiary was South American liberation theology, which both men saw as Marxism masquerading as social justice.

Benedict, a shy Mozart-lover with a deep love of luxury, followed John Paul II, defeating his eventual successor in Francis at the 2005 conclave. He had begun as an influential reformer at Vatican II, but, horrified by the riots in Europe in 1968, became a leading reactionary. As pope, he followed the policies of John Paul, with Marxism and relativism his main foes. His papacy began to unravel with a controversial speech at Regensburg in 2006, where he quoted a fourteenth-century Byzantine emperor’s insults about Islam spreading its faith by the sword. Bad went to worse when sex abuse reports became increasingly prominent. The irony about Benedict’s reluctance to tackle sexual abuse was that it dominated the last years of his papacy, and may, along with the Vatileaks scandal, have been the key to his shocking decision to resign, the first pope to do so in eight hundred years.

The accession of Francis meant a swing of the pendulum towards reformers within the Church. Bergoglio, as head of the Jesuits in Argentina, had been seen as an ambitious and divisive figure, but also deeply ascetic and humble. He refused the luxuries of office, in Argentina and as pope, and when he was dressed in papal clothes after his election he refused luxury garments, saying ‘The carnival is over’. His lifestyle in Rome has been remarkably modest.

His long-time foe, Argentine President Cristina Kirchner, was the first foreign leader he received as pope, yet at his installation he arranged a second Argentine guest, a forty-nine-year-old who survived by scavenging rubbish dumps and whom Francis befriended in the Buenos Aires slums.

Francis, himself theologically conservative, has changed little doctrinally in the church, but he has changed the tone massively, emphasising mercy and gentleness. His most famous remark is that ‘If someone is gay and is searching for the Lord with goodwill, who am I to judge?’ No mention of intrinsic moral disorder – quite revolutionary, and deeply discombobulating to conservatives.

After twelve years and in failing health, Francis is surely near the end. What comes next no one knows, but, given that Francis has appointed a large majority of the cardinals who will elect the next pope, from all around the world, and has reduced the number from the Curia, the prospects for continuity seem good.

The book has one strange defect: at the end, there are 509 footnotes, but no matching numbers on the relevant pages to identify their relation.