The Defence of the Realm: The Authorised History of MI5
By Christopher Andrew
Allen Lane £30, 1,056 pages
FT Bookshop price: £24

How should a national security service, facing the worst-ever terrorist threat to its country, mark its centenary? Not by the light-hearted celebrations with which it marked the end of the first world war. In March 1919, MI5’s own concert party, the “Barmy Breezies”, put on the Hush Hush Review, a mildly risqué production gently mocking the director-general, Vernon Kell, who went on to become the longest-serving head of any 20th-century UK government agency or department.

At the beginning of this century, the then director-general Stephen Lander chose a much more daring project to celebrate the service’s anniversary – an authorised history of the first 100 years, to be written by an independent historian and made publicly available. By then the service was a much more open and accountable organisation than in 1909, or even in 1969, when I joined as a junior assistant officer. Like most other people, I knew very little about what the service actually did – though I had gleaned something by working in New Delhi as a part-time clerk typist for the MI5 liaison officer to the Indian Intelligence Bureau. At the height of the cold war, most recruitment to the service was, like mine, by the tap on the shoulder and the suggestion of a job in a mysterious “government department”.

Now that MI5 is primarily a counter-terrorist service, involved in issues that affect everyone’s daily life and depending on public support for information and operational assistance, much greater openness is appropriate. But even now, with open recruitment, a website, archives regularly released into the public records, a named director-general and various forms of oversight, some of the detail of its work must remain a mystery.

MI5 is the first major security or intelligence service in the world to give a historian free range of its records – nearly 400,000 paper files, some with many volumes, says Christopher Andrew with a touch of exhaustion. I remember them – some on flimsy, crumbling wartime paper, some with scorched edges after a bomb hit the service’s temporary wartime home in Wormwood Scrubs prison. Andrew, professor of modern and contemporary history at Cambridge University, is the author of a number of authoritative books on the intelligence world. He was allowed to see everything, including material on sources and methods that he accepts could never be published, but which he thought he must see to ensure there was nothing there that invalidated his conclusions. The clearance process for his final text was fraught, but significantly it seems to have been the caution of other government departments that caused most problems.

It has been well worth the effort. The Defence of the Realm throws new light on an important area of the running of the country, analysing the changing threats to national security over the 100 years and discussing the appropriateness or otherwise of the service’s response. But just as interestingly, the book gets inside the culture of this secret service, showing how attitudes have changed with those changing threats; how women have worked their way from the fringes to the heart of the organisation and how a sense of humour has always been important. It will be enthusiastically scrutinised by historians, intelligence buffs and conspiracy theorists, all of whom will find material of interest, though those who believe in the Wilson Plot, in the extensive monitoring of the “left wing”, or in a service-sponsored policy of shoot-to-kill in Northern Ireland, will find their theories convincingly debunked.

Will it also fulfil one of its main objectives – a greater understanding by the public of the service and its work? Few of the general public will read it from cover to cover, but those who are not put off by its 1,000-plus pages and £30 price tag will find anecdotes and operational details as gripping as the plot of any thriller.

The story of the “Double Cross System”, by which the service ran and controlled German espionage in this country during the second world war, enabling the deception on which the D-Day landings depended, is the stuff of Boys’ Own fiction.

The long, tortuous investigation of the five Cambridge spies, described here in detail, leads Andrew to conclude that though the Soviet intelligence handling of its well-placed agents was appalling, the British intelligence investigation was confused and slow. Ironically, this convinced the Spycatcher author Peter Wright and others that the investigation must have been deliberately scuppered by Soviet penetration of western intelligence, and led to the baseless investigation of the then director-general Roger Hollis as a spy.

In the later cold war comes the unmasking of Michael Bettaney, who in 1983 pushed classified information through the London letterbox of the KGB resident, with an offer of further secrets to come. What Bettaney did not know, because it was a closely held secret, was that in the KGB residency was Oleg Gordievsky, a recruited agent of MI6, who was able to put the MI5 investigators on to the track that led to Bettaney. A small team known to each other as “the Nadgers” – including one who was pregnant with “little Nadger” – conducted a classic “mole” investigation in the greatest secrecy. When Bettaney finally confessed, and so could be charged, the story was greeted by the Daily Mail with a cartoon lambasting MI5 for incompetence.

The final, and not surprisingly less detailed section, brings the book up to date, with the transformation of MI5 into the largely counter-terrorist service of today. Anyone who was involved in operations against the Provisional IRA in the 1970s and 1980s will agree with Andrew’s conclusion that the confused responsibilities of the various agencies and services slowed down the response to Irish Republican terrorism, but when that was resolved, MI5 and the police working effectively together helped to bring the IRA to negotiations. I suspect there will be less agreement with Professor Andrew’s final comment that the service was slow to tackle the menace of Islamist terrorism. Historians have the luxury of looking back from known facts; intelligence officers, however, have to work forward from what is known at the time, which is often very little.

Dame Stella Rimington was director-general of MI5 from 1992 to 1996. Her latest novel is ‘Present Danger’ (Quercus)