‘Oh, it’s going to be socially annihilating,” the philosopher John Gray says mildly. It has not taken us very long to get onto the subject of artificial intelligence. “Its effects on employment for young people will be huge.” Universities have already “overproduced a certain kind of graduate”. But faced with competition from machines that can write and code to a high level instantly and for free, their “economic prospects are going to get worse and worse”. A resentful, underemployed graduate class, Gray contends, is a historically reliable indicator of disaster. He cites the chaos of the Russian Empire in the decades before the revolution.

In Browns restaurant in Bath — prosperous, tall-ceilinged and filled with pale light — the possibility of social annihilation seems distant. But in Gray’s company one’s convictions of permanence begin to flicker and dim. His career — in universities, think tanks and journalism — has been founded on pointing out that the things that seem solid are transitory. Western analysts once believed in the permanence of the Soviet Union. Gray said it wouldn’t last. It didn’t. When the Soviet Union collapsed, many proclaimed the end of history and the birth of a liberal world order. Gray said that wouldn’t last either. It hasn’t. He has attained that rarest of distinctions among public intellectuals: that of being right.

And so it is alarming to read in Gray’s latest book, The New Leviathans, that he believes we are entering an age of catastrophe comparable to the “general crisis” of the 17th century. Gray quotes the historian Geoffrey Parker, a chronicler of that unhappy time. There was “a proliferation of wars, civil wars and rebellions and more cases of state breakdown around the globe than any previous or subsequent age”.

Our own general crisis will include not only the economic devastation and worsening inequality wrought by AI but climate change, the retreat of democracy (especially in America), the dangerous foreign policy ambitions of China and state breakdown in Russia. By now you may be picturing Gray as a man of Old Testament aspect: raddled, prophetic, prodigiously bearded. In fact, he is soberly turned out with the quiet good manners of a provincial lawyer, retired after a good career. Unlike many prophets of doom there is no part of him that lusts after chaos. Rather he is conscientiously illusionless, scrupulously refusing to believe in any of the ideals and comforting dreams that humans use to protect themselves against reality.

This, perhaps, explains his popularity with my own much-disillusioned generation. Gray’s philosophy is the thread that joins my friends of disparate political inclinations. My Twitter timeline shows that his new book is being read by centrists, Marxists, reactionaries and others. New readers should start either with Straw Dogs, which is perhaps the most comprehensive statement of his thought. Or else with Feline Philosophy, a charming guide to how to live, inspired by Gray’s respect for cats: “Cats show us that seeking after meaning is like the quest for happiness, a distraction.”

Gray says he owes his guiding intuition about the fundamental impermanence of human institutions to growing up “in an industrial street community” in South Shields in the 1950s. In one of the idealistic social projects characteristic of that age, the streets of his childhood were cleared and the residents “moved out to the edge”. “I experienced a very benign form of the 20th-century experience, which was to see a way of life disappear.” Privileged politicians and philosophers who have passed easily through schools and careers hundreds of years old lack that understanding of the fragility of ways of life.

Perhaps the greatest enemy of Gray’s career has been the liberal faith in human progress, which prevailed in the 1990s and 2000s. Men and women who imagined themselves impeccably rational believed that science and social change would lead societies to inevitably become richer, freer and happier. Gray always saw that idea as one based in faith, not reason. Has the advancing technology of our century made us happier? There is little evidence to suggest so — in the case of smartphones, quite the contrary. Today Gray’s scepticism is trained on the “woke” movement, ascendant in universities and many businesses. “Woke” beliefs in Gray’s view are not what some of their right-wing opponents believe them to be, a new variant of campus Marxism. Rather, Gray argues that it is a pathology of liberalism (he sometimes uses the term “hyper-liberalism”) that has exaggerated liberalism’s respect for the rights of the individual into an almost religious reverence. Each individual has a quasi-divine right to the “self-creation” of an identity which must not be contradicted.

But, “by droll necessity”, Gray notes in The New Leviathans, to prevent anybody offending against the sacred principle of individual self-expression, “every aspect of life” must be “monitored and controlled”. “Language must be purified of traces of thought crime. The mind must cease to be a private realm and come under scrutiny for its hidden biases and errors.” “The logic of limitless freedom,” he writes, “is unlimited despotism.”

Gray views this new philosophy as absurd and intellectually tawdry but also hopelessly inadequate as a consolation or guide to the dangerous times to come. For progressives, “their lives are going to be so hard, and so hard for them to understand: Europe turning into a far-right bloc, America descending into a failed state . . . elites who promoted progressive ideas will retreat into gated communities in Arizona or New Zealand”.

For the UK he predicts “drift will continue . . . It won’t be managed decline. It will just be decline.” An important part of our problem, he thinks, is the fashionable trend towards devolution, which means it is hard for any politician to re-engineer the British state. For good or ill, Margaret Thatcher was able to transform society because the levers of power available to politicians were still effective — “no devolution, no lawfare, no Supreme Court, NGOs weren’t as powerful as they were”. Today, “the power of the state has been dispersed through society”, he says, which “stops getting big things done”. “When centres of power are distributed through multiple different institutions, the better-off parts will fight their corner. Nimbyism is just one example.”

He believes the best outcome at the next election would be a hung parliament, which could induce a crisis capable of forcing our politicians to replace our inadequate first-past-the-post voting system with proportional representation. “Something we learnt from the Johnson era was that a huge majority doesn’t produce strong government, it produces weak government. The only solution to shake up our torpid political class is to change all the incentives.” He hopes for the emergence of “a centre coalition which really reflects the values and beliefs of most British people”.

How should I find comfort, I ask, in this bleak world of the future, when my job has been stolen by a chatbot and Britain is drifting into poverty and worse. His answer is characteristically sceptical and modest. “What is valuable in human beings is not intelligence,” Gray tells me. Neither are ideologies or “world-transforming projects”. What matters “is passing sensation: human love, human courage, human defiance. It is passing sensations that make it worth having lived.”