There's an interview with David Foster Wallace, in which Michael Silverblatt miraculously guesses that Infinite Jest is structured like a fractal. In fact, Wallace says, it's built like a Sierpinski gasket (more commonly called a Sierpinski triangle). No one has followed up on that; there's even a new book out from Bloomsbury on form in Infinite Jest (2017) that devotes only half of one paragraph to Wallace's claim. I assume the structure is not legible in the published version because of the "mercy cuts," as Wallace put it elsewhere, but the fundamental idea is very interesting: small things are blown up to large ones, and vice versa, and the structure is recursive, enclosing near-copies of structures within each other, potentially without end.
That is one model for the structure of a long novel. I've looked, but I haven't found any studies of structures of the maximalist postwar novel, at least after Perec; it's a study that's needed for anyone interested in understanding what counts as whole, coherent, or complete in works by Schmidt, Vollmann, Barth, Gaddis, and other long-form writers. (Barth is perhaps especially pertinent to Dixon, because the two of them ran the writing program at Johns Hopkins.)
Stephen Dixon's Frog (1991) is interesting in this context, because it is fractal in a different sense than Infinite Jest. Dixon is a compulsive writer and publisher, and his vita looks like a tabulation of all the literary journals publishing in English in the last forty years. He's said somewhere that he had about 46 publications and half that many publishers, because he'd be dropped as soon as the second book failed to sell.
Most of his output is short fiction, and a fair percentage of that is very short, nearly flash fiction. His novels, with Frog perhaps the most interesting, are studies in the aggregation of short forms into long ones. Frog is 769 pages long, and divided into 21 chapters. Most of those are short, on the order of 15 pages. "Frog's Mom" is a novella at 110 pages, and "Frog Fragments" a full-size novel at 220 pages. As William Ferguson says,

"It is as if the central character, Howard Tetch, represented several versions of what one man might be – a portrait that includes not only his physical attributes but the host of possibilities that swarm around his life like bees around a flower... The most startling of these stories is "Frog Made Free," in which the four members of the Tetch family mysteriously find themselves in a cattle car on their way to a Nazi death camp. (Auschwitz's infamous motto, "Arbeit Macht Frei," is ironically echoed in the title.) We know from other stories – or we think we know – that the Tetch family belongs not to the Holocaust years but to later decades, yet nothing in the text indicates that this episode is a nightmare from which Howard might conceivably awaken. Such a daring imposition of characters on the past recalls one of the fundamental aims of fiction: in the midst of particularities, to be in some way suprapersonal, historic, truer than any individual truth could be." [William Ferguson, "Which Version Do You Prefer?," New York Times, September 4, 1994.]

Dixon's style is telegraphic, abbreviated, compulsive, informal, breathless, concise in grammar and excessive in the permission he gives himself to run on. The basic strategy of Frog is the entertainment of possible alternate stories and futures. In one chapter, the main character imagines what would happen if he went downstairs to investigate a noise; the chapter explores dozens of alternatives, one after another with no segues. In other chapters he thinks about his sister, his wife, and his brother, and their fates and paths through life. Even brief chapters ramify into dozens of stories, all plausible the moment they're told. Sometimes the truth of a death or an illness emerges as the chapter progresses, and in that case the multiple stories carry a heavy burden of pathos, as we're invited to think of the narrator's sad helpless rehearsal of alternate pasts. Other times the multiple possibilities aren't resolved, and readers get a less focused sense of the narrator's frantic mental state.
Frog is nineteen entirely separable short stories, a novel, and a novella, under one cover. The nineteen short stories are mostly entirely self-contained. Often they are as tightly composed as his free-standing short stories. I think Dixon wrote his way toward "Frog Fragments," because it contains echoes and repetitions of some earlier chapters; but most chapters are potentially independent. On the face of it, then, Frog isn't coherent, and in fact it is ostentatiously disunified. But the endlessly multiplied branching narratives in each chapter produce a fractal effect: the chapters divide into dozens or hundreds of parts, and they are in turn aggregated into the whole of Frog. Because the stories endlessly ramify, the ramified chapters are less incoherent. It's an interesting model for a large novel.
Dixon himself had talked about the structure of his novels, but he has a perhaps unhelpfully laissez-faire way of thinking about form. About Frog, he said: "I wrote the first draft of the first story (chapter?) in it in Prague – it's called "Frog in Prague" – and I finished it in Maine, summer, '85, and continued to write stories with Frog in it, and then the stories got longer and I had novella-length and novel-length stories, and that's how it was written. I never know how long a work is going to be when I start it, and I rarely know where it's going to go and what the structure of the work will be." (Sean Carroll, interview in Bookslut, December 2010.)

*
It's a separate question how well this works. Dixon has had an unusual number of negative reviews. According to Vince Pissarro,

"The run-on sentences, the rapid-fire but mundane stream of consciousness, the apparently frank but merely amphetamined dialogue that goes back and forth and back and forth within page after page of unbroken paragraphs that stretch as far as the eye can see: these devices are no longer energized by an author who has anything fresh to say... Exchanges like [that] are too easily achieved – too easily typed, even – and they're not challenging to the reader or to Dixon himself." [Vince Passaro, "S.A.S.E," New York Times, May 16, 1999.]

But this kind of criticism is too simple, and so is the complaint – common on Amazon and Goodreads – about Dixon's endlessness. His narrators are compulsive, and so is the implied author, who writes at speed. His signature style, which omits particles, verbs, punctuation, and prepositions, is a direct effect of his frantic frame of mind: Write! Think! Publish! Don't stop! And when that frame of mind is applied to the dissolution of memories, as it is in Frog, the result is intensely expressive. It has the irritability of some Alzheimer's patients, and the everyday anxiety of any middle-aged person trying to make sense of her unraveling life.

From the blurb....
"Frog is a complex and paradoxical character: petulant, compulsive, overbearing, hostile, and self-righteous, but also imaginative, loving, kind, and strong. No matter how exasperating he gets, it's still hard not to care about him. His story is a nonsequential patchwork of flashbacks, real and imagined stories, and retellings from various perspectives.
Dixon, best known as a gifted short story writer, demonstrates his mastery of the novel here, imaginatively expanding on a variety of themes like a great jazz soloist. Although some will be totally confused after the first 200 pages, the patient reader will be held spellbound as Dixon weaves his web of many versions. While this is not quite the " Finnegan's Wake of our generation" as the publishers claim, Frog 's labyrinthine structure, rapid-fire wordplay, vivid descriptions, and raw emotional power are all reminiscent of Joyce."

Yeah, right.

'No matter how exasperating he gets, it's still hard not to care about him.' ... No, it is not. Trust me on this one.
'Although some will be totally confused after the first 200 pages' ... I was confused after the first 200 words and stayed confused right through to page 769.

I read this over several months. It was like watching cat play with a mouse. I kept coming back to see if it really was as weird as I remembered last time, and hoping it would get better. After all, it was a finalist in the US National Book Awards. All I can say is it must have been a slow year and the author must have been on drugs. Joyce it certaintly ain't. ( )