In a preamble, I would have to agree, re: the strawmanning of academia. If we are to, as Trilling said, attempt to consider things complexly it is necessary for me to acknowledge: there are many in literary departments up and down the Archipelago doing good and interesting work, and that so with passion. Indeed, I enjoyed my degree (in the long ago days) immensely. Primarily, I think my reservations are to do with: a) how this good and interesting work is reaching the Commonweal as a whole*, b) if the modes of language used aren’t a net (an Iris Murdochy–Under-the-Net-type-net) that traps and hampers rather than frees, and c) if the strictures of academia, as currently constructed, deprive (say, in the case of philosophy) us of figures like a Kierkegaard, a Nietzsche, a Plato.
But let’s refine your point further. You mentioned a certain old-fashioned impulse to have the author be live and well on the page contra Barthes, but simultaneously what you’re asking for is something that sounds at least pretty modern, or at least postmodern. A foregrounding of the apparatus, a self-consciousness. You want – in a manner – a self-conscious text, perhaps not metafiction, but meta-nonfiction, a metaessay (though, one assumes, not simply one that only describes its own making, but is also about something else). You note an ennui, a distaste concerning “the mechanics of outright fiction.” I wondered if this had to do with an inauthenticity that you were tasting. Trilling wrote about the distinction between sincerity and authenticity. Broadly, he says sincerity is about saying out loud what is in your heart, and authenticity is to do with being oneself. Your insistence on the mess and stuff and muddle is to ask for a kind of realism or authenticity. Simon says, Thoughts don’t come from nowhere. Simon says, Thoughts emerge from the mess, the stuff and funk. Simon says, Show me this. Is this that familiar move that we have seen in our literature, the restless attempt to get at something truer or ‘real’, etc? So if modernism is (v simplistically) the literature of consciousness (Joyce, Woolf etc), and postmodernism (v simplistically) the literature of self-consciousness (Calvino etc), this is a move away from fiction as such, toward a non-fiction that has this awareness, this self-consciousness about how it is made? Is this a useful way to think about what you’re saying? Or not? What does Simon say?
So far, the prime example you’ve given is Dept. of Speculation (2014), which, broadly speaking, is a novel, and I think an example that is more squarely in what we might call the essay would be helpful.
To add a discordant chime to your literary spidey-senses: your particular thesis doesn’t hold – at least with the evidence you bring to bear. It is without question that, say, Woolf’s diaries and letters are of very great worth**. Francis Spalding has, like you, speculated that it is these that will last, and have the most value. I think this is a stretch – wonderful as they are, the diaries and letters don’t exceed the brightness cast by the luminous stream formed by Mrs Dalloway (1925), To the Lighthouse (1927), and The Waves (1932). Equal in value, perhaps, but they don’t outshine her fiction. Bennet, I think, is fair game here. (I don’t know enough about Simon Gray or Thoreau to say.) But are these representative figures? If one casts the net (not an Iris-Murdochy-Under-the-Net-type-net, but a book-and-writer-nabbing-type-net) further, it isn’t clear to me that you will dredge up enough driftwood you need to prop up your thesis. Your examples are journals, diaries. Things that are done in private, and may or may not, have an intended audience beyond the writer themselves. To run with that, James Joyce’s letters (even the dirtiest ones) don’t have the value of that lodestar Ulysses (1922); Kafka’s Blue Octavo Notebooks (1954) are very beautiful but I doubt will accumulate enough clout to overtake his other work (nor should they, I think). These are very narrow examples, of course, but I think for you to give the tendency of your thoughts (The Simon Tendency) more power there needs to be larger theory of the case, and more luggage inside that case.
* Not in a calculated impact way, but I do think that advocacy of reading and literature as such could play a larger role in what departments do. Maybe. My thoughts are hazy as a Pea Souper, or Air Gravy.
** This description of Woolf’s diary is so great that it needed to be here, and it felt relevant to what you’re thinking about. From A Writer’s Diary (1954): 'What sort of diary should I like mine to be? Something loose knit and yet not slovenly, so elastic that it will embrace anything, solemn, slight or beautiful that comes into my mind. I should like it to resemble a deep old desk, or capacious hold-all, in which one flings a mass of odds and ends without looking them through. I should like to come back, after a year or two, and find that the collection had sorted itself and refined itself and collapsed, as such deposits so mysteriously do, into a mould, transparent enough to reflect the light of our life, and yet steady, tranquil compounds with the aloofness of a work of art.'