Do you know what you want to read? And is that the same as what you need to read?

According to those who market books, even if you don’t think you can answer either of those questions, they can. And they – the marketing strategists who are able to glean “customer insights” from the kind of penetrating data collection which now makes it possible to influence people to buy everything from shoes to votes – are likely to be increasingly influential in the decisions made by big publishers.

Stephanie Withers, Audience Insight Director at Penguin: “Every big decision should be driven by a consumer insight right from the start.”

Withers’ explanation of the importance of data-driven “insight” is included in Justin Ractliffe’s “Instinct, Input and Insight: Reader-centricity in publishing”, which he researched with the assistance of a professional development grant from The Copyright Agency.

Ractliffe, who is Publishing Director of the Australian arm of Penguin Random House, interviewed a range of high-powered marketing experts in the United States of America, to interrogate the way publishing can adapt to the pressures from a market not only crowded within (by the explosion in titles now published) but also from without, as forms of “creative content” proliferate and become less differentiated.

According to Withers, publishers need to think less about the person who might “want a book” and more about those who “consume entertainment”. So the trick is, apparently, to make the book-form of entertainment irresistible by matching it perfectly to what the consumer wants – even if they don’t exactly know they want it.

According to Withers, publishers need to think less about the person who might “want a book” and more about those who “consume entertainment”.

Ractliffe backgrounds his report by describing the way publishing has, traditionally, relied on “instinct and experience”, not just to make decisions about what books to publish but also about how then to market them to potential readers. “Gut instinct” is the way it’s also described, conjuring the image of the canny publisher (person, rather than company, in this instance) who can intuit the bestselling winner from among the also-rans.

Most of the marketing specialists cited by Ractliffe seem to suggest data-created consumer insights can be – indeed should be – used to assist rather than replace the gut-instinct publisher (both person and company). Impressed by the ethos and work-practice of Netflix, he follows the reasoning that it is better for a publisher to be better informed about what will work rather than guessing based on unexamined assumptions, and this will also mean better serving the interests of readers.

He follows the reasoning that it is better for a publisher to be better informed about what will work rather than guessing based on unexamined assumptions, and this will also mean better serving the interests of readers.

It’s implied that the self-reliance of old-style publishing is almost a form of arrogance; we know what’s good, so you should let us be the judge. In his conclusion, Ractliffe calls it a “publishing bubble”, which suggests it’s closed off from the very audiences it intends to reach.

While it sounds sort of democratic to bestow the power to choose on those who are on the receiving end of what is chosen, I think that’s putting a spin on this predicted publishing trajectory that glosses over problematic outcomes. It’s not Ractliffe’s intention to discuss the wider implications in this well-prepared and interesting document, but even so, there are moments in his report that are disconcerting.

Ractliffe is positive about what co-founder of Candela Digital Publishing Michael Bhaskar says about reading being unique among media because it is “where your own imagination and brain does most of the work”. And he’s enthusiastic about the way trade publishers are expanding their information about reader preferences to try to understand “motivations, needs and behaviours … around reading itself” or, as another of his interviewees puts it, understanding “the role of books in their life”.

Ok. Sounds like a plan. But then things start to go a bit odd, a bit bent. One publisher, Ractliffe reports, can distinguish between what you tell them you like (when you tick a box for instance), and what you really like, as evidenced in the way you search online and what you click on.

He quotes Alyson Forbes, VP Executive Director Marketing Strategy at Hachette, who says, “Our new CMS [content management system] has been a game changer. We can see how people are behaving on our site. It allows our marketers to customise content on the fly. All actions on site are tracked and reported on and we then create content based on this.”

That sounds sensible, a cost-effective use of the technologies available to anyone willing to spend money on Google and Facebook – to very powerful effect, as Ractliffe points out. He provides a link within his report to a Wired article that suggests the Trump campaign’s use of “customised audiences” through social media sites such as Facebook was a key factor in the USA election result of 2016. What we now know about the lack of controls or accountability in paid Facebook advertising puts this kind of information into a context that undermines confidence in its use for ethical businesses.

Content, in this context, still refers to the marketing message, I think; but then Ractliffe gives an example of the uses of listening in, “scraping” social media for insights, from a different business: “McDonald’s noticed lots of posts from customers about how they loved dipping their apple pie into McFlurry sundaes, so it combined the two and the Apple Pie McFlurry was born.”

Maybe what I think of as my curiosity as a reader … is really just my self-regard, a kind of intellectual box-ticking which is not really what I want, but what I think I should want.

This McDonald’s example shows how marketing is then used to inform and even modify the product, and that is where some of Ractliffe’s more optimistic interviewees sound like they believe and hope book publishing is heading.

He writes that “as well as relying on their own cultural awareness, publishers are engaging in trendwatching to help generate ideas for books, identify ‘mass niche’ audiences and catch and ride cultural waves before they crash.”

And he quotes CEO of The Idea Logical Company Mike Shatzkin, who says “It’s a logical step to learn from conversations and discussions taking place to publish a book if there is no book on those conversations. Publishers need to work out which genres and how fast. All of that is evolving.’”

“Logical”? Depends what system you’re working in, I’d guess. If it’s a system structured for short-term profit, that does make sense, but if it’s a system created for long-term cultural, ethical and intellectual gains, maybe not.

Towards the end of his interrogation of what’s possible for publishers using the rapidly developing so-called “reader-centric” publishing business model, Ractliffe mentions companies/platforms such as Wattpad and Callisto, which are “using AI to scan stories for commercial success,” collecting data from Google and Amazon searches at a rate that would probably impress even the people behind Cambridge Analytica. Analysing that data apparently lets them know what consumers (readers) want, and “if there is no book to fulfil that need they aim to bring one to market in as little as nine weeks”.

Even that’s probably too long for those hoping to cash in on consumer waves, and for all those writers who labour for years on an idea that then, word by word, turns into a book, we can only surmise that it’s the extreme mass-market end of the publishing spectrum being talked about here.

Nevertheless, when Ractliffe quotes the founder and CEO of Callisto as saying that “based on his own analysis, acquisitions editors pick a winner only three per cent of the time,” it’s pretty clear that the only criteria for winning here is profit. This CEO also says, “In the world of almost infinite consumer data, the idea that you cannot say with specificity what a consumer will want to buy seems frankly ludicrous.”

Maybe he’s right. Maybe what I think of as my curiosity as a reader, which I cherish and nurture with many online searches as well as much meandering through books and writing that often takes me to a dead-end but sometimes opens up vistas I’d not been aware of, is really just my self-regard, a kind of intellectual box-ticking which is not really what I want, but what I think I should want. And maybe, then, an algorithm that collects my data and matches me to a consumer profile will provide me with a book written in nine-weeks that satisfies my needs.

That seems to be what VP, Marketing Strategy, Consumer Insights and Analytics at Penguin Random House US Erica Curtis is saying is the future for this kind of publishing (all publishing?), when she says (in rather cliché-laden language) that their goal is “putting out high-impact deliverables”.

“A lot of our work is reactive – solving problems and answering questions,” Curtis tells Ractliffe. “We are beginning to field questions around editorial direction − suggesting topics and storylines for themes based on what people are interested in.”

Market-driven storylines! Happy thought indeed.