I still puzzle over what differentiates literary from genre.
"I know it when I see it" is so often the final definition. Character driven, high-minded, complex, many layered.
I love this definition: "Work of great sensibility and refinement, where it can take up to 500 pages for nothing to happen." from Frank Baron
In the end, it is, like almost everything in life, a spectrum. There are books that are genre, no question. There are books that are literary, equally without question.
But all along the way, there are books whose ideas, whose writing style, whose detailed evocation of place, mood,or character are so eloquent that, even though there is a romance, or a crime, or an alien invasion, they are also literary.
Over and over we read that everything in the story must move the story forward. Every scene must fill at least two roles: establishing motivation, explaining necessary backstory, revealing character, and so on.
Perhaps part of the difference, then, is that in literary fiction, you have greater latitude in how much you reveal or the pace at which you move the story forward.
In literary fiction, you can have, as in "The Life of Pi", paragraphs of imagery and impressions that serve to create the mood of the story and the way in which the character views the world that in a genre book would be considered too slow. "Tighten it up" I hear an editor cry. "You'll lose your readers' interest."
And you would, if they were reading it as a detective story or a romance story. But when the book aspires to be something besides -- note - not more than -- science fiction or thriller; when the writer wants to draw you into a world or a state of mind; then you are entering the world of literary fiction.
And we are left again with personal preference. I might read it and see it as a literary fantasy. You might read it and see it as fantasy only.
This is part of why I favor the label Speculative Fiction. It allows the idea that elements of fantasy or science fiction are not the absolute arbiter of the story's classification. A space ship can carry the tragedy of Hamlet in its crew, and a mystery can encompass the eternal question of what it means to be human.
I could wish that we weren't so determined to make arbitrary distinctions. I realize that, like standardized testing in the schools, there are times we need to stick labels on things. But if we are interested in understanding what books have to offer, then we need to look past the labels publishers and booksellers require and accept a book on its individual nature.
Any time we break down barriers, we are enriching our world.