Movies labelled as "Art Films" often like to display their titles as though there is some sort of pride within it. Somehow calling a film "art" implies that it is elevated in terms of quality above other types of movies, and this is simply not true. To begin discussing the subject of Art Cinema, we should look at what exactly defines the term and how it compares to another dominant mode of filmmaking. Generally, when critics use the term "Art Cinema", they are referring to European-produced films produced during the 1950's and 1960's. Many of these films are shot in black and white, and make choices intentionally trying to challenge the classical model of filmmaking.
The "classical model" is based on a simple rule: story comes first. In a classical film, the narrative is the primary focus, and everything within is subordinate to it. This means a few specific requirements are in place, namely in that the narrative has to progress as clearly as possible. The characters should be clearly defined, with specific motives and goals (even if those motives or goals might change over the course of the film, but even then those changes have to be clearly marked). Story should of course progress in a linear casually-driven fashion. One scene leads into the next, and any stylistic choices have to serve the narrative. There was also the fact that classical films are required to imitate reality (or at least, what the audience perceives as reality), and appear as a literal window into another world.
The "classical film" is often associated with the Studio Era, mainly because during this time it was strictly enforced as the only way to make movies. However, it goes back long before, originating in early silent film with devices such as the continuity system. It also is hardly confined to American cinema; the entire French New Wave was based on critics being inspired by Hollywood films and trying to find new ways to utilize their structure while also putting a new spin on it. Today there is more room for flexibility and experimentation, but the classical film still remains the dominant mode of filmmaking.
"Art Cinema" is a counter of sorts to classical filmmaking, and one that aims to resist it by trying precisely the opposite. This movement was popularized in Europe during the 1950's, though there is some dispute over precisely what qualifies as Art Cinema. The two most famous directors to take part in Art Cinema are arguably Federico Fellini and Ingmar Bergman, though other directors have produced iconic Art Films such as Michelangelo Antonioni's L'Avventura. Some critics even label Jean-Luc Godard as a major pioneer of Art Cinema even though his work with the French New Wave purported to do precisely the opposite (not to mention his movies were despised by Ingmar Bergman).
Art cinema rejects causality and clarity in favor of ambiguity. Often plots remain unresolved or open-ended, at least as much of a plot as can be applied to these films. Often the narratives were subjective and open to interpretation, inviting speculation from the audience. Characters motives are psychologically-driven but often times their goals or motives are kept vague. In theory, this allows the audience to see art films as an exercise, movies that make them think and come to their own conclusions.
That is a very good description of the films of David Lynch, but as far as "Art Cinema" is concerned this is simply what critics like to say. In truth, Art Cinema cinema claims to be a new way of breaking the trends of Hollywood, but really all it produces is a jumbled and incoherent mess that tries to use pretty images to make itself look more intelligent than it really is. The use of the term "Art" in its name seems to imply that this different approach somehow elevates Art Cinema above Hollywood narratives in terms of storytelling, but this is not true. If anything, it downgrades it.
This is not to say that there is anything wrong with subverting Hollywood conventions. After all, quite a bit of Hollywood storytelling owes itself to movements in other countries (among other things, film noir has some of its roots in the German Expressionist films of the 1920's), and some American filmmakers implemented elements of the Art Film to far greater effect. David Lynch is a perfect example of someone who brought Art Film techniques to the mainstream while also making decent films that succeeded where his predecessors failed.
Lynch's films are precisely what the likes of movies like 8½, I Am Curious (Yellow), and L'avventura attempted to do. A lot of his movies, like Eraserhead, Lost Highway, Mulholland Dr., and Inland Empire are based on creating a subjective experience. Lynch has gone on the record multiple times refusing to divulge the meanings of his films preferring that the viewer draw their own conclusions, and people have. None of his films ever fit perfectly into the classical model, including his more straight forward ones like The Elephant Man and The Straight Story, and they are based on the idea that different viewers will find different meanings in his films.
Stanley Kubrick is also a director capable of exploring these subjective narratives when he wants to. One of his greatest achievements, 2001: A Space Odyssey, ends on a very open note with literally nothing explained about what happened in the last half hour or so of the movie (though there is a popular explanation often accepted as the "official" one). Similarly, Eyes Wide Shut leaves a lot of plot threads unexplained and unresolved. We never find out anything about the bizarre secret society beyond what Bill Harford observes or is told afterward, and even then quite a bit of that information is open to debate.
More recently, Jonathan Glazer's film Under the Skin used a surrealist and non-conventional approach to depicting its story. Like the old art films, it was character-based, with ambiguous goals and motives in its protagonist. Looking at the narrative structure, there is a lot of influence from a variety of sources. Bits and pieces of Kubrick and Lynch are visible throughout, and there is even a bit of the Italian Neo-Realist movement in there. It would not be far-fetched to assume Glazer drew somewhat from Art Cinema as well, and yet somehow it works.
This raises the question, what is it that Lynch, Kubrick, and Glazer have that Fellini and Antonioni do not? I think it is a matter of looking at why those choices were made? Under the Skin's surrealist approach works because it helps to put the viewer into the mind of an alien. Mulholland Dr.'s subjective structure allows Lynch to explore the darker side of Hollywood filmmaking. As weird as Eyes Wide Shut can get, there is still a basic narrative that can be followed, even if they are left to think about specific moments within it. Context is a crucial part of the narrative, and just because breaking the rules can work does not mean it always will.
Italian cinema had already managed to briefly establish its own identity with the Neo-Realist movement. Films like Bicycle Thieves were still able to convey straight forward and comprehensible stories while still trying something new for the time by focusing on everyday occurrences and addressing the lack of closure that often happens in real life. A lot of Art Films may have broken the rules, but perhaps did it too much and in ways they did not need to. To provide a straight forward example, L'avventura had a very basic premise: a girl disappears under mysterious circumstances. A classical film might have used that as the setup for a mystery, with the plot centered around finding out what happened to her and building up to a big reveal at the end.
As it was made, her disappearance is never explained, and instead the focus is on how this one event affects those who with her before it happened. Already they are defying the conventions of a typical Hollywood narrative, and this is perhaps all they had to do. The movie could otherwise have been a fairly straight forward and easily comprehensible narrative, and it might have worked better that way. Instead, the final product was so disjointed it took a Wikipedia synopsis even to understand the key plot points.