It’s a legal compromise really.
From an article on the New York Times:
The design of modern billing blocks illustrates the tension between two intersecting interests: studios want uncluttered marketing materials, and industry organizations want their members prominently and fairly credited.
Thus, it is neither accidental nor for aesthetics that the text in most modern billing blocks is tall and highly condensed. To ensure that billing block credits are legible, the Directors Guild of America (D.G.A) and Writers Guild of America (W.G.A) require that their members’ names are at least 15 percent of the size of the type used for the artwork title. (i.e., the name of the movie, as set in the main body of the poster). If the artwork title has words of varying size, the height of every letter is measured and an average calculated.
The D.G.A and W.G.A also mandate that the credit titles (e.g., story by or directed by) be no less than half the size of the names to which they refer.
So really, in the face of legal obligations this is probably the best they can do. Still just about legible, fulfills the legal requirements to have the key names prominent while also not compromising the aesthetics of the poster artwork too much.
Unfortunately the source article linked to is an image rather than a text so not very accessible. I’ve transcribed the most relevant element of it above.
I think that the closest answer is: a compromise between visibility and legibility (which of course created some standard with time, or even: best practice).
The main purpose for that is the need to fit as much text as possible, while still keeping the letters quite big (this is why the letters are almost always capital here as well). Fonts used here are mainly ‘condensed’ subsets of bigger families, or dedicated ‘condensed’ fonts (you can also find extra-condensed, compressed, narrow or similar naming conventions – you can find more about taxonomy of the type here). One great example is UltraThin Condensed. If a standard font was used here, it would occupy too much space; on a contrary, decreasing the font size of the font would lead to the names being smaller and less visible).
I also believe that, with time, more and more text needed to be put on the posters (so far not-so-important actors didn’t want to be excluded from official credits on the posters, and also new areas became important – special effects for example). You can feel it when you look through the posters in the Wikipedia article about film posters.