Just some quick additional suggestions, that sometimes complement what others have already said.
1) water solution: I never understood why killing the player with a shark or something. Just let him/her swim infinitely (like with a proceduraly generated infinite ocean). That alone would closer resemble the idea of how distant it would be in real life to cross an ocean by swimming. The player would just get tired at some point of going forward and then would decide to reload (because going back all over again would be just as tedious).
Of course, if you have modern boats, ships, jet-skis, etc, and the concept of gas as resource, that works well too and then it becomes part of the item number 4 below, since the player can just sail until running out of gas. Then, if he/she can swim, it becomes again a problem of just letting him/her swim forever.
2) scene barrier: it does not have to be mountains. A dense forest, as someone else told here. A canyon with a beautiful view. A part of a city with too many buildings in the way and the only possible streets blocked. A huge fire destroying parts of a city, or a gas leak, or a radiation accident in part of a city that blocks the way. If the city was invaded by aliens, maybe the gigantic spaceship is itself blocking the path to other parts of the city as it crushed when arriving. Or, why not, you can have flying islands like in Zelda Skyward Sword, where the boundary is given by not-falling-to-death.
3) challenging who tries: you can always increase the challenge exponentially as far as the player goes. Imagine an open-world in the country fields. The player crosses farms in the direction of the inner-lands of your world. More and more enemies show up with no power-up, health kits or something in the way. If you make these field camps procedural and increase the challenge, the player will eventually get killed without having the feeling that he/she was cheated by an artificial border. I like this approach because it even creates in the mind of the players the impression that there might be something hidden ahead (of course, you have to take care and don’t abuse on this illusion, to avoid the players becoming mad at you!).
4) lack of resources: if a player has to eat/take medicine/whatever, and/or if vehicles have gas, you don’t need boundaries. Let the player and/or its vehicles die. That is particularly useful in a space game, where creating a non-artificial barrier is quite difficult. In other words, if that suits your gameplay framework, make the open world finite by not being accessible due to being impossible to accumulate resources to go beyond a give distance.
I won’t keep going with a ton of suggestions based on storyline and gameplay. My reasons: first I don’t know which type of game we are talking about. Second: frankly, as nice as it can be to have a story-line or gameplay related solution such as failing missions because some hostage has died when you tried to get out of the borders, I don’t think that is exactly what you meant and, on the contrary, I found those to be usually harmful for immersion. But if you happen to be interested in these types too, let me know and I can drop some ideas that I have either seen or thought.
Also, as a general point, notice that more than once I touched procedural generation. Really, I think procedural generation is your friend here. You don’t need to use it only for having infinite worlds. I always say we should start using it much more often to make the finite open-games more realistic and diverse. Using it for a fake endless border would be a neat use.
Different games have different requirements in how realistic they are to their genre, e.g. FPS games can constrain to a building, whereas RPG games like Rust / DayZ / Skyrim have larger and more open world maps to suit their style.
Some common ones across games (and examples) include:
- Constrain to an Island and:
- Ruin the only bridge out (GTA 3, Vice City, Unturned, etc)
- The ferry only comes sometimes
- You’re trapped (Dead Island, Just Cause, Crysis)
- Forest/Open World games with large continents (parent maps):
- Blocked by trees except for a few areas (Pokemon, Legend of Zelda)
- Blocked mountains (Skyrim, as you pointed out)
- Out-of-bounds areas (A higher governing force controls where you can and can’t go, meaning you are killed when exiting):
- Turrets shoot you down (Borderlands, Batman: Arkham City)
- You fail the mission (CoD, probably Battlefield too)
- Objectives unlock continuation (Literal, not the metaphoric idea embodied by gaming):
- More enemies spawn until you do and you die eventually (CoD: Zombies, Left 4 Dead)
- You get moved by a higher power:
- Picked up / teleported back (Imagine AM from Harlan Ellison’s I have no mouth and I must scream (the story, not the game adaption))
- Don’t have borders, increase map size until your program cannot handle it
- Walking and discovering increasing size of world (Minecraft and the concept of program failure at a certain distance from the centre of the map).
- In a city:
- Barricades / cars block you from moving (Left 4 Dead)
- Buildings blocking your path (Pretty much every urban level of FPS games)
- On a cliff / high place:
- If you traverse past the edge, you fall off and die (A few Borderlands levels)
- Your character physically cannot handle the outside environment:
- Other planets & the moon with an artificial environment (WORLD END ECONOMiCA)
I am a bit hesitant to add this option, but it could work.
When seen in 2D (neglect height for a moment) the inside AND the outside of a torus are endless. They simply wrap around on both axes.
Placing your characters on a shape like that could be tricky. You could always go easy on yourself and get a less exact torus.
Now we’re talking 3D objects, how about a plain old sphere? They don’t necessarily have to be as big as Earth. Make them the size of a small moon, like Dactyl, the moon of 243 Ida. I can imagine writing a story for such a setting can be quite the challenge (same goes for the torus).
Mario Galaxy makes use of this by making very small planetoids. If it works for Nintendo there’s probably some merit to the concept.