Computer science has some really rich concepts and metaphors. One idea is the concept of JIT which stands for Just-In-Time, usually in reference to compilers. The meaning is basically this; if you have some expansive body of work, do the bit you need right now and worry about the rest only as you need it. This works out well in practice because you often only need to do a fraction of the work you think you need to do.
Word building is the term writers and developers use to describe the process of creating a world for a game or novel. This usually means a few maps, notes on characters, and anything special or unusual about the world. It can be a few pieces of paper to a full on database. Curious about the world building process? Check out this article on building worlds layer by layer if you want to learn how to build your own world.
World building should be approached differently depending on your goals. If your game is tightly scripted, a little like a novel, the world is more a backdrop. While you can create a fully fleshed out world, it's not required for what the player experiences. If your game is more like Dwarf Fortress or Daggerfall, then you can explore the entire world! In this case you need to make a consistent world with a lot of content for the player to explore.
Time building a world isn't wasted time but it's not time spent making a finished game. A well built world is something you can use again and again but you need to know your goals. If your goal is to make a game, then avoid delaying tactics and dive right in. If you spend a lot of time creating a world but never get to the part where you make the game you're not going to release anything! Don't use world building as a delaying tactic to avoid starting on the game.
JIT World Building
Every game should have something unique about the world or characters that occupy it, otherwise it's not interesting. You need a hook.
Spend time coming up with an initial premise. Once you get the premise you can start making the game. Fill out the rest of the game world and story as the game requires it. To ensure the world is consistent, document any decisions you make. This way you build the world in parallel with the game.
Build the world in parallel with the game.
The Balancing Act
When building a world in parallel with the game - you'll need to task switch a little. It's hard to know where and when to concentrate your efforts. Listen to your gut but also prioritize getting your main game loop finished as early as possible. Once the main loop is finished you can start to iterate and build out areas by adding more and more story and world development.
Fantasy worlds tend to have mountains, a long journey, dragons, orcs and magic swords. They're following the works of Tolkien and that's ok. It's ok to pull parts of your world from the existing works, then work hard on the places you're most interested in and find most fun to change. You don't have to develop a world from scratch! (Though if you do enjoy this thing that's cool too!)
People who like to playing games like to create. That's a massive source of power which, in video games, has yet to effectively tapped. Minecraft with it's freeform world building is a great success but the players aren't creating something cohesive. It's missing a curation function - but this seems possible.
The "Open Gaming" movement associated with table top gaming, sometimes encourages and allows player to create and sell their own supplementary materials. Again this is a little reminiscent of the current state of modding community for games like Elderscrolls. It feels there are a few more steps that could taken to make some really interesting.
Are you working on an extensive design doc, story bible but haven't touched an IDE? Might be time to re-assess and make sure you're actually interested in making a game and not just world building. If you really want to make a game then put the giant docs aside for now and start creating the game pulling from the docs as needed and extending them as the game world requires it.