what is a good balance for dungeon design in terms of linearity vs. nonlinearity

What would you makes a good balanced dungeon progression in terms of item usage? should every dungeon use an item from the previous dungeon or should it only apply to a few while others focus on their own exclusive gimmick?  I presume it is best to have mandatory dungeon items where they must be used in the dungeon unlike the moon pearl, which can be skipped, but is still required for the dark world.  

@Puzzledude, how do you generally design your hacks in terms of its progression? what approach do you usually go for?

@Zarby.

what is a good balance for dungeon design in terms of linearity vs. nonlinearity

There is none. Do note that critics will find your dungeon either too linear. And if it is not linear, they will call it backtracking. No good no matter what you do, thus it is best to design according to your own wishes. However do note: a standard Zelda dungeon is not linear, and thus will have backtracking. This is how ALTTP was made.

What would you make a good balanced dungeon progression in terms of item usage?

Small key, unlock small key doors, gain access to chests, peg switches, gain big key, gain big chest item, use this item, get to the boss. Parallel Worlds also has: big chest key separated from big door key, which really makes it interesting.

should every dungeon use an item from the previous dungeon or should it only apply to a few while others focus on their own exclusive gimmick?

You choice. Usually the entrance of (for instance) dungeon 2 is blocked with main item from dungeon 1, but this is not necessary at all.

I presume it is best to have mandatory dungeon items where they must be used in the dungeon.

Indeed.

@Puzzledude, how do you generally design your hacks in terms of its progression? what approach do you usually go for?

Carefully though out game-play in terms of global item-gaining. Otherwise you don't construct cronologically, but geographically, in which you then use cheats to test out all aspects of a certain overworld area, including sprites, music etc etc (you thus test all visits which will occur in different game-play moments). Dungeons are done last.

My biggest issue with the item distribution in native Zelda3 was that the titan's mitt and the moon pearl could be skipped after defeating the boss.  to be fair, you are given information to get the moon pearl and you should locate the big chest in every dungeon.  

one idea I had was to have a second entrance in the thieve's hideout or program the large grey blocks to be only liftable by the titan's mitt (which some suspect is a developer's oversight).  as for the moon pearl: I guess the best option would be to enter the dark world to reach a second section in the dungeon, but the path is blocked by rocks or bushes.  

because the magic mirror is necessary as a fail safe item, i left the location of it alone. i decided to move the moon pearl to sahasrahla and prevent bunny link from leaving the pyramid section. i thought of an idea of having bunny link opening chests, but then realize it could cause a program paradox if he gets the moon pearl.

one thing i cannot defend Zelda 1 was the item placement and I am working on a hack that fixes this problem by placing the items that must be used against the boss.

I would usually try to limit branching paths to only 2-3 at a time that each only go about 2-3 rooms deep. Otherwise it's easier to get lost and requires more back-tracking which I personally find very annoying.

Technically you can design the game however you want.

Games are really an art form. There is no right or wrong way to approach something.

The only thing that really matters in the end is if people like your game and that it has some value in it which others appreciate.

But there are also some unwritten rules which games, in general seem to follow.

If you want a good balanced game, play some other good games and study what they did for balance. Then try and recreate that same vibe in your game.

I second Erock's suggestion. Before building some of my dungeons I studied some of the original dungeons by making notes like:

-find way to open door
-path branches, one path is locked
-follow unlocked path to find key

That sort of thing helped me get ideas a number of times. My original notes were much better than what I have typed out here, but eventually I began to refer to this stuff as "dungeon theory".

Here are the criteria that I keep:

All dungeon items that are mandatory must be used to finish the dungeon, but not necessarily more than once.

No puzzles or locked rooms will cause players to get stuck where they either need to use the mirror to warp back or reset the game (Zelda 1 has this issue many times, which I am fixing in my hack).

Optional keys are allowed, but it does not always have to be the case.

Bosses do not need to use the dungeon item to be defeated, but it does help make the fight go much smoother.

Keys also must match the same amount of locked doors and no one way paths with a dead end will have a key behind a locked room (e.g. Temple of Darkness's dark room next to the big chest).

After observing the Hylian Legacy hack, one idea that I came up with was to have the Pyramid of Power as an accessible dungeon once you enter the Dark World, but you cannot reach Ganon until after Ganon's Tower. The advantage of entering the dungeon early would be to obtain the dungeon item. It is unfortunate that native Zelda 3 never took advantage of this situation and OoT did it with the Water Temple.