This question is one I’ve thought about a lot. I’m guessing most other photographers have, too . If you’ve shopped for light modifiers you’ve encountered a bazillion light mods that fall predominantly into just two types: round or square-isn. Round modifiers would include things like umbrellas, octaboxes and beauty dishes. Square or rectangular-shaped modifiers are things like softboxes, reflectors, scrims and light panels.
Photo gear and marketing hype go hand in hand. But for me there are just a couple of factors that determine which shape of modifier I’ll choose for a shoot. Number one is the effect of the catchlight in my subject’s eyes. I prefer a round catchlight—maybe just a subconscious preference for a light source shaped like the sun. If I were simulating an open doorway or the light from a window, a softbox would be my choice instead.
There are a bunch of other, secondary considerations when choosing a modifier that are partly determined by the shape of the modifier but also by the material and construction of the modifier itself. For example, I determine the quality of light I want: directional and harsh, with very well-defined areas of light and shadow; or soft, diffuse, enveloping. To get the effect I want, I’m less worried about the shape of the modifier than by the size of the light modifier relative to the subject (the larger, the softer and more diffuse) and the construction of the modifier (a grid will focus the light, or a modifier with multiple layers of diffusion material will soften it). For me, the shape only becomes a factor when I want to control the light precisly and thus I might use a square softbox rather than an octabox because I want the light to have a more defined fall-off or edge.
You can find descriptions of the effects of different light modifiers, but one of my favorite write-ups is this one by light guru Paul C. Buff.