Companies and families are a lot alike: rewarding, passionate, dysfunctional, infuriating. Cram the two together, though, and life really gets complicated.

By some estimates, nearly one-third of family businesses don't survive the transition from the first to the second generation. The potential pitfalls are too numerous to count, but there is a common thread: Family businesses tend to act less like businesses and more like families. That leads to poor decision-making, resistance to change and, ultimately, an inability to compete.

Perhaps the biggest problem for family businesses is lack of structure. "Formalization is a dirty word when it comes to family-owned and operated businesses," says Richard Dino, executive director of the Connecticut Center for Entrepreneurship and Innovation.

• In Pictures: How to Run a Family Business
• In Pictures: The Do's And Don'ts Of Co-Preneurship
• In Pictures: How To Get 'Em To Call Back

Little surprise, then, that only 37% of family-business owners bothered to craft formal business plans, according to a 2003 survey of 1,143 family-run companies with at least $1 million in sales by the MassMutual Financial Group and the Raymond Family Business Institute.

Small businesses have to stay flexible, of course, so not strictly hewing to a fully defined plan isn't necessarily a recipe for disaster. Still, there needs to be some semblance of order. While enforcing draconian lunch breaks might be over-the-top, establishing basic guidelines like job descriptions, compensation standards and objective performance measures is probably a good idea.

Another hazard for family owners: hiring their own, even when they aren't a good fit. One cautionary tale--care of Joel Getzler, vice chairman of Getzler Henrich, a consultancy that specializes in turning around family businesses--involves two brothers who owned a financial services company in New York City. (Getzler won't divulge names.)

The trouble started when one brother chose to hire his 35-year-old son as a vice president in charge of raising capital from high-net-worth investors. Bad move. Though the son had no experience in the business (indeed, he sent clients running the other way), he came on at the same level and pay grade as his uncle's children, who had grown up in the company. "The whole issue created a terrible ruckus," says Getzler. "They had some great yelling matches."

• In Pictures: Most Expensive Celebrity Divorces
• In Pictures: World's Most Romantic Hotels

To settle things down, the brothers agreed to put the son in charge of back-office operations, from purchasing supplies to monitoring returns on the company's portfolios, and they trimmed his pay. That way, he could learn about the business while being productive at the same time. And his cousins felt vindicated, too.

Then there are succession issues. If you assume Junior is salivating to take the reins, just look at 33-year-old Lachlan Murdoch, heir apparent to News Corp., the $27 billion (sales) media conglomerate founded by his father Rupert. Back in 2005, amid a swirl of rumors, Lachlan left his vaunted post as deputy chief operating officer to start his own media-investment venture in Australia. (Meanwhile, Rupert, now 76, has yet to name a successor.)

The big question for family owners: When times get tough, which comes first--the family or the business? Make the right moves and you don't have to choose, says Getzler.

His first tip: Take time to communicate, openly and honestly. Schedule regular gab sessions about the business--and leave family matters at home. If that doesn't work, debrief with a business psychologist who can act as a mediator.

Along those lines, family businesses should also take on outside advisers who can make dispassionate decisions about touchy matters. The trouble with advisory boards, warns Getzler, is that they only work when you use them. According to that MassMutual/Raymond Institute survey, of the businesses with advisory boards, about half only met with the boards once or twice a year, and 13% never met at all.

While traces of nepotism are virtually unavoidable, the key is not going overboard. Take Getzler himself. He spent eight years raising money for fledgling companies before joining Getzler Henrich, his father's consultancy, at age 29. But rather than coming in as Dad's right-hand man, Joel started as a lowly consultant and didn't even report to his father. Better yet, his salary (half as much as he pulled down as a VC) was set by an advisory board.

No matter what tactics you use to manage the complications of a family business, sometimes the best strategy--for both the business and the family--is selling out. The hard part, as with most business decisions, is making that decision in time.

"We try to get people to realize they should let the family move on in life," says Getzler. "Sometimes, though, families hold on. And then they have nothing."

• In Pictures: Sex Lives of the Super-Rich