Companies that have rejected hierarchies have typically lacked diversity and, in at least one case, faced scandal. A new crop of them is aiming to find a middle ground.