While bad bosses can certainly drive employees away, this study shows that they may not be the only reason most workers leave a company. A 2015 Human Resources Online study found that three out of four workers said the boss was the most stressful aspect of a job. But its also possible that structural problems with job duties — hours worked, type of work — could have a larger impact on employee engagement than management alone.