It’s long been known that supervisors often apply different standards during employee evaluations and that personal biases can often slip into the process.
However, there is now empirical evidence that calibration committees–which serve to adjust employee ratings assigned by supervisors–are an effective approach to help ensure consistency and fairness across performance ratings, according to a three-year academic study.
The study, titled “The Role of Calibration Committees in Subjective Performance Evaluation Systems,” is scheduled to be published in an upcoming issue of Management Science journal. It was conducted by Will Demere, assistant professor of accountancy at the University of Missouri; Karen Sedatole, professor of accounting at Emory University; and Alex Woods, associate professor of accounting at the College of William and Mary.
After collecting data on more than 1,300 performance evaluations from a large, multinational corporation that used calibration committees, the researchers found that committees only adjusted about 25 percent of the ratings, but they lowered them four times as often as they raised them.
Through these adjustments, researchers say, the committees improved the consistency of ratings across supervisors and mitigated leniency bias.
However, one of the negative consequences of these adjustments, researchers acknowledge, is that they compressed most ratings into the middle range, otherwise known as centrality bias.
“In general, we felt performance evaluations would be better if they differentiated between employees more,” Demere says, “which is helpful to determine who should be promoted or [awarded] signing bonuses.”
But researchers also identified how supervisors applied what they learned from the process. For example, if the calibration committee adjusted a manager’s ratings for an employee upward, Demere says, that manager tended to give that employee a higher rating the following year. Likewise, if a rating was adjusted downward, that manager often submitted a partial downward adjustment the following year.
Demere says there’s a general tendency for supervisors to be more lenient by giving higher ratings, partially to avoid conflict with subordinates.
Researchers also found that when more layers of the company hierarchy existed between committee members and employees they rated, fewer adjustments were made. But when there were less layers between committee members and those being rated, there were more adjustments made.
With fewer layers between them, committee members may have worked with some of the individuals, Demere says, which means they may have been better informed about those workers’ skills, productivity level or work style and therefore more likely to make an adjustment.
For the past 15 years, Dick Grote, president of Grote Consulting in Dallas, has helped large companies create and facilitate calibration committees. Most of the committees he has worked with–which typically consist of between three and six supervisors, department heads or direct reports–are also facilitated by an HR professional who delivers a brief orientation at the beginning of the meeting to “go over the ground rules, the process and ways of screwing it up,” he says.
“Some of the ground rules are that every person must be actively engaged [during the meeting]. There’s no dropping out, no checking phones for email messages and they must tell the truth.”
Grote believes calibration committees offer several powerful benefits, including giving management a sneak peek into how well supervisors know their staff, which ones support their direct reports and which ones don’t, and exposes when supervisors may argue too much on behalf of a weak performer or not enough for a superstar.
“[Management] gets a very good, first-hand picture of the strength of their supervisory cadre,” he says, “and that’s a benefit that’s often overlooked.”
However, the process often presents HR leaders with real barriers, Grote says.
Besides being an administrative burden, or “one more meeting to go to” as Grote says, some supervisors don’t want to defend their appraisals before their peers or are very good at persuading members to agree with their unjustifiable, high employee ratings. Other times, supervisors taint the calibration process by telling employees they were forced by the committee to assign them a low rating.
Although many HR professionals facilitate calibration-committee meetings, not all of them provide hypotheses based on analytics, says Asumi Ishibashi, a senior consultant in the talent and rewards practice for North America at Willis Towers Watson.
For example, committee members need to know if employee ratings are, for example, typically higher in the Southeast than in the Northeast, lower for certain departments or skewed higher for specific segments of the workforce.
“That’s the intelligence that HR should be prepared to bring to the table,” she says. “You’re not necessarily coming up with a solution or reason for everything, but [you should be] prepared to share hypotheses that are going to help facilitate a robust discussion by the calibration committee.”
It’s also vital that HR sets the rules of engagement in advance of the meeting, she says. HR leaders can do this by discussing who has veto power if members can’t reach an agreement or the need for all committee members to share comments about employees. HR leaders must be equipped to offer the committee guidance, talking points and documentation about how they can explain ratings to their staff without bashing the process or other committee members.
“It’s the whole idea of how you can make your current process better,” Ishibashi says, adding that this strategic process can help drive business performance. “If done right, it adds value to the business.”
