ECI^® Blog

One of our associates passed along this New York Times article about Google’s Project Oxygen to me earlier this week.  Google wanted to identify the factors associated with high performing managers.   Being the experts they are with data analysis, they sliced and diced all of their performance review ratings and other anecdotal information to identify the behaviors that are unique to their best managers. They were surprised to find that technical skills are not what enables good managers to make the list.

I liked this article because it more or less confirms what we have been doing in our research for the past 15 years.  Our business, ECI, founded in 1996, is built upon the identification of high performance behaviors in a variety of environments and roles using statistical analysis of performance metrics. Like Google, we have found that this type of data analysis yields a valid and reliable formulation of the root cause for success.

But since we have been focusing all of our attention on identifying high performance behaviors within organizations, here are our best practices that Google’s analysts might want to consider on the next round of Oxygen studies:

1. It is not sufficient to screen for key words in performance reviews and anecdotal information. While that practice might put you in the ballpark, it won’t get you to your seat. There is too much variance and inconsistency in prose type performance reviews. If you really study a block of performance reviews, you find that most managers are not appropriately trained in giving objective, actionable feedback, nor are they consistently assigning ratings to performers.  This inconsistency of ratings across the review process skews the data.
2. Use force rank against a Behaviorally Anchored Rating Scale to identify quartiles of performance for your overall population.  The overall ratings assigned in the standard performance review process cannot be relied upon to indicate who is the better manager. In our studies, we find that in 60% of companies, ratings are assigned for some other purpose than to evaluate actual performance levels. These include attempting to norm a population to a bell curve for compensation purposes, feeling that someone deserves a raise and having to justify this with the performance rating, and favoritism by the manager for the most politically savvy performers on the team.
3. Use multiple measures to confirm or overturn the presence of key high performance behaviors.  ECI’s rule is that if you identify a factor in one segment of the study, you must verify its presence in another segment in order to consider it applicable to the model.
4. Use valid and reliable metrics, such as indices, personality assessments, and other proven tools to identify core performance behaviors and behavioral preferences. If you incorporate a couple valid and reliable metrics in the study process, you can statistically compare the findings from these more rigorous tools to the less objective sources of data in your study to know with good certainty that you have proven a relationship to the high performance behaviors/factors you identify.
5. Make sure you include results from job analysis within your study process.  By observing the work in context, using a standardized interview form designed to assess the work environment, and identifying differentiating performance factors using this process, the criteria you establish should  pass the muster of the EEOC, if you decide to use this model for selection or promotional purposes.
6. Use professional statistical tools, such as SPSS, to confirm the validity around your model. When you put people into a room and say “does this look right to you?” or “how would you modify this finding?”, the only thing you are verifying is face validity. That is insufficient, in my estimation, to devise a management development program or another talent management process. You need the numbers to prove your model. Hopefully, the standard you achieve is at least a correlation significance of .70 against the ratings you used to identify your high performing population.
7. Don’t forget to look at the entire population, not just the high performing group. If you only study the top performers, you don’t know if the factor you identified is present for everyone in the group or only high performers possess it. In our studies, for example, we find that all sales people within a large sales force have good self-confidence, can withstand rejection and are motivated to persuade others. While these factors are critical to selling success, the only thing we can say with certainty is that the original screening process used to hire the sales force is doing a good job of identifying these factors. These are the rudimentary factors associated with all successful sales forces; they are essential, but they do not help us to identify the additional factors needed for success in a specific company culture, marketplace or customer group. The unique factors are those that drive exceptional results, lower turnover, and higher job satisfaction.

Google did recognize that generalized industry principles and recommendations are not good enough to really drive their organization’s unique high performing manager behaviors. I commend them for that perspective. I would love to take a look at their data and make a couple of recommendations on how they might enhance the validity and reliability of their study process, however. That would surely be a wonderful conversation.