The Validity and Utility of Selection Methods in Personnel                                                                               

 Psychology: Practical and Theoretical Implications of 95 Years of Research Findings.

                                                                   Frank Schmidt

                                                                       Abstract

This presentation is an update of an article Jack Hunter and I published in Psychological Bulletin that summarized research findings on the validity of job selection methods up to 1998. The two co-authors on this update are In-Sue Oh and Jon Shaffer, who are former PhD students of mine. The ensuing 12 years have seen additional research findings that improve the accuracy of the validity estimates presented in the 1998 article.  During this time, a new and more accurate procedure for correcting for the downward bias caused by range restriction has become available (Hunter et al., 2006). This more accurate procedure has revealed that the older, less accurate procedure had underestimated the validity of general mental ability (GMA) and specific cognitive aptitudes (e.g., verbal ability, quantitative ability, etc) by 25% or more. Also, the increased availability of primary validity studies has allowed new and expanded meta-analyses of some selection methods, which has refined and changed some of the validity estimates for the prediction of job performance. For some personnel measures, these new data have produced important changes in estimated validity and incremental validity over GMA. For example, an updated meta-analysis shows that job sample or work sample tests are somewhat less valid than had been indicated by the older data. Finally, meta-analytic results are now available for some newer predictors not included in the 1998 article. These include Situational Judgment Tests (SJTs), measures of so-called “Emotional Intelligence”, person-job fit measures, person-organization fit measures, and the personality trait of Emotional Stability.  No new meta-analyses of validity for the prediction of performance in training programs have been reported. But application of the improved method of correcting for range restriction has changed the earlier estimates of validity for performance in training programs.

For each of 25 personal selection procedures used to predict job performance, I will present the mean operational validity as revealed by meta-analyses. I will also present the incremental validity (if any) produced by each procedure over that produced by GMA. I will present this same information for 11 procedures used to predict performance in job training programs. Results show that many procedures that are valid predictors of job performance nevertheless have little or no incremental validity over that of GMA. This reduction in apparent incremental validity results from the relatively large increase in the estimated validity of GMA resulting from use of the more accurate correction for range restriction. At the time of the earlier 1998 article, it was apparent that GMA plays a central role in the determination of both job and training performance. However, the updated findings indicate that the dominance of GMA is even greater than previously believed.

                                                                References [Includes recent relevant articles not cited in the Abstract.]

Hunter, J.E., Schmidt, F.L., & Le, H. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91, 594 – 612.

Schmidt, F. L. (2010). How to detect and correct the lies that data tell. Perspectives on Psychological Science, 5, 233 – 242.

Schmidt, F. L. (2011). A theory of sex differences in technical aptitude and some supporting evidence. Perspectives on Psychological Science, 6, 560 – 573. 

Schmidt, F. L. (2012). Cognitive tests used in selection can have content validity as well as criterion validity: A broader research review and implications for practice. International Journal of Selection and Assessment, 20, 1 – 13.

Schmidt, F. L. (in press). A general theoretical integrative model of individual differences in interests, abilities, personality traits, and academic and occupational performance. Perspectives on Psychological Science.

Schmidt, F.L., & Hunter, J.E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262 – 274.

Schmidt, F. L., Le, H., & Oh, I-S. (2013). Are true scores and constructs the same? A critical examination of their substitutability and the implications for research results. International Journal of Selection and Assessment, 21, 341 – 354.

Schmidt, F. L., & Oh, I-S. (2013). Methods for second order meta-analysis and illustrative applications. Organizational Behavior and Human Decision Processes, 121, 204 – 218.

Schmidt, F. L., Oh, I-S., & Le, H. (2006). Increasing the accuracy of corrections for range restriction: Implications for selection procedure validities and other research results. Personnel Psychology, 59, 281 – 305.

Schmidt, F. L., Oh, I-S., & Shaffer, J. A. (2008). Increased accuracy for range restriction corrections: Implications for the role of personality and general mental ability in job and training performance. Personnel Psychology, 61, 827 – 868.

Oh, I-S., Postlethwaite, B. E., & Schmidt, F. L. (2013). Re-thinking the validity of unstructured interviews: Implications of recent developments in meta-analysis. In D. J. Svyantek, & K. Mahoney (Eds.), Received wisdom, kernels of truth, and boundary conditions in organizational studies. Charlotte, NC: Information Age Publishing. Pp 297 – 329.
Table 1. Selection methods for job performance

Selection procedures/predictors

Operational

validity (r)

Multiple R

Gain in
validity

% gain in validity

Standardized

Regression weights

GMA

Supple-
ment

GMA testsa

.65

 

 

 

 

 

Integrity testsb

.46

.78

.130

20%

.63

.43

Employment interviews (structured)c

.58

.76

.117

18%

.52

.43

Employment interviews (unstructured)d

.60

.75

.099

15%

.48

.41

Conscientiousnesse

.22

.70

.053

8%

.67

.27

Reference checksf

.26

.70

.050

8%

.65

.26

Biographical data measuresg

.35

.68

.036

6%

.91

-.34

Job experience h

.13

.67

.023

4%

.66

.17

Person-job fit measuresi

.18

.67

.020

3%

.64

.16

SJT (knowledge)j

.26

.67

.018

3%

.76

-.19

Assessment centersk

.37

.66

.014

2%

.78

-.19

Peer ratingsl

.49

.66

.013

2%

.55

.16

T & E point methodm

.11

.66

.009

1%

.65

.11

Years of educationn

.10

.66

.008

1%

.65

.10

Interestso

.10

.66

.008

1%

.65

.10

Emotional Intelligence (ability)p

.24

.65

.007

1%

.70

-.11

Emotional Intelligence (mixed)q

.24

.65

.005

1%

.63

.09

GPAr

.34

.65

.004

1%

.71

-.10

Person-organization fit measuress

.13

.65

.004

1%

.64

.07

Work sample testst

.33

.65

.003

0%

.69

-.07

SJT (behavioral tendency)u

.26

.65

.000

0%

.64

.03

Emotional Stabilityv

.12

.65

.000

0%

.64

.02

Job tryout procedurew

.44

.65

.000

0%

.63

.02

Behavioral consistency methodx

.45

.65

.000

0%

.64

.02

Job knowledgey

.48

.65

.000

0%

.65

-.01

Note. SJT = situational judgment tests; T & E = training and experience. Selection procedures whose operational validity is equal to and greater than .10 are listed in the order of gain in operational validity.

Unless otherwise noted, all operational validity estimates are corrected for measurement error in the criterion measure and indirect range restriction (IRR) on the predictor measure to estimate operational validity for applicant populations. The correlations between GMA and supplementary predictors (used to compute multiple Rs, gain in validity, and standardized regression weights) are corrected for IRR on GMA but not for measurement error in either measure; these correlations indicate observed correlations between predictors in applicant populations (unrestricted observed correlation).

a From Schmidt, Shaffer, and Oh (2008, Table 3). Individual meta-analytic estimates are reported in Table 1 on p. 838. The average of these estimates across eight meta-analytic estimates (.647) is presented in Table 3 on p. 843. We used this average in the current analyses.

b From Ones, Viswesvaran, and Schmidt (1993, Table 8). This operational validity is from predictive studies conducted on job applicants as in Schmidt and Hunter (1998); the same source was used in Schmidt and Hunter (1998), but the operational validity reported in this table was corrected for IRR. The unrestricted observed correlation with GMA is .046 (Ones, 1993, Table 3).

c, d From McDaniel, Whetzel, Schmidt, and Maurer (1994, Table 4). This operational validity is from primary studies where overall job performance was measured using research-purpose measures, and thus represents the most unbiased estimates available. The same source was used in Schmidt and Hunter (1998), but the operational validity estimates used in this table were corrected for IRR with the meta-analytic reliability estimates for the interview measure from Conway, Jako, and Goodman (1995). When the predictor reliability estimate from the McDaniel et al. (1994) was used in correcting for IRR, the operational validity for structured and unstructured interviews were .53  and .46 and gain in validity over GMA tests was .088 and .33, respectively. The unrestricted observed correlations with GMA are .305 and .402 for structured and unstructured interviews, respectively (Salgado & Moscoso, 2002, Tables 4 and 3, respectively).

e, v From Schmidt et al. (2008, Table 1). The unrestricted observed correlations with GMA are -.069 for Conscientiousness and .159 for Emotional Stability (Judge, Jackson, Shaw, Scott, & Bruce, 2007, Table 3). True score correlations alone were reported in Judge et al. (2007). We attenuated the true score correlations for predictor unreliability in both variables using the psychometric information provided by Timothy A. Judge.

f From Hunter and Hunter (1984, Table 9). The same source/validity was used in Hunter and Schmidt (1998). The correlation with GMA is assumed to be zero as in Hunter and Schmidt (1998).

g From Rothstein, Schmidt, Erwin, Owens, and Sparks (1990, Table 5). The same source/validity was used in Schmidt and Hunter (1998). The unrestricted observed correlation with GMA is .761 (Schmidt & Hunter, 1988, p. 283).

h From Sturman (2003, Table 1).  The unrestricted observed correlation with GMA is -.069 (Judge et al., 2007, Table 3).

i From Kristof-Brown, Zimmerman, and Johnson (2005, Table 1). The correlation with GMA (in fact, college GPA) is .023 (Cable & Judge, 1996); note that the value is based on one primary study.

j, u From McDaniel, Hartman, Whetzel, & Grubb III (2007, Table 3). The unrestricted observed correlations with GMA are .589 and .364 for SJT (knowledge) and SJT (behavioral tendency), respectively (McDaniel et al., 2007, Table 3).

k From Arthur, Day, McNelly, and Edens (2003, Table 3). The correlation with GMA is .710 (Collins, Schmidt, Sanchez-Ku, Thomas, McDaniel, & Le, 2003).

l From Hunter and Hunter (1984, Table 10). The same source/validity was used in Hunter and Schmidt (1998). Based on Schmidt and Hunter (1998), we used the unrestricted observed correlation with GMA is .594 (.50 without correcting for RR).

m, x From McDaniel, Schmidt, & Hunter (1988). The correlations with GMA are .000 and .682 for T &E point and behavioral consistency methods, respectively (Schmidt & Hunter, 1998); note that these are assumed values.

n From Hunter and Hunter (1984, Table 9). The same source/validity was used in Hunter and Schmidt (1998). The correlation with GMA is zero (Schmidt & Hunter, 1998); note that this is an assumed value.

o From Hunter and Hunter (1984, Table 9). The same source/validity was used in Hunter and Schmidt (1998). The correlation with GMA is zero (Schmidt & Hunter, 1998); note that this is an assumed value.

p, q From Van Rooy and Viswesvaran (2004, Table 1 for ability-based measures and Table 2 for mixed traits-based measures). The unrestricted observed correlations with GMA are .497 and .245 for ability-based measures and mixed traits-based measures, respectively (Van Rooy, Viswesvaran, & Pluta, 2005, Tables 3 and 4, respectively).

r From Roth, BeVier, Switzer, and Schippmann (1996). The operational validity estimates for GPA (combination of college, graduate, and PhD/MD GPAs) and college GPA are the same.  The unrestricted observed correlation with GMA is .619 (Robbins, Lauver, Le, Davis, Langley, & Carlstrom, 2004, Table 5).

s From Arthur, Bell, Villado, and Doverspike (2006, Table 1). The correlation with GMA (in fact, GPA) is .092 (Cable & Judge, 1996; 1997); we performed a small meta- analysis of these two articles in order to derive the estimate used in this study.

t From Roth, Bobko, and McFarland (2005, Table 1). The unrestricted observed correlation with GMA is .585 (Roth et al., 2005, Table 4).

w From Hunter and Hunter (1984, Table 9). The same source/validity was used in Schmidt and Hunter (1998). Based on Schmidt & Hunter (1998), we used the correlation of .663 (.38 without correcting for RR) for this analysis.

x From Hunter and Hunter (1984, Table 9). The same source/validity was used in Schmidt and Hunter (1998). The correlation with GMA is zero (Schmidt & Hunter, 1998); note that this is an assumed value.

y From Hunter and Hunter (1984, Table 11). The same source/validity was used in Schmidt and Hunter (1998). Based on Schmidt and Hunter (1998), we used the correlation of .747 (.48 without correcting for RR) for this analysis.


Table 2. Selection methods for training performance

Selection procedures/predictors

Operational

validity (r)

Multiple R

Gain in
validity

% gain in validity

Standardized

Regression weights

GMA

Supple-
ment

GMA testsa

.67

 

 

 

 

 

Integrity testsb

.43

.78

.109

16%

.65

.40

Biographical data measuresc

.30

.74

.073

11%

1.04

-.50

Conscientiousnessd

.25

.73

.061

9%

.69

.29

Employment interviewse

.48

.72

.051

8%

.57

.28

Reference checksf

.23

.71

.038

6%

.67

.23

Years of educationg

.20

.70

.029

4%

.67

.20

Interestsh

.18

.69

.024

4%

.67

.18

Peer ratingsi

.36

.67

.002

0%

.70

-.06

Emotional Stabilityj

.14

.67

.001

0%

.66

.03

Job experience (years)k

.01

.67

.000

0%

.67

.01

Note. Operational Validity estimates in parentheses are what is reported in Schmidt and Hunter (1998, Table 2). Selection procedures whose operational validity is equal to and greater than .10 are listed in the order of gain in operational validity.

Unless otherwise noted, all operational validity estimates are corrected for measurement error in the criterion measure and indirect range restriction (IRR) on the predictor measure to estimate operational validity for applicant populations. The correlations between GMA and supplementary predictors (used to compute multiple Rs, gain in validity, and standardized regression weights) are corrected for IRR on GMA but not for measurement error in either measure; these correlations indicate observed correlations between predictors in applicant populations (unrestricted observed correlation). Details on these correlations are reported in the footnote for Table 1; the same correlations were used in Tables 1 and 2.

a From Schmidt, Shaffer, and Oh (2008, Table 3). Individual meta-analytic estimates are reported in Table 2 on p. 840. The average of these estimates across eight meta-analytic estimates (.668) is presented in Table 3 on p. 843. We used this average in the current analyses.

b From Schmidt, Ones, and Viswesvaran (1994). The same source was used in Schmidt and Hunter (1998), but the operational validity reported in this table was corrected for IRR.

c From Hunter and Hunter (1984, Table 8). The same source/validity was used in Schmidt and Hunter (1998).

d, j From Schmidt et al. (2008, Table 2).

e From McDaniel et al. (1994, Table 5). It is noted that the validities for structured and unstructured interviews are very similar (the difference is .03), so we averaged them as in Schmidt and Hunter (1998, Table 2). The same source was used in Schmidt and Hunter (1998), but the operational validity reported in this table was corrected for IRR with the meta-analytic reliability estimates for the interview measure from Conway et al. (1995). When the predictor reliability estimate from the McDaniel et al. (1994) was used in correcting for IRR, the operational validity for structured and unstructured interviews were .43 and gain in validity over GMA tests was .032. The unrestricted observed correlation between employment interviews and GMA was assumed to be the average (.354) of the correlations of GMA with structured and unstructured interviews (Salgado & Moscoso, 2002).

 

f, g, i From Hunter and Hunter (1984, Table 8). The same source/validity was used in Schmidt and Hunter (1998).

h From Hunter and Hunter (1984, Table 6). The same source/validity was used in Schmidt and Hunter (1998).

k From Hunter and Hunter (1984, Table 6). The same source/validity was used in Schmidt and Hunter (1998).


References

Arthur, W. Jr., Bell, S. T., Villado, A. J., & Doverspike, D. (2006). The use of person-organization fit in employment decision making: An assessment of its criterion-related validity. Journal of Applied Psychology, 91, 786-801.

Arthur, W. Jr., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). Meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125-154.

Cable, D. M., & Judge, T. A. (1996). Person-organization fit, job choice decisions, and organizational entry. Organizational Behavior and Human Decision Processes, 67, 294-311.

Cable, D. M., & Judge, T. A. (1997). Interviewers' perceptions of person-organization fit and organizational selection decisions. Journal of Applied Psychology, 82, 546-561.

Collins, J. M., Schmidt, F. L., Sanchez-Ku, M., Thomas, L., McDaniel, M. A., & Le, H. (2003). Can basic individual differences shed light on the construct meaning of assessment center evaluations? International Journal of Selection and Assessment, 11, 17-29.

Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews.  Journal of Applied Psychology, 80, 565-579.

Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72-98.

Judge, T. A., Jackson, C., Shaw, J. C., Scott, B. A., & Rich, B. L. (2007). Self-efficacy and work-related performance: The integral role of individual differences. Journal of Applied Psychology, 92, 107-127.

Kristof-Brown, A. L., Zimmerman, R. D., & Johnson, E. C. (2005). Consequences of Individual’s Fit at Work: A Meta-Analysis of Person-Job, Person-Organization, Person-Group, and Person-Supervisor Fit. Personnel Psychology, 58, 281-342.

McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Mauer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599-616.

McDaniel, M.A., Hartman, N.S., Whetzel, D.L. & Grubb. W.L., III (2007). Situational judgment tests, response instructions and validity: A meta-analysis. Personnel Psychology, 60, 63-91.

McDaniel, M. A., Schmidt, F.L., & Hunter, J. E. (1988). A meta-analysis of the validity of methods for rating training and experience in personnel selection. Personnel Psychology, 41, 283-314.

Ones, D. S. (1993). The construct validity of integrity tests. Unpublished doctoral dissertation, University of Iowa, Iowa City.

Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology Monograph, 78, 679-703.

Robbins, S.B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.

Roth, P. L., BeVier, C. A., Switzer, F. S., & Schippmann, J. (1996). Meta-analyzing the relationship between grades and job performance. Journal of Applied Psychology, 81(5), 548-556.

Roth, P. L., Bobko, P., & McFarland, L. A. (2005). A meta-analysis of work sample test validity: Updating and integrating some classic literature. Journal of Personnel Psychology, 58, 1009-1037.

Rothstein, H. R., Schmidt, F. L., Erwin, F. W., Owens, W. A., & Sparks, C. P. (1990). Biographical data in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75, 175-184.

Salgado, J. F., & Moscoso, S. (2002). Comprehensive meta-analysis of the construct validity of the selection interview. European Journal of Work and Organizational Psychology, 11, 299-324.

Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.

Schmidt, F. L., Ones, D. S., & Viswesvaran, C. (1994). The personality characteristic of integrity predicts job training success. Presented at the 6th Annual Convention of the American Psychological Society, Washington, DC.

Schmidt, F. L., Shaffer, J. A., & Oh, I.-S. (2008). Increased accuracy of range restriction corrections: Implications for the role of personality and general mental ability in job and training performance. Personnel Psychology, 61, 827-868.

Sturman, M. C. (2003). Searching for the inverted U-shaped relationship between time and performance: Meta-analyses of the experience/performance, tenure/performance, and age/performance relationships. Journal of Management, 29, 609-640.

Van Rooy, D. L., & Viswesvaran, C. (2004). Emotional intelligence: A meta-analytic investigation of predictive validity and nomological net. Journal of Vocational Behavior, 65, 71–95.

Van Rooy, D. L., Viswesvaran, C., & Pluta, P. (2005). An evaluation of construct validity: What is this thing called emotional intelligence? Human Performance, 18, 445-462.