[Updated: Sat, Sep 07, 2024 - 08:48:35 ]

1. Overview

The itemanalysis package in R is designed to facilitate classical item analysis for both multiple-choice (dichotomous) and polytomous (rating scale) test items. Classical Test Theory (CTT) provides essential item statistics such as item difficulty, discrimination, and distractor analysis, which help in evaluating the quality of test items.

This tutorial will guide you through running item analysis for both dichotomous and polytomous datasets using the itemanalysis package.

(Note that there are much better tools available for free to run classical item analysis. This package was originally developed in 2013 to supplement the content I was teaching in a measurement course at the University of Miami. Therefore, it is customized and aligned with the content I taught.)

2. Installation and Setup

Before starting, you need to install and load the package. Since the package may not be available on CRAN, you can download it from the source or GitHub (if applicable) or use the pre-installed package.

# Install the package (if not already installed)
# install.packages("itemanalysis")

# Load the package
require(itemanalysis)

3. Running Item Analysis on Dichotomously Scored Items

We will use a multiple-choice dataset where each item has one correct answer, and the responses are nominal (e.g., A, B, C, D). The dataset dichotomous is included in the package. It contains 6,000 rows (examinees) and 56 columns (items), with each cell representing the nominal response to a particular item.

# Load the dataset
data(dichotomous)
head(dichotomous)
  item1 item2 item3 item4 item5 item6 item7 item8 item9 item10 item11 item12
1     A     A     D     D     C     B     C     D     D      D      C      A
2     A     B     C     C     C     B     A     D     A      D      C      A
3     A     C     D     B     D     D     D     D     A      D      C      A
4     A     B     D     B     D     D     C     D     A      C      C      B
5     A     D     C     B     C     B     C     D     B      D      C      A
6     D     C     B     C     A     B     C     D     A      D      A      B
  item13 item14 item15 item16 item17 item18 item19 item20 item21 item22 item23
1      D      C      A      B      D      B      A      B      B      A      D
2      D      C      A      B      A      D      A      C      B      A      C
3      D      D      A      B      A      A      A      C      B      C      C
4      D      D      A      A      D      B      D      C      D      A      C
5      D      C      A      A      A      B      A      C      B      C      A
6      D      D      B      C      B   <NA>      D      B      B      A      C
  item24 item25 item26 item27 item28 item29 item30 item31 item32 item33 item34
1      B      C      B      A      C      A      C      A      B      B      A
2      B      D      B      C      A      D      B      C      A      B      A
3      B      B      B      B      D      D      A      A      B      B      C
4      B      C      B      A      D      A      B      D      A      B      C
5      B      C      B      B      C      A      C      D      B      D      B
6      A      C      A      C   <NA>   <NA>      C      B      A      C      B
  item35 item36 item37 item38 item39 item40 item41 item42 item43 item44 item45
1      B      D      C      A      D      C      D      A      B      C      C
2      D      B      D      B      C      C      A      D      B      C      A
3      C      D      B      A      C      C      D      D      B      D      A
4      B      B      A      C      A      C      A      D      C      A      B
5      B      D      B      A      A      C      D      A      C      B      C
6      D      B      C      B      A      C      A      B      C      B      A
  item46 item47 item48 item49 item50 item51 item52 item53 item54 item55 item56
1      C      D      B      D      D      D      D      C      A      D      D
2      B      A      D      A      D      D      C      C      B      A      C
3      D      B      D      C      B      D      A      C      C      A      D
4      D      D      C      B      A      D      C      B      A      D      A
5      B      B      D      B      B      D      C      D      B      D      D
6      D      A      C      B      A      B      A   <NA>      B      C      C

The answer key is a vector that specifies the correct answer for each item.

# Define the correct answers for each item

key <- c("A","D","C","B","C","B","C","D","A","D","C","A","D","C","A",
         "B","D","B","A","C","A","A","C","B","C","B","D","A","A","A",
         "C","B","B","A","B","D","D","A","D","C","D","A","B","B","C",
         "D","B","C","C","B","D","A","C","B","A","D")

Use the itemanalysis1 function to run the analysis. You can customize parameters such as the number of score groups (ngroup) and whether to apply correction for point-biserial correlation.

item.analysis <- itemanalysis1(
  data       = dichotomous,             # a data frame object with N rows and m columns
  key        = key,                     # a vector of correct answers with length m
  options    = c("A", "B", "C", "D"),   # a vector of available response options
  ngroup     = 10,                      # number of score groups to be use for plotting the item trace lines
  correction = F,                       # if it is TRUE, then an adjustment is made for point-biserial correlation
                                        # recommended for tests with small number of items
  span.par   = 0.3,                     #   a smoothing parameter to pass to ggplots when creating empirical item trace lines,
  verbose    = T                        # if it is TRUE, text output is printed in console
)
************************************************************************ 
itemanalysis: An R package for Classical Test Theory Item Analysis 
 
Cengiz Zopluoglu 
 
University of Oregon 
College of Education 
 
cen.zop@gmail.com 
 
Please report any programming bug or problem you experience to improve the code. 
************************************************************************* 
Processing Date:  Sat Sep  7 08:48:41 2024 

                                   ITEM STATISTICS 
 
       Item Difficulty Item Threshold Point-Biserial Biserial
item1            0.709         -0.551          0.349    0.462
item2            0.530         -0.075          0.304    0.381
item3            0.812         -0.885          0.440    0.638
item4            0.574         -0.185          0.281    0.354
item5            0.653         -0.393          0.334    0.431
item6            0.733         -0.621          0.384    0.517
item7            0.460          0.101          0.361    0.453
item8            0.856         -1.060          0.442    0.684
item9            0.544         -0.110          0.361    0.453
item10           0.808         -0.872          0.490    0.707
item11           0.827         -0.941          0.477    0.705
item12           0.553         -0.132          0.372    0.468
item13           0.772         -0.745          0.505    0.701
item14           0.473          0.067          0.345    0.433
item15           0.640         -0.360          0.480    0.616
item16           0.447          0.133          0.343    0.431
item17           0.434          0.166          0.379    0.477
item18           0.488          0.030          0.341    0.427
item19           0.668         -0.434          0.559    0.725
item20           0.502         -0.005          0.377    0.473
item21           0.572         -0.181          0.445    0.561
item22           0.437          0.159          0.420    0.529
item23           0.370          0.331          0.353    0.451
item24           0.569         -0.175          0.543    0.684
item25           0.503         -0.009          0.461    0.578
item26           0.741         -0.647          0.467    0.632
item27           0.163          0.982          0.209    0.313
item28           0.394          0.270          0.298    0.378
item29           0.446          0.136          0.351    0.441
item30           0.422          0.196          0.347    0.438
item31           0.364          0.348          0.258    0.331
item32           0.324          0.456          0.225    0.293
item33           0.665         -0.428          0.431    0.559
item34           0.489          0.027          0.445    0.558
item35           0.647         -0.379          0.532    0.684
item36           0.604         -0.264          0.543    0.689
item37           0.527         -0.068          0.493    0.619
item38           0.406          0.237          0.318    0.402
item39           0.334          0.428          0.291    0.377
item40           0.602         -0.257          0.418    0.530
item41           0.623         -0.313          0.490    0.626
item42           0.509         -0.024          0.381    0.477
item43           0.627         -0.323          0.420    0.536
item44           0.370          0.332          0.374    0.478
item45           0.527         -0.067          0.499    0.626
item46           0.537         -0.092          0.538    0.676
item47           0.458          0.105          0.426    0.534
item48           0.473          0.068          0.448    0.561
item49           0.489          0.028          0.380    0.476
item50           0.622         -0.310          0.583    0.744
item51           0.642         -0.365          0.612    0.785
item52           0.495          0.013          0.483    0.605
item53           0.494          0.016          0.452    0.567
item54           0.380          0.306          0.283    0.360
item55           0.415          0.216          0.395    0.500
item56           0.629         -0.329          0.560    0.716
 
 
 
                  DISTRACTOR SELECTION PROPORTIONS 
 
           A     B     C     D Missing
item1  0.709 0.077 0.082 0.126   0.006
item2  0.143 0.148 0.171 0.530   0.008
item3  0.046 0.060 0.812 0.074   0.009
item4  0.140 0.574 0.138 0.142   0.007
item5  0.076 0.115 0.653 0.147   0.009
item6  0.073 0.733 0.134 0.054   0.006
item7  0.190 0.181 0.460 0.162   0.008
item8  0.029 0.062 0.044 0.856   0.009
item9  0.544 0.163 0.092 0.186   0.015
item10 0.036 0.077 0.069 0.808   0.010
item11 0.058 0.074 0.827 0.034   0.008
item12 0.553 0.296 0.094 0.048   0.010
item13 0.059 0.077 0.080 0.772   0.013
item14 0.110 0.193 0.473 0.211   0.012
item15 0.640 0.095 0.123 0.128   0.013
item16 0.256 0.447 0.180 0.104   0.013
item17 0.389 0.092 0.067 0.434   0.018
item18 0.110 0.488 0.137 0.245   0.021
item19 0.668 0.092 0.080 0.139   0.021
item20 0.128 0.198 0.502 0.149   0.023
item21 0.572 0.230 0.119 0.060   0.018
item22 0.437 0.180 0.252 0.109   0.023
item23 0.177 0.274 0.370 0.154   0.025
item24 0.134 0.569 0.096 0.174   0.027
item25 0.124 0.221 0.503 0.126   0.026
item26 0.075 0.741 0.080 0.077   0.027
item27 0.313 0.247 0.248 0.163   0.028
item28 0.394 0.175 0.224 0.174   0.033
item29 0.446 0.126 0.313 0.085   0.030
item30 0.422 0.188 0.261 0.098   0.031
item31 0.191 0.203 0.364 0.212   0.031
item32 0.164 0.324 0.234 0.244   0.034
item33 0.136 0.665 0.105 0.063   0.030
item34 0.489 0.252 0.143 0.085   0.030
item35 0.136 0.647 0.084 0.101   0.032
item36 0.097 0.122 0.144 0.604   0.033
item37 0.138 0.179 0.121 0.527   0.035
item38 0.406 0.136 0.273 0.150   0.034
item39 0.181 0.263 0.186 0.334   0.036
item40 0.100 0.198 0.602 0.066   0.034
item41 0.150 0.137 0.056 0.623   0.034
item42 0.509 0.128 0.147 0.180   0.036
item43 0.082 0.627 0.171 0.083   0.037
item44 0.304 0.370 0.149 0.137   0.040
item45 0.173 0.173 0.527 0.087   0.041
item46 0.090 0.152 0.179 0.537   0.041
item47 0.134 0.458 0.144 0.223   0.041
item48 0.093 0.158 0.473 0.232   0.044
item49 0.170 0.161 0.489 0.132   0.048
item50 0.096 0.622 0.121 0.117   0.044
item51 0.125 0.112 0.076 0.642   0.044
item52 0.495 0.248 0.131 0.078   0.047
item53 0.135 0.181 0.494 0.138   0.052
item54 0.189 0.380 0.245 0.133   0.054
item55 0.415 0.145 0.128 0.257   0.055
item56 0.093 0.112 0.108 0.629   0.058
 
 
 
                         DISTRACTOR Point-Biserial 
 
            A      B      C      D
item1   0.349 -0.197 -0.255 -0.086
item2  -0.023 -0.139 -0.229  0.304
item3  -0.222 -0.233  0.440 -0.230
item4  -0.105  0.281 -0.093 -0.184
item5  -0.218 -0.147  0.334 -0.125
item6  -0.239  0.384 -0.148 -0.219
item7  -0.238 -0.086  0.361 -0.121
item8  -0.201 -0.258 -0.236  0.442
item9   0.361 -0.191 -0.223 -0.078
item10 -0.235 -0.283 -0.244  0.490
item11 -0.272 -0.278  0.477 -0.182
item12  0.372 -0.113 -0.260 -0.209
item13 -0.228 -0.257 -0.267  0.505
item14 -0.199 -0.136  0.345 -0.097
item15  0.480 -0.229 -0.231 -0.206
item16 -0.060  0.343 -0.196 -0.168
item17 -0.094 -0.249 -0.199  0.379
item18 -0.177  0.341 -0.225 -0.034
item19  0.559 -0.291 -0.254 -0.250
item20 -0.181 -0.140  0.377 -0.134
item21  0.445 -0.183 -0.228 -0.190
item22  0.420 -0.176 -0.148 -0.165
item23 -0.162 -0.136  0.353 -0.054
item24 -0.235  0.543 -0.260 -0.221
item25 -0.197 -0.250  0.461 -0.097
item26 -0.187  0.467 -0.284 -0.178
item27 -0.131 -0.051  0.081  0.209
item28  0.298 -0.106 -0.080 -0.108
item29  0.351 -0.252 -0.037 -0.149
item30  0.347 -0.166 -0.081 -0.135
item31 -0.087 -0.117  0.258 -0.030
item32 -0.145  0.225 -0.174  0.132
item33 -0.206  0.431 -0.156 -0.183
item34  0.445 -0.115 -0.245 -0.167
item35 -0.236  0.532 -0.232 -0.219
item36 -0.177 -0.260 -0.244  0.543
item37 -0.303 -0.103 -0.176  0.493
item38  0.318 -0.230 -0.025 -0.062
item39 -0.084  0.062 -0.222  0.291
item40 -0.190 -0.109  0.418 -0.237
item41 -0.253 -0.146 -0.224  0.490
item42  0.381 -0.250 -0.220  0.048
item43 -0.153  0.420 -0.144 -0.212
item44 -0.058  0.374 -0.152 -0.145
item45 -0.180 -0.235  0.499 -0.146
item46 -0.211 -0.288 -0.142  0.538
item47 -0.147  0.426 -0.225 -0.077
item48 -0.217 -0.206  0.448 -0.076
item49 -0.177 -0.112  0.380 -0.085
item50 -0.246  0.583 -0.256 -0.239
item51 -0.289 -0.277 -0.219  0.612
item52  0.483 -0.112 -0.259 -0.191
item53 -0.125 -0.210  0.452 -0.129
item54 -0.064  0.283 -0.047 -0.102
item55  0.395 -0.140 -0.136 -0.098
item56 -0.232 -0.245 -0.216  0.560
 
 
 
                               DISTRACTOR Biserial 
 
            A      B      C      D
item1   0.462 -0.365 -0.462 -0.138
item2  -0.036 -0.214 -0.340  0.381
item3  -0.484 -0.466  0.638 -0.428
item4  -0.163  0.354 -0.145 -0.286
item5  -0.404 -0.242  0.431 -0.192
item6  -0.448  0.517 -0.234 -0.451
item7  -0.344 -0.125  0.453 -0.182
item8  -0.511 -0.509 -0.518  0.684
item9   0.453 -0.286 -0.391 -0.113
item10 -0.553 -0.522 -0.467  0.707
item11 -0.548 -0.520  0.705 -0.437
item12  0.468 -0.149 -0.453 -0.447
item13 -0.457 -0.475 -0.488  0.701
item14 -0.331 -0.196  0.433 -0.137
item15  0.616 -0.397 -0.373 -0.328
item16 -0.082  0.431 -0.287 -0.284
item17 -0.119 -0.436 -0.383  0.477
item18 -0.294  0.427 -0.353 -0.047
item19  0.725 -0.509 -0.464 -0.390
item20 -0.288 -0.201  0.473 -0.205
item21  0.561 -0.254 -0.370 -0.378
item22  0.529 -0.257 -0.202 -0.275
item23 -0.238 -0.182  0.451 -0.082
item24 -0.371  0.684 -0.450 -0.326
item25 -0.317 -0.350  0.578 -0.155
item26 -0.348  0.632 -0.518 -0.329
item27 -0.171 -0.069  0.110  0.313
item28  0.378 -0.157 -0.111 -0.159
item29  0.441 -0.404 -0.048 -0.267
item30  0.438 -0.241 -0.110 -0.233
item31 -0.126 -0.166  0.331 -0.042
item32 -0.217  0.293 -0.241  0.181
item33 -0.323  0.559 -0.263 -0.360
item34  0.558 -0.156 -0.381 -0.298
item35 -0.371  0.684 -0.418 -0.373
item36 -0.306 -0.422 -0.378  0.689
item37 -0.475 -0.150 -0.286  0.619
item38  0.402 -0.362 -0.033 -0.096
item39 -0.123  0.083 -0.323  0.377
item40 -0.325 -0.157  0.530 -0.458
item41 -0.388 -0.229 -0.456  0.626
item42  0.477 -0.400 -0.339  0.070
item43 -0.278  0.536 -0.213 -0.382
item44 -0.076  0.478 -0.233 -0.228
item45 -0.266 -0.348  0.626 -0.260
item46 -0.372 -0.439 -0.208  0.676
item47 -0.231  0.534 -0.347 -0.108
item48 -0.380 -0.312  0.561 -0.105
item49 -0.263 -0.168  0.476 -0.134
item50 -0.427  0.744 -0.415 -0.391
item51 -0.464 -0.459 -0.405  0.785
item52  0.605 -0.153 -0.410 -0.350
item53 -0.196 -0.307  0.567 -0.202
item54 -0.092  0.360 -0.064 -0.161
item55  0.500 -0.217 -0.217 -0.132
item56 -0.406 -0.405 -0.361  0.716

The function prints the output in the console by default. If you want to turn this feature off, you can add verbose=F argument. In either case, you can always access to the outputs from the function as separate data objects using the following code.

item.analysis$item.stat  # returns a matrix of basic item statistics
      
item.analysis$dist.sel   # returns a matrix of distractor selection proportion statistics
      
item.analysis$dist.disc  # returns a matrix of point-biserial statistics for distractors

The function also creates and saves the item trace lines for each item in the plot output, a list object, that can be accessed as follows. A default value of 0.3 is used for the span.par argument in the function and this is being passed to the ggplot pacage for smoothing the lines. If you’d like to change it, you can add span.par= argument with a custom value to the function.

# Plot item trace lines for the first item
item.analysis$plots[[1]]

# Plot item trace lines for the second item
item.analysis$plots[[2]]

4. Running Item Analysis on Polytomously Scored Items

Next, we’ll analyze a dataset where the responses are polytomous, meaning that they use multiple response categories such as those found in rating scales (e.g., Likert scales). The example below uses the TIMSS 2011 USA dataset (timss2011_usa), which includes items measuring attitudes towards mathematics.

The items are as follows:

  • Q14A, I enjoy learning mathematics

  • Q14B, I wish have not to study Math

  • Q14C, Mathematics is boring

  • Q14D, I learn interesting things in mathematics class

  • Q14E, I like mathematics

  • Q14F, I think it’s important to do well in mathematics

Before analysis, note that Q14B and Q14C must be reverse coded to align with the other items. These two items reflect negative attitudes towards mathematics, whereas the other four reflect positive attitudes.

This dataset, which is included in the package, contains responses to six items. Each item is scored on a scale from 0 to 3, where 0 indicates “strongly disagree” and 3 indicates “strongly agree.”

# Load the dataset
data(timss2011_usa)
head(timss2011_usa)
  Q14A Q14B Q14C Q14D Q14E Q14F
1    0    3    3    1    0    3
2    2    2    3    3    2    3
3    2    1    2    2    1    3
4    3    1    1    2    3    3
5    3    0    2    3    3    3
6    0    3    3    1    0    3
#  Two items needs to be reverse coded for alignment 

timss2011_usa$Q14B <- recode(var = timss2011_usa$Q14B,
                             recodes = "c(0)=3;c(1)=2;c(2)=1;c(3)=0")

timss2011_usa$Q14C <- recode(var = timss2011_usa$Q14C,
                             recodes = "c(0)=3;c(1)=2;c(2)=1;c(3)=0")

Use the itemanalysis2 function to run the analysis for polytomous items. You can specify response categories using the options argument.

# Run item analysis for polytomous items
item.analysis.poly <- itemanalysis2(
  data       = timss2011_usa, # a data frame object with N rows and m columns
  options    = c(0, 1, 2, 3), # a vector of numerical code of the response categories available in the data
  ngroup     = 10,            # number of score groups to be use for plotting the item trace lines
  correction = T,             # if it is TRUE, then an adjustment is made for point-biserial correlation
                              # recommended for scale with small number items
  span.par   = .3,            # a smoothing parameter to pass to ggplots when creating empirical item trace lines
  verbose    = T              # if it is TRUE, text output is printed in console.
)
************************************************************************ 
itemanalysis: An R package for Classical Test Theory Item Analysis 
 
Cengiz Zopluoglu 
 
University of Oregon 
College of Education 
 
cen.zop@gmail.com 
 
Please report any programming bug or problem you experience to improve the code. 
************************************************************************* 
Processing Date:  Sat Sep  7 08:48:46 2024 
                                   ITEM STATISTICS 
 
       Mean Score Item Difficulty Point-Biserial Polyserial
Item 1      1.763           0.588          0.811      0.877
Item 2      1.599           0.533          0.654      0.704
Item 3      1.379           0.460          0.694      0.745
Item 4      1.864           0.621          0.663      0.717
Item 5      1.702           0.567          0.817      0.884
Item 6      2.709           0.903          0.346      0.480
 
    * Item difficulty is the ratio of mean score to possible maximum score 
      and assumes the minimum score is 0 
 
 
           RESPONSE CATEGORY SELECTION PROPORTIONS 
 
           0     1     2     3
Item 1 0.161 0.189 0.375 0.274
Item 2 0.200 0.263 0.275 0.262
Item 3 0.240 0.330 0.242 0.188
Item 4 0.114 0.206 0.384 0.297
Item 5 0.189 0.191 0.351 0.270
Item 6 0.021 0.026 0.176 0.777
 
 
 
      RESPONSE CATEGORY Point-Biserial Correlation 
 
            0      1      2     3
Item 1 -0.602 -0.281  0.109 0.625
Item 2 -0.545 -0.136  0.152 0.478
Item 3 -0.627 -0.023  0.232 0.458
Item 4 -0.469 -0.278  0.059 0.509
Item 5 -0.628 -0.247  0.132 0.629
Item 6 -0.195 -0.135 -0.230 0.329
 
 
            RESPONSE CATEGORY Biserial Correlation 
 
            0      1      2     3
Item 1 -0.905 -0.407  0.139 0.837
Item 2 -0.779 -0.183  0.203 0.645
Item 3 -0.862 -0.030  0.318 0.664
Item 4 -0.773 -0.395  0.075 0.672
Item 5 -0.909 -0.356  0.171 0.845
Item 6 -0.552 -0.357 -0.338 0.459

The function similarly prints the output in the console by default. If you want to turn this feature off, you can add verbose=F argument. In either case, you can always access to the outputs from the function as separate data objects using the following code.

item.analysis.poly$item.stat  # returns a matrix of basic item statistics
      
item.analysis.poly$dist.sel   # returns a matrix of distractor selection proportion statistics
      
item.analysis.poly$dist.disc  # returns a matrix of point-biserial statistics for distractors

The function similarly creates and saves the item trace lines for each item that can be accessed as follows. A default value of 0.3 is used for the span.par argument in the function and this is being passed to the ggplot pacage for smoothing the lines. If you’d like to change it, you can add span.par= argument with a custom value to the function.

# Plot item trace lines for the first item
item.analysis.poly$plots[[1]]