OBJECTIVE: To assess the agreement between prospectively and retrospectively determined variables comprising the Pneumonia Severity Index (PSI), assignment to PSI risk class, and designation as low risk, based on these two methods of data collection. STUDY DESIGN AND SETTING: We analyzed data from a randomized trial of patients with community-acquired pneumonia managed in 32 hospital emergency departments (EDs). For all enrolled patients, the 20 PSI variables were collected prospectively by ED providers and retrospectively by medical record abstractors. We examined the agreement for each of the 20 PSI variables, assignment to the five PSI risk classes, and classification of patients as low (classes I-III) vs. high (classes IV and V) risk. Agreement was measured using total percent agreement and the kappa statistic. RESULTS: Among the 3,220 enrolled patients, percent agreement was >90% for 18 of the 20 variables comprising the PSI, with most unweighted kappa's being >0.6. Agreement was substantial for assignment to PSI risk class (percent agreement: 92.7%; weighted kappa: 0.79) and for classification as low vs. high risk (percent agreement: 88.5%; unweighted kappa: 0.74). CONCLUSION: There was substantial agreement between retrospective and prospective assignment to PSI risk class, classification as low vs. high risk, and the determination of most individual variables that constitute the PSI.