Background. Social theories suggest that neighborhood quality affects health. Observer ratings of neighborhoods should be subjected to psychometric tests. Methods. African American Health (AAH) study subjects were selected from two diverse St. Louis metropolitan catchment areas. Interviewers rated streets and block faces for 816 households. Items and a summary scale were compared across catchment areas and to the resident respondents' global neighborhood assessments. Results. Individual items and the scale were strongly associated with both the catchment area and respondent assessments. Ratings based on both block faces did not improve those based on a single block face. Substantial interviewer effects were observed despite strong discriminant and concurrent validity. Conclusion. Observer ratings show promise in understanding the effect of neighborhood on health outcomes. The AAH Neighborhood Assessment Scale and other rating systems should be tested further in diverse settings.