Skip Navigation

Link to  the National Institutes of Health  
The Science of Drug Abuse and Addiction from the National Institute on Drug Abuse Archives of the National Institute on Drug Abuse web site
Go to the Home page
   
Assessing the Impact of Childhood Interventions on Subsequent Drug Use Home
Assessing the Impact of Childhood Interventions
on Subsequent Drug Use
skip navigation About the Conference
Agenda
Commissioned Papers
Barbara J. Burns, Ph.D.
Scott N. Compton, Ph.D.
Helen L. Egger, M.D.
Elizabeth M.Z. Farmer, Ph.D.
E. Jane Costello
Tonya D. Armstrong
Alaattin Erkanli
Paul E. Greenbaum
Chi-Ming Kam
Linda M. Collins
Selected Bibliography
Program Contacts

Annotated Bibliography on Research Methods

Kam & Collins

Links to other parts of this paper:


III. STUDY DESIGN

A. General References in Research Design and Data Analysis

  1. Ager, J.W. Discussion: Statistical analysis in treatment and prevention program evaluation. In: Kilbey, M.M., & Asghar, K., eds. Methodological Issues in Epidemiological, Prevention, and Treatment Research on Drug-Exposed Women and Their Children. National Institute on Drug Abuse Research Monograph 117. DHHS Pub. No. (ADM)92-1881. Washington, DC: Supt. of Docs., U.S. Govt. Print. Off., 1992. pp. 31-40.

The chapter focuses on problems of statistical analysis in the context of evaluations of substance abuse treatment and prevention programs. The statistical and design issues discussed includes the following: (1) types of designs and their associated statistical analyses; (2) covariance and other adjustment techniques in analyses of quasi-experimental design data; (3) modeling change; and (4) meta-analysis.

  1. Anderson, V.L., & McLean, R.A. Design of Experiments: A Realistic Approach. New York: Marcel Dekker, Inc., 1974.

This book on design of experiments is arranged so that the reader may go from the simple to the complex designs and grasp the appropriate analyses from the resulting data. Chapters 1 to 3 cover basic concepts in experimental designs. Chapters 4 through 7 cover various experimental designs and illustrate the usage of restriction error concept. Chapter 8 expresses a view on Latin square type designs, and chapters 9 and 10 provide descriptions on the completely randomized factorials. Chapters 11 and 12 deal with three-level factorial experiments, mixed factorial experiments, and incomplete block designs. The last chapter of the book describes response surface exploration.

  1. Bryant, K.J., Windle, M., & West, S.G., eds. The Science of Prevention: Methodological Advances from Alcohol and Substance Abuse Research. Washington, DC: American Psychological Association, 1997.

The volume describes latest developments of methodological methods in the field of substance abuse research. Most of the contributors to the book are active researchers in the field of substance abuse prevention who are also methodological experts. It aims at promoting critical thinking among new and established investigators about how to design research and analyze research findings. Although the substantive focus of many chapters is on applications to the prevention of alcohol and substance abuse, nearly all of the methodological principles and statistical models are general and have potential application to the full range of areas in which prevention research takes place.

  1. Collins, L.M., & Millsap, R. Innovative methods for prevention research. Special issue of Multivariate Behavioral Research. Mahwah, NJ: Erlbaum, 1998.

The special issue includes six examples of innovative methodlogical procedure useful in prevention research. Hedeker and Mermelstein present a model for multilevel logistic regression with ordinal outcomes. Reboussin et al. discuss a method for including continuous predictors in latent transition models. Boker and Graham provides a conceptual introduction to dynamical systems analysis. Sayer and Willet describe cross-domain analysis in latent growth curve analysis. Schafer and Olsen present the technique of multiple imputation and its use to solve missing data problems. Bacik et al. discuss a type of missing data that is common in prevention research: the participant who is absent for one data collection occasion and then returns to the study. They explore the researcher's options in performing survival analysis on this kind of data.

  1. Coursey, R.D., ed. Program Evaluation for Mental Health: Methods, Strategies, Participants. New York: Grune & Stratton, Inc., 1977.

The book focuses on setting up and running evaluation programs for real-life mental health delivery systems. It summarizes literature and provides both broad conceptualizations about the structure of evaluation activity as well as practical knowledge. The book emphasizes the human dimensions of evaluation, such as staff's need for economic survival and esteem. It also emphasize the role of program evaluation in staff education and development. The author also take the perspective that program evaluation is one component of the overall process of program development and program planning. The book is divided into four sections: (1) methods and techniques of evaluation, (2) strategies for implementing those methods, (3) the participants in program evaluation, and (4) resources available to mental health workers.

  1. Daniel, W.W. Biostatistics: A Foundation for Analysis in the Health Sciences, 7th ed. NY: John Wiley & Sons, 1999.

The text is written for health professionals in need of a reference book on statistical methodology. It covers basic concepts in statistics such as probability, sampling distribution, estimation and hypothesis testing. It describes common statistical techniques such as analysis of variance, simple and multiple regression, logistic regression, chi-square test and nonparametric and distribution-free statistics.

  1. Dowdy, S., & Wearden, S. Statistics for Research. New York: Wiley, 1983.

This is an introductory textbook to statistics and it is intended for students who have no prior background in statistical methods. The author try to provide both an understanding of the concepts of statistical inference as well as the methodology for the most commonly used analytical procedures. The book covers basic concepts in statistics, such as probability distributions, sampling distribution of averages and paired variables. It then describes commonly used statistical techniques, such as ANOVA, ANCOVA, and multiple regression.

  1. Flay, B.R., & Petraitis, J. Methodological issues in drug use prevention research: Theoretical foundations. In: Leukefeld, C.G., & Bukoski, W.J., eds. Drug Abuse Prevention Intervention Research: Methodological Issues. NIDA Research Monograph 107. DHHS Pub. No. (ADM) 91-1761. Washington, DC: Supt. of Docs., U.S. Govt. Print. Off., 1991. pp. 81-109.

The chapter discusses the theoretical foundation of drug use prevention program development and research. The authors reviewed theories of drug use onset and behavior change and then focus on the functions and roles of theory and their methodological applications. Issues like implementation quality, external validity, construct validity, and special method—theory relationships are discussed in later parts of the chapter.

  1. Jason, L., Thompson, D., & Rose, T. Methodological issues in prevention. In: Edelstein, B., & Michelson, L., eds. Handbook of Prevention. New York: Plenum Press, 1986, pp. 1-19.

The chapter reviews methodological issues that need to be considered in designing and implementing preventive-oriented interventions. The authors first review theoretical notions underlying preventive programs. Then they discuss specific methodological issues like goal selection, assessment and screening, experimental and quasi-experimental designs, generalization and maintenance, cost-benefit analyses, meta-analyses, social validation, and monitoring the integrity of preventive interventions. The final section summarizes some of the obstacles to designing well-planned preventive interventions, and speculates on the types of new directions.

  1. Judd, C.G., & Kenny, D.A. Estimating the Effects of Social Intervention. Cambridge: Cambridge University Press, 1981.

The authors present a discussion of the various research designs used to evaluate social interventions. Designs described in the book include randomized experiments, regression discontinuity design, nonequivalent control group design, interrupted time-series design, post-only correlational design. Judd and Kenny describe each design and the usual statistical analysis procedures to analyze the design. They then discuss possible complications in the analysis and suggest practical solutions to the complications.

  1. Maxwell, S.H., & Delaney, H.D. Designing Experiments and Analyzing Data: A Model Comparison Perspective. Belmont, CA: Wadsworth, 1990.

The book is written to serve as either a textbook or a reference book on the topic of designing experiments and analyzing experimental data. The authors proposed a model comparison approach, which allows the use of a few basic formulas that can be applied to every experimental design. Such an approach also allows for further extension to more complex data-analytic methodologies such as structural equation modeling. Part I of the book explains the logic of experimental design and the role of randomization in behavioral research. Part II deals with model comparison for between-subject designs, starting with a discussion of the general linear model. Part III discusses model comparisons for designs involving within-subject factors. Part IV covers alternative analysis strategies such as robust ANOVA and ANCOVA, repeated measures designs.

  1. Montgomery, D.C. Design and Analysis of Experiments, 3rd ed. New York: John Wiley & Sons, 1991.

This is an introductory textbook dealing with the design and analysis of experiments. The book is intended for readers who have some background in elementary statistics and some familiarity with matrix algebra is required in portions of chapters 15 and 16. The third edition is a major revision of the book. While maintaining a balance between design and analysis topics, new materials and examples are added. The reorganization of the material on factorial and fractional factorial designs in chapters 9, 10, and 11 provide more in-depth treatment of these topics. Chapter 12 covers the Taguchi approach to parameter design along with a critique of the method. Chapter 16 describes response surface design and introduce the reader to the mixture problem.

  1. Rose, J.S., Chassin, L., Presson, C.C., & Sherman, S.J. Multivariate Applications in Substance Use Research: New Methods for New Questions. Mahwah, NJ: Lawrence Erlbaum Associates, 2000.

This edited volume introduces the latest advances in quantitative methods and illustrates ways to apply these methods to important questions in substance use research. Reflecting current research trends, the book examines the use of longitudinal techniques to measure processes of change over time. Researchers faced with the task of studying the causes, course, treatment, and prevention of substance use and abuse will find this volume helpful for applying these techniques to make optimal use of their data.

Back to Top

B. Different Types of Study Designs

1. Case Studies

  1. Bromley, D.B. The Case-Study Method in Psychology and Related Disciplines. Chichester: John Wiley & Sons, 1986.

This book deals with both theory and practice of individual case-studies. It describes in detail how to conduct psychological case-studies, how to evaluate them, and how to use diagrams and decision analysis in case-studies.

  1. Gottman, J.M. N-of-one and N-of-two research in psychotherapy. Psychol Bull 80:93-105, 1973.

Gottman's paper suggests that time-series methods are useful in N-of-one research and we can use it to draw weak and strong causal inferences. The author discusses the use of the method in process research, outcome research, and measurement design. The paper also pay special attention to the use of interrupted time series designs. According to the author, time-series designs have the following advantage: (1) It permits the study of the single subject and the use of subject-as-his-own-control research; (2) It allows the study of the form of the effect of the intervention over time; and (3) It allows one to use information over time as feedback for making decisions.

  1. Hersen, M., & Barlow, D. Single Case Experimental Designs: Strategies for Studying Behavior Change. New York: Pergamon Press, 1976.

The book discusses general issues in single-case approach, such as behavior trends and intrasubject averaging and the problem of generality of findings. It also describes general procedures in such research. The authors then write about different types of single-case designs. Topics covered include basic A-B-A withdrawal designs and their extensions, multiple baseline designs, alternative treatment design. The last two chapters of the book introduce readers to some appropriate statistical analyses that can be used to analyse data from single-case designs and various ways to replicate findings from these experiments.

  1. Hoyle, R.H. Statistical Strategies for Small Sample Research. Thousand Oaks, CA: Sage Publications, 1999.

The book describes and illustrates statistical strategies that are appropriate for analyzing data from small samples of fewer than 150 cases. It covers such topics as the use of multiple imputation software to deal with missing data in small data sets; ways to increase power when sample size cannot be increased; strategies for computing effect sizes and combining effect sizes across studies; how to hypothesis test using bootstrapping; methods for pooling effect size indicators from single-case studies; frameworks for drawing inferences from cross-tabulated data; how to determine whether a correlation or covariance matrix warrants structure analysis; under which conditions latent variable modeling is a viable approach to correct for unreliability in the mediator; the use of dynamic factor analysis to model temporal processes by analyzing multivariate, time-series data; techniques for coping with estimation problems in confirmatory factor analysis in small samples; how the state space model can be used with small samples; and the use of partial least squares as an alternative to SEM when n is small and/or the number of variables in a model is large.

  1. Kratochwill, T.R., & Levin, J.R. Single-Case Research Design and Analysis: New Directions for Psychology and Education., Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers, 1992.

The book summarizes the latest development in the field of single-case research design and analysis. The edited volume consists of contributions from researchers from various fields who utilize or analyze data from such designs. The introductory and overview chapter of the book also have a list of past textbooks that covered the topic of single subject research design.

  1. Richards, S.B., Taylor, R.L., Ramasamy, R., & Richards, R.Y. Single Subject Research: Applications in Educational and Clinical Settings. San Diego: Singular Publishing Group, Inc., 1999.

The textbook provides background knowledge, basic concepts, and understanding of relevant issues related to applied behavior analysis and specifically to single subject research designs. It presents summaries of the use of the designs and it outlines the major features of such procedures. It also provides a review of the single subject research literature as well as descriptions on how to actually implement these designs. The designs that are covered includes withdrawal designs, multiple baseline designs, alternative treatment designs. The last chapter in the book also touched on the various kinds of analyses that are useful for analyzing single subject design data.

Back to Top

2. Observational Studies

  1. Ahlbom, A. Biostatistics for Epidemiologists. Boca Raton: Lewis Publishers, 1993.

The author presents biostatistical methods used in the analysis of epidemiological studies. It examines the theoretical background of the methods described and discussed general principles that apply to the analysis of epidemiological data. Specific topics addressed include statistical interference in epidemiological research, important methods used for analyzing epidemiological data, multivariate models, dose-response analysis, analysis of the interaction between causes of disease, meta-analysis, and computer programs.

  1. Anderson, S., Auquier, A., Hauck, W.W., Oakes, D., Vandaele, W., & Weisberg, H.I. Statistical Methods for Comparative Studies: Techniques for Bias Reduction. New York: John Wiley & Sons, 1980.

Anderson et al. present various techniques for the design and analysis of comparative studies, which are often used when randomization is not feasible. The book covered main conceptual issues in the design and analysis of comparative studies and possible bias incurred. The book provide concise and useful discussion of the techniques of matching, standardization and stratification, analysis of covariance, logit analysis and log-linear analysis, survival analysis. The later part of the book discuss the comparative effectiveness of the techniques in reducing bias and other practical issues that must be faced before drawing causal inferences from comparative studies.

  1. Clayton, D., & Hills, M. Statistical Models in Epidemiology. Oxford: Oxford University Press, 1993.

The book is intended for students enrolled for a masters degree in epidemiology, clinical epidemiology, or biostatistics. In showing how to use probability models in epidemiology the authors have chosen to emphasize the role of likelihood. Such an approach to statistics is both simple and intuitively satisfying, and has the additional advantage that it requires the model and its parameters to be made explicit, even in the simplest situations. More complex problems can then be tackled by natural extensions of simple methods and do not require a whole new way of looking at things. The book also covers in depth topics in regression models.

  1. Elwood, J.M. Causal Relationships in Medicine: A Practical System for Critical Appraisal. Oxford: Oxford University Press, 1988.

The book focuses on issues related to etiological research, early diagnosis and screening, randomized and non-randomized clinical trials, prognostic studies, health service issues, and the evaluation of health education and promotion. The book start with a discussion of the concept of causation, and then the author discuss the types of study design that can be used to demonstrate causation and the way in which their key results can be expressed precisely and simply. Chapters 4 to 8 deal with issues pertaining to subject selection, observation bias, confounding, and chance variation and various forms of validity. Chapters 9, 10, and 11 give examples of the application of the scheme to three study designs: cohort study, case-control study, and randomized clinical trial.

  1. Feinstein, A.R. Clinical Epidemiology: The Architecture of Clinical Research. Philadelphia: W.B. Saunders Co., 1985.

The content of this book can act as a "primer" for non-clinical readers, and it explains the scientific approach used by clinical investigators to quantitative challenges in group data of diagnosis, prognosis, therapy, etiology, and other medical topics. The text also cover topics and methods that are not included in text of public health epidemiology. The first six chapters provide an outlined overview of the field. The next four chapters deal with statistics. Chapters 11 to 17 offer a single standard and model for studies of cause-effect relationships in both therapeutic agents and etiologic agents. Chapters 18 to 24 concern nonrandomized designs, such as various forms of case-control studies. Chapters 25 to 27 are devoted to process evaluation. The last three chapters contain some special topics: the definition of "normal"; the contribution and limitation of randomized trials and outline of other types of epidemiology not mentioned in other parts of the book.

  1. Friedman, G.D. Primer of Epidemiology. New York: McGraw-Hill Inc., 1994.

The book was originally written to be a brief, simple, clear introduction to epidemiology for health care professionals. In subsequent editions of the book, there were elaboration and updating of some methodological concepts and factual information, as well as problems to be solved in areas in epidemiology. In this fourth edition, some new concepts and methods were introduced: the proportional mortality rate and its use in occupational studies, the coefficient of variation, the kappa coefficient and intra-class correlation coefficient, open versus closed cohorts, Kaplan-Meier survival analysis, quality-adjusted life years (QALYs), some principles concerning confounding variables, Poisson regression, receiver operating characteristic (ROC) curves. The chapter on case-control studies was considerably revised in this new edition too.

  1. Kahn, H.A., & Sempos, C.T. Statistical Methods in Epidemiology. New York: Oxford University Press, 1989.

This book is a major revision of H.A. Kahn’s An Introduction to Epidemiologic Methods, which is meant to be a book about statistical methods for chronic disease epidemiology that could be read and understood by non-statisticians. Chapters 1 and 2 cover concisely selected elementary statistics as well as various survey sampling designs and statistics. Chapters 3 and 4 describe in detail the concepts of relative risk, odd ratios, and attributable risk. Chapters 5 and 6 deal with different kinds of adjustment methods that can be applied to epidemiologic data. Chapters 7 and 8 describe followup studies, the use of life tables and person-year data. Later chapters compare results for various methods of adjustment and lay out in detail the data collection process in epidemiologic studies.

  1. Kelsey, J.L., Thompson, W.D., & Evans, A.S. Methods in Observational Epidemiology. New York: Oxford University Press, 1986. (Also see new edition in 1996.)

This text is written for researchers who have background for elementary epidemiology and biostatistics. The book mainly focuses on observational epidemiologic studies but it also covers procedures and concepts in experimental epidemiologic studies. Chapters 4 to 8 in the book describe major types of study designs used in epidemiology, including the specific situations in which each is useful, the important issues to be considered in carrying them out, and methods of statistical analysis. Chapter 9 discusses techniques of epidemic investigation, and chapter 10 describes common methods of sampling in epidemiologic studies and presents methods and tables of investigation.

  1. Selvin, S. Statistical Analysis of Epidemiologic Data. New York: Oxford University Press, 1995.

The aim of this book is to develop a clear understanding of issues important to epidemiologic data analysis without depending on sophisticated mathematics or advanced statistical theory. The level of this text is beyond introductory but short of advanced. It draws materials from the fields of statistics, biostatistics, vital statistics, and epidemiology. A number of statistical methods are surveyed in a way that should be useful to researchers concerned with the application of statistics to epidemiologic data. Additionally, these methods are chosen to illustrate general principles. For example, "jackknife" estimation (chapter 5) is an excellent way to estimate specific parameters from collected data but, at the same time, illustrates the application of a "computer-intensive" estimation method.

  1. Weiss, N.S. Clinical Epidemiology: The Study of the Outcome of Illness, 2nd ed. New York: Oxford University Press, 1996.

This book intends to pull together a number of areas of research that are devoted to measuring and determining the factors that affect the outcome of illness. It gives these areas of research a collective label: clinical epidemiology. The author assumes that the readers have background in introductory epidemiology or biostatistics. The book begins with a description of the clinical context into which the research findings ought to fit, hence the discussion of decision analysis. Next, there are chapters on the evaluation of diagnostic tests with respect to both their accuracy and their measurable contribution to illness outcome. The book deals with both the experimental and nonexperimental approaches. The concluding chapter of the book concentrates on the role of studies that measure the natural history of illness. An appendix presents selected statistical methods commonly used in planning and analyzing data from clinical epidemiological data.

  1. Woodward, M. Epidemiology: Study Design and Data Analysis. London: Chapman & Hall/CRC, 1999.

This book is about the quantitative aspects of epidemiological research. The authors assume that readers have some basic knowledge of statistics. The text goes through analytical methods for general and specific epidemiological study designs, leading to a discussion of statistical modeling in epidemiology in the final three chapters. Chapter 1 includes a broad introduction to study design, and later chapters are dedicated to particular types of design (cohort, case-control, and intervention studies). Chapter 8 concerns the problem of determining the appropriate size for a study.

Back to Top

3. Randomized Studies

  1. Fleiss, J.L. Design and Analysis of Clinical Experiments. New York: John Wiley & Sons, 1986.

The book focuses on experimental designs that are most relevant to clinical investigators. It also serves as a reference for biostatisticians who work in clinical settings. Compared with other texts on clinical trials, the book concentrates more on technical aspects of design and statistical analysis. In particular, it deals with the distinction between blocking and stratification, the use of tables of random permutations to carry out randomized assignments, the handling of unequal sample sizes, study of change estimation and interpretation of factorial effects, and how to analyze data from multicenter studies and crossover studies.

  1. Kirk, R.E. Experimental Design: Procedures for the Behavioral Sciences. 2nd ed. Belmont, CA: Brooks/Cole, 1982.

This volume on experimental design is meant to be both a text and a reference for researchers in the behavioral science. It covers different kinds of experimental designs, starting from the simplest completely randomized design to more complicated designs like the split-plot factorial designs and the fractional factorial designs. Compared to its first edition, the book also add in in-depth coverage for multiple comparison procedures, the circularity assumptions associated with block design, the partition of interactions into interpretable contrast-contrast interactions, and the analysis of factorial designs with unequal sample sizes and missing observations. It also included a chapter on general linear model approach.

  1. Miettinen, O.S. The need for randomization in the study of intended effects. Statistics in Med 2:267-271.

The author discusses the need for randomization as a means of controlling confounders that is accentuated in the study of intended effects (efficacy) as compared with unintended effects (toxicity). The indication for intervention is itself a confounder in the study of efficacy but not of toxicity. On the other hand, contraindications represent only a minor confounder even in toxicity research. Control of the indication in nonexperimental terms is commonly infeasible. The author proposes that the solution to these problems is the randomized clinical trial.

  1. Piantadosi, S. Clinical Trials: A Methodologic Perspective. New York: John Wiley & Sons, 1997.

This book attempts to acquaint investigators with ideas of design methodology that are also helpful in conducting, analyzing, and assessing clinical trials. The discussion in the book pertains to all types of trials: developmental, safety, comparative, and large-scale studies, although there is an emphasis on comparative designs. The author guides readers through the process of planning an experiment, putting together a study cohort, assessing data, and reporting results, and addresses the problems that are likely to confront any such study.

Back to Top

4. Quasi-Experiments

  1. Campbell, D.T., & Stanley, J.C. Experimental and Quasi-Experimental Design for Research. Chicago: Rand McNally, 1963.

The book is the reprint of a chapter in the Gage’s Handbook of Research on Teaching. It discusses alternatives in the arrangement or design of experiments, with particular regard to the problems of control of extraneous variables and threats to validity. The authors then discuss various kinds of threats to internal and external validity in different kinds of experimental designs. The authors classify study designs into pre-experimental, experimental, and quasi-experimental. Throughout the book, the authors try to illustrate how one can utilize idiosyncratic features of any specific research situation in designing unique tests of causal hypotheses.

  1. Cook, T.D., & Campbell, D.T. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston: Houghton Mifflin Company, 1979.

The classic text on quasi-experiment designs starts with a good discussion of plausible alternative interpretations of findings from field research. The core of the book described various quasi-experimental designs, such as research with nonequivalents groups and interrupted time-series experiments. The authors raise issues about analyzing results from such designs and how such analyses should be carried out. The book also includes chapters that deal with causal inferences from observational studies and the use of randomized experiments.

  1. Salzberg, A.J. Removable selection bias in quasi-experiments. American Statistician 53(2):103-107.

Quasi-experiments are prone to selection bias, where the effect of the treatment is confounded with pre-existing differences in the treated and control sequence groups. Some quasi-experimental designs are immune to certain specific selection biases. The article shows that immunity to selection bias is not well characterized in terms of selection-by-time interaction, and in particular some good designs can be immune to certain types of selection bias even in the presence of such interaction.

  1. Trochim, W.M.K. Research Design for Program Evaluation: The Regression-Discontinuity Approach. Beverly Hills, CA: Sage Publications, 1984.

The volume tries to make a case that we should move beyond the traditional thinking on quasi-experiments as a collection of specific designs and threats to validity toward a more integrated, synthetic view of quasi-experimentation as part of a general logical and epistemological framework for research. The papers in the edited volume cover topics like the role of judgment in quasi-experimental designs; the use of tailored designs; the crucial role of theory; the attention to program implementation; the importance of quality control; the advantages of multiple perspectives; evolution of the concept of validity; and the development of increasingly complex realistic analytic models.

Back to Top

5. Survey Research

  1. Rossi, P.H., Wright, J.D., & Anderson, A.A., eds. Handbook of Survey Research. New York: Academic Press, 1983.

The handbook is an introduction to current theory and practice of sample survey research. It address both the student who desires to master these topics and the practicing survey researcher who needs a source that codifies, rationalizes, and presents existing theory and practice. Part I of the book sets forth the basic theoretical issues involved in sampling, measurement, and the management of survey organizations. Part II deals mainly with hands-on, how-to-do-it issues, such as how to draw theoretically acceptable samples, and how to write questionnaires. Part III considers the analysis of survey data with separate chapters for each of the three major multivariate analyses most in use, and one chapter on the uses of surveys in monitoring overtime trends.

Back to Top

6. Program Evaluation

  1. Boruch, R.F., & Gomez, H. Sensitivity, bias, and theory in impact evaluations. Professional Psychol 8:411-434, 1977.

Ordinary design of experiment technology invites underpowered (insensitive) experiments because the measurement facts of life go unrecognized. The purpose of this article is to identify these measurement lapses and contribute to the development of more rigorous and socially beneficial program evaluations. Measurement should concern itself not only with reliability of dependent variable but, more importantly, with validity of measurement and with measurement (systematic observation) of the treatment variable. A technical appendix outlines a theory of measurement in field evaluation.

  1. Chen, H.T. Theory-driven evaluation. In: Advances in Educational Productivity 7:15-34, 1998.

This chapter introduces the basic concepts, rationale, and methodology of designing and conducting theory-driven evaluations with an emphasis on application so that this evaluation approach could be widely applied in areas such as education.

  1. Crits-Christoph, P., & Mintz, J. Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. J Consult Clin Psychol 59(1):20-26, 1991.

The authors present technical reasons why therapists should be included as a random design factor in the nested analysis of (co)variance (AN[C]OVA) design commonly used in psychotherapy research. Incorrect specification of the ANOVA design can, under some circumstances, result in incorrect estimation of the error term, overly liberal F ratios, and an unacceptably high risk of type I errors. Both results from simulation studies and a reanalysis of data from 10 psychotherapy outcome studies are discussed, and implications of these results for future research designs are examined.

  1. Pentz, M.A., Trebow, E.A., Hansen, W.B., MacKinnon, D.P., Dwyer, J.H., Johnson, C.A., Flay, B.R., Daniels, S., & Connack, C. Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP). Eval Rev 14:264-289, 1990.

The study evaluates the relationship between level of program implementation and change in adolescent drug use behavior in the Midwestern Prevention Project (MPP), a school- and community-based program for drug abuse prevention. Trained teachers implement the program with transition year students. Implementation was measured by teacher self-report and validated by research staff reports. Adolescent drug use was measured by student self-report; an expired air measure of smoking was used to increase the accuracy of self-reported drug use. Regression analyses were used to evaluate adherence; exposure, or amount of implementation; and reinvention. Results showed that all schools assigned to the program condition adhered to the research by implementing the program. Exposure had a significant effect on minimizing the increase in drug use from baseline to 1 year. Exposure also had a larger magnitude of intervention effect than experimental group assignment. Reinvention did not affect drug use. Results are discussed in terms of research assumptions about quality of program implementation, and possible school-level predictors of implementation.

  1. Hawkins, J.D., Abbott, R., Catalano, R.F., & Gillmore, M.R. Assessing effectiveness of drug abuse prevention: Implementation issues relevant to long-term effects and replication. In: Leukefeld, C.G., & Bukoski, W.J., eds. Drug Abuse Prevention Intervention Research: Methodological Issues. National Institute on Drug Abuse Research Monograph 107. DHHS Pub. No. (ADM)91-1761. Washington, DC: Supt. of Docs., U.S. Govt. Print. Off., 1991, pp. 213-234.

The chapter outlines a strategy for assessing the long-term effects of drug abuse prevention interventions in replicable studies. It consists of a theory-driven data collection and analysis approach that implies the need to link proximal intervention outputs to more distal outcomes desired. The approach also requires prospective longitudinal followup studies in which complete panels of subjects who vary with respect to the levels of key predictor constructs are followed up through the period of their highest risk for drug use.

  1. Leithwood, K.A., & Montgomery, D.J. Evaluating program implementation. Eval Rev 4:193-214,1980.

The authors describe a methodology for evaluating program implementation. Requirements for such a methodology are derived from an analysis of the functions to be performed by implementation evaluation, the nature of the program being implemented, and characteristics of the implementation process. Central features of the methodology involve procedures for the development of a multidimensional profile of the program as it evolves in practice from non- to full implementation. The profile then serves as the basis for instrument development; data collected through the instruments locate program user behavior in relation to the dimensions and levels of use described by the profile. Uses of resulting data to serve program management goals are outlined.

  1. Rich, K.C. Discussion: Research environment and use of multicenter studies in perinatal substance abuse research. In: Kilbey, M.M., & Asghar, K., eds. Methodological Issues in Epidemiological, Prevention, and Treatment Research on Drug-Exposed Women and Their Children. National Institute on Drug Abuse Research Monograph 117. DHHS Pub. No. (ADM)92-1881. Washington, DC: Supt. of Docs., U.S. Govt. Print. Off., 1992, pp. 293-304.

The chapter discusses the challenges presented by research environment. Often investigators have to contend with political, social, environmental, and medical factors that could determine the type and content of studies that can be performed and the likelihood of their success. The latter half of the chapter discusses the advantages and challenges of multicenter collaborative studies.

  1. Wortman, P.M. Evaluation research: A methodological perspective. Am Rev Psychol 34:223-260, 1983.

The review chapter first took a historical perspective and examines the predominant methodological school of thought from the "early days" in the development of evaluation research. It then considers more recent development and concerns in the field. The core of the chapter consists of the examination of three evaluative methods that have become increasingly important in the field: (1) social experimentation; (2) meta-analysis; and (3) cost-effectiveness analysis.

Back to Top



Archive Home | Accessibility | Privacy | FOIA (NIH) | Current NIDA Home Page
National Institutes of Health logo_Department of Health and Human Services Logo The National Institute on Drug Abuse (NIDA) is part of the National Institutes of Health (NIH) , a component of the U.S. Department of Health and Human Services. Questions? See our Contact Information. . The U.S. government's official web portal