Study Purpose

Literature Review




Return to JUNS




Reference Accuracy in Nursing Textbooks:
A Pilot Study

The Senior Nursing Students of Oregon Health & Science University
School of Nursing

Fall Term, Research in Nursing Practice, Southern Region

Research Advisor and Teacher: Linda S. Smith, MS, DSN, RN


Allensworth, Angela
Bethel, Stephanie
Smith, Linda S.

Bruhn, Tristan

Bybee, David

Cahill, Sydney

Catlin, Jeneane

Chesnut, Annie

Davis, Kacy
Drake, Kris
Jambalos, Claire
Johnson, Jody

Tracy, Abby

Grinde, Jennifer

Hambly, Betsi

Heikkala, Emily

Hill, Marcie

Hilson, Melissa

Hooper, Kati

Huitt, Nikowa

Huntsinger, Karen

Iwen, Molly

Jordan, Shannon

Klakulak, Amy

Maeda, Melissa

Mayo-Adams, Melisa

Medlock, Kari

Meuser, Melissa

Miller, Wendie

Price, Steve

Robledo, Connie

Runsvold, Jessie

Smith, Andrea

Smith, Deborah

Smith, JoAnne

Swanson, Siri

Wallace, Heather


Students enrolled in a ten-week nursing course completed a quantitative, descriptive research proposal and pilot investigation related to reference accuracy in nursing textbooks. The class was divided into nine groups, with each group responsible for completing an assigned portion of the proposal (Smith, 2003). Groups met weekly to compose and collaborate.


Research components included the question/problem statement, the purpose and study significance, theoretical framework including assumptions, limitations, and hypotheses, a literature review, study methodology, pilot investigation, and collection and analysis of pilot data.


Problem Statement


The research question was, “What is the level of accuracy of reference citations in

current English-language U.S. nursing textbooks published from the years 1997-2002?” As senior nursing students and future professionals, we believe that it is imperative for textbook authors and publishers to substantiate information provided relative to nursing knowledge and science.


Study Purpose


The purpose of this study was to describe, quantify, and communicate the quality and accuracy of reference citations in current English-language US nursing textbooks, published from the years 1997-2002. The unit measured was the reference citations in nursing textbooks. 


Study Significance


This problem is of interest both to nursing students and practicing nurses.  Nursing textbook authors have but one goal and function: to communicate the ideas and principles of  nursing science to the reader. This information is essential for the growth and development of knowledge for our profession.  Erroneous information results in unsuccessful communication and fails to achieve the author’s intended goal.


As nursing literature consumers, we believe accurate references and information are needed and expected. Nursing populations use textbooks as learning materials, references, and as sources for resource validation. Finally, there are no available published articles evaluating nursing textbook accuracy.


Theoretical Framework


The focus of the framework is information accuracy and the ease of its retrieval. According to Berlo (1975) information systems management has become one of the central competencies needed in modern society.  It does little good to generate information if there are not adequate systems for storage and retrieval, and for information dissemination, utilization, and evaluation. 


Through Berlo’s theory (1960) the process of communication begins with a source that attempts to intentionally affect person(s) through communicated information. Berlo recognizes the dyadic relationship of author/reader. When textbook references (message) are erroneous or unclear, communication (message) collapses because readers (receivers) are unable to accurately interpret the author’s (source’s) message.  The more credibility the source has, the more likely the receiver is to accept the message being relayed.  If numerous flaws are present in the body of the text or reference list, receivers may question whether additional mistakes have been made (Berlo, 1970).


Term Definitions


Error - unintentional deviation from what is correct


Major Error - prevents document retrieval; obstructs purpose of giving cited authors credit for work; interferes with information retrieval


Minor Error - Incorrect use of diacritical marks in author names, minor title errors, or errors in the listing or spelling of publisher names/place of publication


Reference/Citation - allows location, attribution, and identification of original work


Nursing Textbook - book used for the study of nursing




The following assumptions are made:

1.   Nursing textbook references are intended to be complete including authors, full titles, publishers, editors, content, dates, volume numbers, and page numbers.

2.   Nursing textbook references are reviewed for accuracy prior to publication.

3.   Sample population of nursing textbooks is representative of the entire target population of nursing textbooks.

4.   Textbook references can be accurately measured by a tool.

5.   Reference inaccuracy negatively impacts information dissemination and author attribution.



This study may have the following limitations:

1.      The pilot study uses a small, non-random sample of only four textbooks.

2.      The references are compared only with online database information, which could also have errors.

3.      The Reference Analysis Tool (RAT) may have reliability/validity limitations.

4.      The study’s accuracy relies on evaluators having correct referencing knowledge.

5.      The study’s accuracy may be affected by evaluator bias.

Literature Review


Researchers have investigated reference accuracy in nursing and non-nursing journal articles; however, no research was found pertaining to nursing textbook accuracy. Reviews of published nursing and non-nursing papers have revealed major and minor reference errors.


Nursing Journal Articles


In 1998, Schulmeister outlined reference list importance. Functioning as a tool for readers as well as a method for authors’ attribution, reference accuracy is vital in the retrieval of citied literature and the integrity of the research article itself. Because of the rate of error in research journals, researchers encourage their peers to compare each element of citations with the original article.


To evaluate references in journal articles, Taylor (1998) defined major errors that impede source retrieval: omitted/misspelled author names, titles; errors in page numbering; and inaccurate publication year, journal edition, and volume.  Minor errors included minor title misspellings, incorrect punctuation, errors in publisher names, and page number errors within correct page number limits. A total of 262 reference citations were analyzed; 45.8% of the citations had errors with 38.2% of the references containing at least one major error.


One study investigated the accuracy of references in neonatal maternal nursing literature (Oermann, Wilmes, & Braski, 2002). The researchers randomly selected 10% of over 2000 articles from two journals. After analysis, 24.4% of references had errors, and of these errors, 21.3% were major errors and 3.2% were minor errors.  Oermann, et al believed that while minor errors are merely a nuisance,  major errors endangered article credibility for the author and the publisher.


Non-Nursing Journals


“All uses of citations… are predicated on truthfulness and accuracy,” (Hinchcliff, Bruce, Powers & Kipp, 1993, p.397).  However, research on non-nursing journal reference accuracy revealed inaccuracy rates among these article references to range from 20% to 50%.  Reference accuracy investigators divided references into correct and incorrect.  A correct reference was identical to the source (Doms, 1989). Incorrect references were further divided into those with major and minor errors.  Major errors were defined as errors that impede immediate retrieval or source identification. These errors included author name, journal name, title, volume, year, and page number (Doms; Goldberg, et al., 1993; Hinchcliff et al., 1993; Lee & Lee, 1999; Nishina, Mikawa, Asano, Maekawa & Obara, 1995).  Minor errors were misspellings, omissions, and substitutions that did not prevent the reader from obtaining the source (Hinchcliff et al.1993; Nishina et al 1995.).


When Lee and Lee (1999) evaluated reference accuracy in the Journal of Dermatology and the Korean Journal of Dermatology, 100 references from each journal were randomly selected and checked against original sources.  These investigators found error rates of 24% and 33%, respectively.  Title and author errors were the most common. In another study of references in three emergency medicine journals, there were rates of 10.3% for major errors, and 17.2% for minor errors (Hinchcliff et al., 1993).




When an article has erroneous references, the quality of the entire article is questioned. Though primary responsibility of reference accuracy belongs with authors, editors and staff may need to check at least 10% of all citations.  Unusual high error rates could be returned to authors for additional verification and correction (Doms, 1989; Goldberg et al., 1993).






The subjects for this proposed study will be nursing textbooks published in The United States. Researchers will access the needed textbooks in local, regional, or national libraries. Other textbooks will be solicited from  faculty and from nursing textbook publishers.




Publication dates for all the nursing textbooks in the Brandon/Hill Selected List of Print Nursing Books and Journals (Hill & Stickell, 2002) will be 1997-2002. Of the 370 listed books, 10 books (3%) were published in 1997, 33 books (9%) in 1998, 99 books (27%) in 1999, 87 books (23%) in 2000, 68 books (18%) in 2001, 73 books (20%) in 2002. Percentages represented by publication year will be the number of books used to make up the sample. Sample size will be 100 books; researchers will use the random number generator to select 20% of the books (20% of 100 is 20). After the 20 books have been selected, researchers will randomly select 15% of the chapters in each book. Researchers will then randomly select five references from each chapter.




This is a descriptive, quantitative study looking at reference errors in nursing textbooks.


Ethical considerations


This study will not involve human subjects. Study units are references published in nursing text books; thus, references are public information.


Problem/Question statement


What is the level of reference accuracy of reference citations in current English-language U.S. nursing textbooks published from the years 1997-2002?




The Reference Analysis Tool (RAT) will be used by researchers to analyze nursing textbook reference citations.  RAT was derived from criteria used in other research studies and formulated to specifically analyze nursing textbook references.


Tool validity was established in a number of ways; 1) through criteria pulled and adapted from published reference accuracy studies; 2) A draft RAT was distributed and evaluated by doctorally prepared nurses; 3) feedback was gathered from faculty and 38 class members; 4) feedback comments/suggestions were incorporated into the revised RAT.


Tool reliability was established by using the same criteria that proved reliable in other research studies.  Post-pilot reliability statistics established internal consistency with tool criteria.


The tool incorporates criteria for major and minor reference citation errors. Additional RAT criteria include reviewer initials, reference source, and whether the reference related to a primary or secondary source.


Summary and Conclusion of Pilot Study


Cluster random sampling was used to identify 125 references from four pre-chosen nursing textbooks. These books were from different nursing levels and different publishers. Citations were collected using primarily Worldcat (for books), and OVIDweb/MEDLINE databases. Of the 125 citations, only 112 were obtainable. Using the RAT, six trained raters analyzed  references. Two trained student raters evaluated each reference independently.


SPSS (2002) version 11.5 software program was used to enter and analyze our pilot data. We computed a one-way ANOVA comparing the rate of total number of major errors with one of four different textbooks. A significant difference was found among the textbooks (F=7.725, p<.001). Tukey’s HSD was used to determine the nature of the differences among the textbooks. This analysis revealed that the Kozier text (2000) had fewer (M=1.21 SD=1.47) major errors than the Lewis (2000) text (M=2.7 SD=2.67) and the Lewis text had more major errors than the Mason (M=.62 SD=.69) text (2002). Major error rates for the Dempsey (2000) text (M=1.1 SD=.78) were not significantly different from the other texts used in this study. Additional tool training for each reference rater, including a specific definition of each of the items on the RAT, would improve inter-rater reliability.


Data Collection and Analysis for Major Study


All raters will be RAT trained and given definitions of tool items. References collected will be independently evaluated by two researchers. Results will be entered into the SPSS version 11.5 for analysis. Collected references will be analyzed using the revised RAT. References will be compared with corresponding on-line database information. Parametric inferential statistical procedures will be used for data analysis with an apriora alpha of < 0.05.  To compare 100 textbooks Chi squared and/or one-way ANOVA analysis techniques will be used.


Nursing Implications


Reference accuracy affects the tripartite of nursing practice, education, and research.  High quality referencing information is necessary to verify that nursing practice is evidence-based.  Professional nurses need to locate appropriate published studies and evaluate papers for scientific merit and clinical practice applicability. Thus, retrievable quality nursing research improves nursing practice.


When reference inaccuracies are present, consumers of evidenced-based research are prevented from basing their knowledge and teaching on primary sources.  Nursing students need to have confidence in sources cited in their textbooks.


Finally, reference accuracy facilitates research and research dissemination. Nurse researchers identify healthcare problems and investigate solutions to those problems.  Referencing reflects on research quality.  Research referencing must be accurate so that other investigators can replicate the study in order to build a stronger scientific basis for research utilization.  Complete and accurate referencing improves nursing practice, education, and science.



Berlo, D. K. (1960). The process of communication: An introduction to theory and practice. New York, NY: Holt, Rinehart, and Winston, Inc.

Berlo, D. K. (1970). Dimensions for evaluating the acceptability of message sources. Public Opinion Quarterly, 33(4), 563-576.

Berlo, D. K. (1975). The context for communication. In Addison-Wesley (Eds.),      Communication and behavior (pp. 3-18). New York, NY: Holt, Rinehart, and Winston, Inc.


Doms, C. A. (1989). A survey of reference accuracy in five national dental journals. Journal of Dental Research, 68(3), 442-444.


Goldberg, R., Newton, E., Cameron, J., Jacobson, R., Chan, L., Bakata, W.R., et al., (1993).  Reference accuracy in the emergency medicine literature. Annals of Emergency Medicine,   22(9), 1450-1454.


Hill D., & Stickell, H. (2002). Brandon/Hill selected list of print nursing books and Journals. Nursing Outlook, 50(3), 100-112.


Hinchcliff, K. W., Bruce, N.J., Powers, J.D., & Kipp, M.L. (1993). Accuracy of references and quotations in veterinary journals. American Veterinary Medical Association, 202(3), 397- 400.


Kozier, B., Erb, G., Berman, A. J., & Burke, K (Eds.). (2000). Fundamentals of nursing:  Concepts, process, and practice (6th ed.). Upper Saddle River, NJ: Prentice Hall Health.


Lee, S.Y., & Lee, J.S. (1999). A survey of reference accuracy in two Asian dermatologic journals: The journal of dermatology and the Korean journal of dermatology. International Journal of Dermatology, 38, 357-360.


Lewis, S. M., Heitkemper, M. M., & Dirksen, S. R. (Eds.). (2000). Medical-surgical nursing: Assessment and management of clinical problems (5th ed.). St. Louis: Mosby, Inc.


Lok, C.K.W., Chan, M.T.V., & Martinson, I.M. (2001). Risk factors of citation errors in peer-reviewed nursing journals.  Journal of Advanced Nursing, 34(2), 223-229.


Mason, D. J., Leavitt, J. K., & Chaffee, M. W. (Eds.). (2002). Policy & politics in nursing and health care (4th ed.). St. Louis, MO: Saunders An imprint of Elsevier Science.


Nishina, K., Mikawa, K., Asano, M., Maekawa, N., & Obara, H. (1995). Reference accuracy in critical care medicine. Critical Care Medicine, 23(9), 1610-1611.


Oermann, M.H., Wilmes, N.A., & Braski, P. (2002). Reference accuracy in neonatal-maternal nursing literature.  Neonatal Network, 21(1), 23-26.


Schulmeister, L. (1998). Quotation and reference accuracy of three nursing journals. Image-the Journal of Nursing Scholarship, 30(2), 143-146.


Smith, L. S. (2003, Fall). Course syllabus NURS 470: Research in nursing practice. (Available from the Oregon Health &  Science University School of Nursing, 3201 Campus Drive,  Klamath Falls, OR 97601)


Statistical Package for the Social Sciences [SPSS]. (2002). SPSS 11.5 for Windows. Chicago, Illinois: SPSS, Inc.


Taylor, M.K. (1998). The practical effects of errors in reference lists in nursing research journals. Nursing Research, 47(5), 300-303.



Copyright© by The University of Arizona College of Nursing; All rights reserved.