Perspectives in Clinical Research

RESEARCH METHDOLOGY
Year
: 2011  |  Volume : 2  |  Issue : 4  |  Page : 137--144

Patient-reported outcomes: A new era in clinical research


Prasanna R Deshpande, Surulivel Rajan, B Lakshmi Sudeepthi, CP Abdul Nazir 
 Department of Pharmacy Practice, Manipal College of Pharmaceutical Sciences, Manipal, India

Correspondence Address:
Prasanna R Deshpande
Department of Pharmacy Practice, Manipal College of Pharmaceutical Sciences, Manipal University, Manipal - 576 104
India

Abstract

Now-a-days there is significant discussion about patient-reported outcomes (PRO) in medical world. The following article covers almost all the areas of PRO including-their importance, important concepts for understanding of PRO, significance, ideal properties, types, development and evaluation of PRO instruments. It is useful for physicians, pharmacists and patients for the assessment and improvement of the therapy.



How to cite this article:
Deshpande PR, Rajan S, Sudeepthi B L, Abdul Nazir C P. Patient-reported outcomes: A new era in clinical research.Perspect Clin Res 2011;2:137-144


How to cite this URL:
Deshpande PR, Rajan S, Sudeepthi B L, Abdul Nazir C P. Patient-reported outcomes: A new era in clinical research. Perspect Clin Res [serial online] 2011 [cited 2019 Sep 23 ];2:137-144
Available from: http://www.picronline.org/text.asp?2011/2/4/137/86879


Full Text

 Introduction



'Patient' can be considered as the centre for any healthcare system. Now-a-days there is growing realization for the patient-centered healthcare system. [1] The outcomes of a clinical intervention obtained by the patient i.e. patient-reported outcomes (PROs) are seemed to be of more importance in future than any other outcomes like clinical, physiological or caregiver-reported. As per studies, enhanced treatment adherence and outcomes can be obtained by giving attention to patient feedback on healthcare outcomes and patient behavior change. [2]

As per US-FDA, a PRO is any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone else. [3]

 Importance of PRO



Though medical technology allows to measure physical, physiological or biochemical data of the patient; it is not able to give all the data about the treatment or the disease. Some data can be obtained only from the patient. Such data includes the things as mentioned in [Table 1]. [4] {Table 1}

In short, patient can tell many things like thoughts, complaints, opinions that technology or any observer can't and which is actually more valuable. As discussed earlier PRO are the measurements come directly from the patient. Further to add, in some diseases survival is not the ultimate goal of the treatment but quality of life also plays an essential role in the treatment. E.g. Cancer chemotherapy. [5] Therefore, PROs are gaining the importance in clinical world.

 PRO and Other Types of Outcomes



The outcomes are broadly classified into- clinical (e.g. cure, survival), humanistic (e.g. role performance, emotional status) and economical (e.g. expenses, saving). [6] In clinical scenario the outcomes can be clinician reported (e.g. performance of the patient), caregiver reported (e.g. functional status), physiologic (e.g. tumor size by using MRI) or patient reported (e.g. symptoms). If the patient is observed for the outcomes by clinician, researcher or caregiver then the outcomes become observer reported outcomes (OROs). [4] If the patient is revealing in the written questionnaire that he/she is experiencing morning stiffness is PRO but if the clinician is asking to describe about the morning stiffness i.e. severity and nature is considered to be ORO.

Proxy reported outcome is a measurement based on the reportby someone other than the patient reporting as if he or she is the patient. It is different from a PRO or ORO. [3]

 Difference Between Symptoms and Health Related Quality of Life



Although symptoms of the patient and health related quality of life (HRQOL) hear to be similar, they are the two different concepts. Symptom is a one-dimensional property while HRQOL is multidimensional. Symptoms are often main objective of treatment, mirror clinician-patient discourse and vary dynamically with time but in case of HRQOL all the above things are rarely. Symptoms are directly related to disease and treatment effect while there is indirect relation of them with HRQOL. In case of PRO concepts symptoms are often considered for behavioral objective measure but seldom for HRQOL. Complexity of the concepts is complex in HRQOL and simple in case of symptoms. [7]

 Important Concepts



Prior to know about PRO one should be familiar with the following concepts:

PRO instrument

A PRO instrument (i.e., a questionnaireplus the information and documentation that support its use) is a means to capture PRO data used to measure treatment benefitor risk in medical product clinical trials. [3]

PRO concept

It is nothing but the thing or event being measured. It can be called as the specific goal of the instrument. [3]

e.g. symptom or group of symptoms

PRO domain

A subconcept represented by a score of an instrument that measures a larger concept comprised of multiple domains. [3]

e.g. depression

PRO item

An individual question, statement, or task (and its standardized response options) that is evaluated by the patient to address a particular concept. [3]

e.g. Are you feeling depressed?

Conceptual framework

The conceptual framework explicitly defines the concepts measured by the instrument in a diagram that presents a description of the relationships between items, domain (sub-concepts) and concepts measured and the scores produced by a PRO instrument [Figure 1]. [3] {Figure 1}

Endpoint

The measurement that will be statistically compared among treatment groups to assess the effect of treatment and that corresponds with the clinical trial's objectives, design, and data analysis. [3]

End point model

A diagram of the hierarchy of relationships among all endpoints, both PRO and non-PRO, that corresponds to the clinical trial's objectives, design, and data analysis plan [Figure 2]. [3] {Figure 2}

Conceptual equivalence

It is the equivalence in relevance and meaning of the concepts being measured in different languages and /or cultures. [8]

Health Related Quality of Life

In simple words HRQOL can be defined as personal health status of the individual. [9] Actually, it is a multi-domain concept which represents the patient's general perception of the effect of illness and treatment on various aspects of life such as physical, psychological, and social. [3] Some other aspects also can be predicted to affect HRQOL like- economical, disease symptoms, adverse drug reactions, patient-education, disease-treatment given to the patient [Figure 3]. {Figure 3}

 Ideal Properties of PRO Instrument



As per the literature, [3],[9] following ideal properties can be extracted-



It should be specific to the concept being measured.It should be based on end-point model. It should have conceptual equivalence.It should be based on the conceptual framework.It should contain optimum number of items.It should have easy and specific measurement properties i.e. use of the scales which is easiest for the intended population to understand.It should have proper evidences for the conceptual framework.It should maintain the confidentiality of the patient.It should be reproducible.

 Significance of PRO Instruments



By using PRO, various types of outcomes can be measured such as physical functions, symptoms, global judgments of health, psychological well-being, social well-being, cognitive functioning, role activities, personal constructs, satisfaction with care, [10] health related quality of life (HRQOL), [11] adherence to medical regimens [12] and clinical trial outcomes. [10]

PROs may be helpful in determination of the eligibility of the patient for certain clinical trials. E.g. Inclusion criterion for the trial is female patients with hot flashes. PROs can be used for confirmation of the measures. E.g. patients with morning stiffness are most likely to be suffered from rheumatoid disease. PROs can help to interpret other measures and also help to eliminate other possibilities. E.g. If the patient has breathlessness and the patient is a smoker then COPD can be expected rather than anemia. PROs are useful in determination of patient compliance or reasons for non-compliance. E.g. Are the side effects so severe? PROs may be used as study endpoint. E.g. Efficacy of analgesic drug by determining pain levels. [4]

In diseases like cancer it is important to determine the quality of life of the patient as patients with progression of cancer frequently experience multiple symptoms, economical burden, home management problems and lack of emotional well-being, all of which can adversely affect quality of life. [13] PROs are helpful in the determination of quality of life in cancer patients. Role of QOL/PROs in cancer care can be considered in the following conditions like [14]



Determination of negative effects of the adjuvant therapyIdentification of the needs for the supportive careComparison of two standard therapies having similar survival outcomesIdentification of negative effects of the therapy when survival time is longTo find out whether a new therapy is preferable to standard therapyTo determine whether a therapeutic regimen is better than supportive care only, when survival time is shortTargeting problems and making the communication easier in clinical practice

As PRO instruments are based upon the scores obtained by the patient without any interpretation; it gives a clear picture about the patient condition. PROs are useful for the patients, patient-family members and loved ones, healthcare providers, payers, regulatory authorities, technological assessors and researchers. PROs are beneficial for the patients as he/she asks for and experience the treatment. It is useful for patient family members and loved ones as they desire the best for the patient and themselves. The PRO data is also useful for healthcare providers and payers as they make a decision with patients on prescribing of treatments/therapy and pay for the treatment respectively. Information obtained from PRO can also be used by regulatory authorities (e.g. FDA) and technology assessors [e.g. National institute for health and clinical excellence (NICE)]. PRO studies make the consequences of the studies accessible through various publications like formulary submission dossiers, direct to consumer advertisements, journals and continuing medical educations (CME). [9],[15],[16]

Conclusively, PROs are unique indicators of impact of disease on the patient, helpful in empowerment of the patients, necessary for determination of efficacy of the treatment, by communication helpful in creating a rapport between patient and healthcare providers, useful in the interpretation of clinical outcomes and treatment decision making. [9]

 Types of PRO Instruments



Broadly, PRO instruments can be classified as per the measurement of the interested concept as [10],[17]



Generic (anticipated to detain a very wide variety of aspects of health status and the outcomes of illness and hence to be applicable to a wide range of patient groups) e.g. Psychological General Well-Being Index (PGWBI)Disease specific e.g. Rheumatoid Arthritis Quality of Life questionnaire (RAQoL)Dimension specific e.g. Physical Activity Index (PAI)Region/site specific e.g. Cambridge Pulmonary Hypertension Outcome Review (CAMPHOR)Individualized (instruments in which the respondent is allowed to select issues, concerns or domains of personal concern that are not predetermined by the investigator's list of questionnaire items) e.g. Schedule for the Evaluation of Individual Quality of Life (SEIQoL)Utility measures (considered as generic type but with one particular form of numerical evaluation of healthiness) e.g. Utility measure for major, unipolar depression (McSad)Summary items e.g. questions (regarding limiting enduring illness) in General Household Survey

All types of PROs have their own advantages and disadvantages.

 Development of PRO Instrument



As per US-FDA, [3] [Figure 4] there are five steps for the development of a PRO instrument. First, there will be theorization of a conceptual network. It may include outlining of potential claims and hypothesized concepts, determination of intended population and application i.e. scores, mode and frequency of administration, performance of expert/literature review, development of the framework, assignment of PROs in preliminary endpoint model and documentation of the preliminary instrument.{Figure 4}

The second step may include finding of the patient inputs, making of new items, assortment of recall period, response options and format, selection of mode of administration and data collection, conduction of patient cognitive discussion, pilot testing of the outlined instrument and documentation of content validity.

Third step may include the following-confirmation, assessment and finalization of the instrument, documentation of measurement progress. In assessment of the score reliability, construct validity and ability to detect change will be done.

Fourth step may contain- preparation of protocol and SAP (statistical analysis plan), compilation and analysis of data, assessment of treatment response (by means of responder definition and cumulative distribution), documentation of interpretation of treatment advantage (in relation to claim).

Fifth and last step in the development of the instrument is modification of the instrument which include- altering of phrasing of items, population response options and mode of administration, translation and culturally adaptation of the instrument, evaluation and documentation of the changes.

 Administration of PRO Instrument and Data Collection



Data collection by using a PRO instrument can be done in three ways [3] as by interview, self administration or combination of both. Live interviews may happen in person or by means of communication medium. Every method has its own positive and negative aspects. [4]

 Response Scales



It is a set of answer choices that fall into an order e.g. from highest to lowest. A scale may be composed of pictures, numbers or categories. [4] Following types of response scales or options may be used in a PRO instrument. [3] The response scales are mentioned with example in the [Table 2]. {Table 2}

Recording of events is also one of the methods to determine the response that can be included by the patient. E.g. dairy maintained by the patient for the episodes of urination in night.

 Evaluation of PRO



For the evaluation of a PRO instrument following things may be checked [3]



Validity

Content validity

Construct validity

Criterion validityReliability

Test-retest or intra-interviewer reliability

Internal consistency

Inter-interviewer variabilityAbility to detect change

Validity

It is the degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses of tests. [18] Validity is not the property of the instrument i.e. PRO instrument but of instrument scores and their interpretations. [18],[19] Therefore, validity must be established for all proposed interpretation.

Content validity

It is the extent to which the instrument measures the concept of interest. [3] Content validity of the instrument can be performed qualitatively as well as quantitatively. [20] According to US-FDA(1), following attributes can be checked- item generation, data collection method and instrument administration mode, recall period, response options, instrument format instruction and training, patient cognition, scoring items and domains, respondent and administrator burden. The instrument should have proper evidence in the literature linking between concept being measured by the instrument and targeted claims. [3],[20] Quantitative analysis of the items is also desirable and helpful. [20] E.g. evaluation of how well items deal with complete continuum of patient experiences. Item response theory (IRT) and Rasch analysis can be used to evaluate item information curves and what fraction of the response continuum items address. [21],[22]

IRT is a powerful tool for the evaluation of the questionnaire (PRO instrument) both at item and scale levels and tailors use of the same to the patients. [23] Some important methodological considerations for IRT modeling are [24]



Should IRT methods be used by the researcher for the questionnaire? Which IRT model should be used?What sample size is needed for IRT analysis?

Rasch model is the only IRT model in which a person is characterized totally by the total score across the questions/items. Further to add, it is the simplest of such models having least number of the parameters for the person and only one constraint for each category of an item. It may be used to verify the degree to which scoring and summing is justifiable in the collected data and how well the information fit in the model. [25]

Construct validity

It is defined as evidence that relationships among concepts, domains and items conform to a priori hypotheses concerning logical relationships that should exist with other measures or characteristics of patients and patient groups. [3] It includes convergent, divergent and known groups' validity. In construct validity, measures of constructs theoretically should be related to each other and should in fact observe to be related to each other. In the similar way, in discriminant validity, one should be able to discriminate between dissimilar constructs. [26] In known groups' validity, evidence for the thing that the instrument can differentiate between clinically distinct groups is checked. [3]

Criterion validity

It is nothing but the extent to which the scores of a PRO instrument are related to a known gold standard measure of the same concept. [3]

Here, the performance of the operationalization is checked against some criterion. In content validity, the criteria are the construct definition itself i.e. it is the direct judgment while in criterion validity, often there is a prediction about how the operationalization will perform based on the theory of construct. It may be of two types as "predictive" and "concurrent". In the former one there is assessment of the ability to predict something in the real world which is present theoretically. For instance, if there is measurement of slowness of movement in Parkinson's disease, we could give our measure to the respective patients and see if there is a high correlation between the scores and the degree of slowness. The presence of high correlation makes available an evidence for "predictive validity". In concurrent validity, there is assessment of the ability of the item to distinguish between the groups in real world as mentioned theoretically. E.g. in the assessment of breathlessness, the item should be able to distinguish between asthma and COPD. [26]

Practically, it is difficult to determine the criterion validity as there is no gold standard for most PROs. [3]

Reliability

It is the ability of a PRO instrument to yield reproducible and consistent estimates of true treatment effect. [3] It is a necessary (but not adequate) component of validity. [27],[28] To establish the reliability of the scores before using the PRO-instrument in practice, it is necessary that there should be sufficient accrual of evidence. [29] It can be predicted by three ways as 1) test-retest reliability- it is a measure of the ability of a psychologic testing instrument to yield the same result for a single Patient at 2 different test periods, which are closely spaced so that any variation detected reflects reliability of the instrument rather than changes in the Patient's status. [30] It may be considered as intra-interviewer reliability for interviewer administered PROs. This type of reliability is checked by intra-class correlation coefficient (ICC). [3] As per the generic definition ICC is the ratio of true variance i.e. the variability between the targets and observed variance i.e. the total variance-true variance plus other variance.Time period between test and retest is an important factor here. 2) Internal consistency-The consistency of the results delivered in a test, ensuring that the various items measuring the different constructs deliver consistent scores. [31] Internal consistency is determined with Cronbach's alpha. Cronbach's alpha is coefficient of reliability. [32] It is defined as [33]

[INLINE:1]

Where, K = the number of items

[INSIDE:1] = the variance of the observed total test scores

[INSIDE:2] = the variance of component i for the current sample of persons

The α value theoretically ranges from 0 to 1. The value of ≥0.7 is considered to be optimum. Item-total correlation is yet another test that may be considered for internal consistency. It is the measure of the association between an item and the total score from the set of the items within the scale. [34] It can be measured by using Pearson correlation coefficient. If the value of the Pearson correlation coefficient is high then stronger is the relationship between the items. 3) Inter-interviewer reliability - It determines the changes in the results when the instrument is administered by two or more interviewers. It is generally measured by Pearson's 'r' value.

Ability to detect the change

It is considered to be an important part in the evaluation of PRO instrument. The instrument should be able to detect the changes in the expected outcomes. For instance if there is an instrument which is determining the quality of life of breast cancer patient, then it should be able to predict the quality of life after the treatment. The quality of life score should be able to predict the improvement/stability in condition/worsening. Ability of the instrument to detect the changes may be affected by the various factors as per the type and concept to be determined. [3]

 Electronic PRO



Here the outcomes of the patients are collected by using electronic methods like electronic diaries, computers, telephones etc. The use of Electronic PRO (e-PRO) are found to be beneficial than paper based PRO. [35] e-PROs avoid data entry errors, give immediate access to data, enable to trigger alerts/notifications, reduce the missing information as compared to paper-based PRO, increase patient's willingness to report sensitive information. Further to add, data obtained from e-PRO gives real-time tracking of survey compliance. Some important barriers in use of e-PRO are- increased expense, limited time for patient-training and infrastructure study site. There may be some cultural resistance for the administration of electronic instrument. [36]

The use of e-PRO was found to successful in various recent clinical trials. [37],[38]

 Summary





Important concepts to be known - PRO, Concept, Domain, Item, End-point model, Conceptual framework, Conceptual equivalence, HRQOLIdeal properties of PRO instrument - Concept specific, End-point model, Based on conceptual framework, Optimum number of items, Reproducibility, Easy accessibility of scales by patients, Availability of proper evidence, Conceptual equivalence, Maintenance of confidentiality.Types of PRO instruments - Generic, Disease specific, Dimension specific, Region specific, individualized, Utility measures, Survey itemsDevelopment of PRO instrument- (Steps)

Theorization of a conceptual framework;Adjustment of the conceptual framework and outlining instrument;Authentication of the framework and evaluation of further measurement properties;Assemble, examine and interpret data; Adapt the PRO instrumentTypes of response scales - Likert scale, visual analog scale (VAS), Anchored VAS, Pictorial scale, Rating scale, ChecklistEvaluation of PRO instrument-

Validity - Content, Construct, Criterion;Reliability - Test-retest reliability, internal consistency, inter-interviewer reliabilityAbility to detect change

References

1International Alliance of Patients' Organizations. What is Patient-centred Health Care? A Review of Definitions and Principles, 2 nd ed. London:IAPO;2007. p. 1-34.
2Institute of Medicine, Crossing the Quality Chasm-A New Health System for the 21 st Century, 1 st Ed. Washington, DC:National Academies Press;2001
3U.S Department of Health and Human Services Food and Drug Administration Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. U.S. FDA, Clinical/Medical 2009 available form: [http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf]. [Last Accessed on2011 Apr 20].
4Chin R, Lee BY. Economics and patient reported outcomes, Principles and practice of clinical trial medicine, London, Amsterdam, Burlington, San Diego:Elsevier Inc;2008 p.145-66.
5Singh DP. Quality of life in cancer patients receiving palliative care. Indian J Palliat Care 2010;16:36-43.
6Kozma CM, Reeder CE, Schulz RM. Economic, clinical, and humanistic outcomes: a planning model for pharmacoeconomic research. ClinTher 1993;15:1121-32; discussion 1120.
7ShieldsA, Gwaltney C, Paty J, Shiffman S. Distinguishing Among Symptom vs. Health Related Quality of Life PRO Concepts: Developing a Conceptual Framework August 22, 2006.Available from;http://www.invivodata.com/media/files/PRO%20Consulting_White%20Paper_Symptoms%20and%20HRQOL.pdf. [Last Accessed on 2011 May 20]
8Pashos CL, Klein EG,Wanke LA (editors). ISPOR Lexicon™. Princeton: International Society for Pharmacoeconomics and Outcomes Research . NJ 1998.
9Acquadro C, Berzon R, Dubois D, Leidy NK, Marquis P, Revicki D, et al. Incorporating the Patient's Perspective into Drug Development and Communication: An Ad Hoc Task Force Report of the Patient-Reported Outcomes (PRO) Harmonization Group Meeting at the Food and Drug Administration, February 16, 2001. Value Health2003;6:522-31.
10Fitzpatrick R, Davey C, Buxton MJ, Jones DR. Evaluating patient-based outcome measures for use in clinical trials. Health Technol Assess 1998;2:1-74.
11Chen H, Taichman DB, Doyle RL. Health-related quality of life and patient-reported outcomes in pulmonary arterial hypertension.Proc Am ThoracSoc 2008;5:623-30.
12Chao J, Nau DP, Aikens JE. Patient-reported perceptions of side effects of antihyperglycemic medication and adherence to medication regimens in persons with diabetes mellitus.clinTher 2007;29:177-80.
13Miller R,Walsh D. Psychosocial aspects of palliative care in advanced cancer. J Pain Symptom Manage 1991;6:24-9.
14Yarbro CH, Wujcik D, Gobel BH. Quality of life as an outcome in cancer care. Cancer Nursing: Principles and Practice. Sudbury, Ontario, London; John and Bartlett;2011. p. 201-18.
15Assessing and reporting outcomes important to patients in clinical trials and Cochrane reviews XV Cochrane collaboration 25 Oct. 2007 by Donald L. Patrick. Available from:http://www.colloquiumbrasil.info/program/public/documents/plenary_2b_patrick-151949.pdf. [Last Accessed on 2011 June 1].
16Willke RJ. Measuring the value of treatment to patients: Patient-reported outcomes in drug development. Am Health Drug Benefits 2008;1:34-40.
17Patient reported outcomes quality of life outcomes instruments database. Mapi research institute. Available from: http://www.proqolid.org/proqolid/search__1/alphabetical_list[Last Accessed on 2011 Apr 1].
18American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC:American Educational Research Association; 1999.
19Messick S. Validity. In: Linn RL, editor. Educational Measurement,3rd Ed. New York: American Council on Education and Macmillan;1989.
20Assessing Respondent Understanding of Newly-Developed Patient-Reported Outcome (PRO) Instruments for Medical Product Evaluation: ISPOR PRO Good Research Practices on Establishing and Reporting Evidence of Content Validity Task Force Report - Part IIAvailable from: www.ispor.org/taskforces/.../ISPOR-PRO-CV-TF-2-Assessing-for-Understanding-for-REVIEW.pdf [Last Accessed on 2001 Mar 25].
21Andrich D. Rasch models for measurement. Beverly Hills: Sage Publications; 1988.
22Hambleton R, Swaminathan H, Rogers HJ. Fundamentals of item response theory. Newbury Park, CA: Sage Press; 1991.
23Bryce Reeve, Ph.D. Applications of Item Response Theory (IRT) Modeling for Building and Evaluating Questionnaires Measuring Patient-Reported Outcomes. Available from:
24http://www.outcomes.cancer.gov/conference/irt/reeve.pdf [Last accessed on 2011 Jul 15].
25BB Reeve, P Fayers. Applying item response theory modeling for evaluating questionnaire item and scale properties. Available from:fds.oup.com/www.oup.com/pdf/13/9780198527695.pdf. [Last Accessed on 2011 May 25].
26Rasch Analysis RUMM2020 is the Trademark of RUMMLaboratory Pty Ltd.Available from: http://www.rasch-analysis.com/rasch-measurement.htm[Last accessed on 2011 Jul 15].
27Research methods knowledge base. Available from:http://www.socialresearchmethods.netLast Revised: 10/20/2006. [Last accessed on 2011 Mar 27]
28Downing SM. Validity: On the meaningful interpretation of assessment data. Med Educ 2003;37:830-7.
29Feldt LS, Brennan RL. Reliability. In: Linn RL, editor. Educational Measurement, 3rd Ed. New York: American Council on Education and Macmillan; 1989.
30CookDA, BeckmanTJ. Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application.Am J Med2006;119:166.e7-166.e16.
31Farlex. Huntington Valley, PA: Farlex; The Free Dictionary. Dictionary/Thesaurus >test-retest reliability [Web definition] Available on: http://medical-dictionary.thefreedictionary.com/test-retest+reliability. Accessed 20 Aug 2011
32Internal consistency reliability. Available from: http://www.experiment-resources.com/internal-consistency-reliability.html. [Last Accessed on 2011 Mar 30].
33Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297-334.
34Devellis RF. Scale Development Theory and Applications, London. Sage Publications, 1991. p. 24-33.
35Key terms, abbreviations, and definitions. Available from:http://outcomes.cancer.gov/conference/irt/key_terms.pdf. [Last Accessed on 2011 Mar 30].
36Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, RevickiDA, et al. ISPOR ePRO Task Force. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient reported outcome (PRO) measure: ISPOR ePRO Good Research Practices Task Force Report. Value Health 2009;12:419-29.
37Prescribing Information Climara Pro® (Estradiol/Levonorgestrel Transdermal System) Available from: http://dailymed.nlm.nih.gov/dailymed/drugInfo.cfm?id=10549#nlm34076-0. [Last Accessed on 2011 Jul 15].
38Kearney N, McCann L, Norrie J, Taylor L, Gray P, McGee-Lennon M, et al. Evaluation of a mobile phone-based, advanced symptom management system (ASyMS) in the management of chemotherapy-related toxicity. Support Care Cancer 2009;17:437-44.