Now that you’re gathered and analysed your data, you’ll probably want to share your findings with stakeholders. But can you be certain that they will believe what you’re saying? How can they trust your conclusions. Remember that your conclusion is informed by your analysis of the data, and your data is gathered through a variety of methods, which have been informed by your question and design. In other words, trustworthiness is not something you do later in the study…trustworthiness is something that is built throughout your study.
We can also think of the issue of trust with respect to credibility and quality. When your research is well-designed from start to finish, and is of a high quality, then the findings are credible and trustworthy. The process is like a circular loop that feeds back on itself. The quality of studies that use quantitative data collection methods can be informed by the following four questions (Shipman, 1988):
- If the study is carried out again by different researchers, using the same methods, will they find the same results? This is a measure of the reliability of the study and the reproducibility of the findings. In other words, if other researchers conduct the same study with the same (or similar) participants, will they reach the same conclusions?
- Does the evidence reflect the reality under investigation (note that this presumes an objective reality that exists independently of the researcher)? This is a measure of the (internal) validity of the study.
- What relevance do the results have beyond the situation under investigation? This is a measure of the ability to generalise the findings to a larger group than the study sample (external validity).
- Is there sufficient detail on the way the evidence was produced for the credibility of the research to be assessed? This is a measure of how well the study was designed, implemented, and reported.
Determining reliability
Reliable studies yield consistent results from repeated samples and different researchers over time. Differences in results come from differences between participants, not from inconsistencies in how the items are understood or how different observers interpret the responses. A standardised questionnaire is one that is written and administered so all participants are asked the precisely the same questions in an identical format and responses recorded in a uni- form manner. Standardising a measure increases its reliability.
One common way to test the reliability of an instrument is to ask a sample of subjects (different – but similar – to the main study sample) to complete the test on two different occasions. This is called test-retest reliability and is used to determine if the same instrument is given to the same people at different times, do they provide the same responses? Or, to put it another way, are their responses reliable?
Validity
Validity is a measure of whether or not the instrument measures what it claims to measure i.e. do results on specific questions that sample the vast domain predict or correlate with results at the larger goal or total domain?
Validity is about inference. What can we and can’t we infer from the results on the question? Does this specific question and the test results on it permit me to draw conclusions about some more general goal(s)? Do the answers to this specific question predict/correlate with performance at a more general goal? Before we can make a decision about the validity of a test we must first know something about the goal of the test.
Trustworthiness
Trustworthiness in qualitative research can be improved by paying attention to the following four concepts (Lincoln & Guba, 1988):
- Credibility is an indicator of how confident the researcher is that the findings are true and accurate, which can be enhanced through triangulation.
- Transferability is an indicator of how well the findings are applicable to other contexts (e.g. similar situations, similar populations, and similar phenomena). The researcher may use thick description to achieve this.
- Confirmability is the degree of neutrality in the research study’s findings (i.e. that the findings are not influenced by the researcher’s bias or personal motivations). This can be enhanced with an audit trail highlighting every step of the analysis of data.
- Dependability is the extent that the study could be reproduced with consistent findings. This may require an independent reviewer to examine the research process.
Figure above from Mays & Pope (2000).
Remember that even if you don’t intentionally set out to deceive your stakeholders and audience, it is still possible to lose their trust, and once trust is lost – especially in research – it is very difficult to retrieve.
References and resources
- Boynton, P. M. (2004). Selecting, designing, and developing your questionnaire. BMJ: British Medical Journal, 328(7451), 1312–1315.
- Guba, E. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. Educational Communication and Technology Journal 29, 75–91.
- Mays, N., & Pope, C. (2000). Assessing quality in qualitative research. BMJ : British Medical Journal, 320(7226), 50–52.
- Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22, 63–75.
