Original Article 1, Issue 2.3

A Brief Report on the Development of the Usability of Sleep Apnea Equipment–Oral Appliance (USE-OA) Questionnaire: A Pilot Study

http://dx.doi.org/10.15331/jdsm.4866

Constance H. Fung, MD, MSHS¹ ; Jennifer L. Martin, PhD¹ ; Robert Merrill, DDS² ; Cathy Alessi, MD<sup>1

1</sup> David Geffen School of Medicine at UCLA, Los Angeles, CA; ²UCLA School of Dentistry, Los Angeles, CA

ABSTRACT

Study Objectives:

To develop a questionnaire for measuring human factors (usability) associated with oral appliance use and to assess the feasibility of administering the questionnaire to patients with obstructive sleep apnea in a clinic setting.

Methods:

We developed the 11-item Usability of Sleep Apnea Equipment–Oral Appliance (USE-OA) Questionnaire by adapting items from a published questionnaire that had been developed to assess human factors associated with positive airway pressure device use. Then we distributed the USE-OA to patients at a university dental clinic between January and July 2014. We evaluated our survey methods qualitatively, calculated the response rate, and assessed completeness and response patterns of the USE-OA.

Results:

Our formative evaluation revealed that the questionnaire was easy to distribute and administer in a clinic setting. Our response rate was 23%. A majority of respondents gave favorable usability ratings, and a small number of respondents gave unfavorable usability ratings.

Conclusions:

The USE-OA questionnaire can be easily administered in a dental clinic setting. Additional studies conducted in highvolume sleep oral appliance clinics are needed to assess the psychometric properties of the USE-OA and to compare the results of the USE-OA to direct observation of patients getting their oral appliance ready for use and cleaning their oral appliance.

Keywords:

sleep apnea syndromes, orthodontic appliances, human engineering

Citation:

Fung CH, Martin JL, Merrill R, Alessi C. A brief report on the development of the Usability of Sleep Apnea Equipment– Oral Appliance (USE-OA) questionnaire: a pilot study. Journal of Dental Sleep Medicine 2015;2(3):67–70.
 

Oral appliances (OA) such as mandibular advancement devices are a common form of therapy for obstructive sleep apnea (OSA). Studies show that OAs can improve the severity of OSA (i.e., decrease apnea-hypopnea index) and symptoms of OSA such as excessive daytime sleepiness.¹ Many patients accept and adhere to OA therapy, but unfortunately, nonacceptance and nonadherence are also common.² Patients may reject therapy due to excessive salivation, oral lesions, malocclusion, or worsening of temporomandibular joint pain. ² Regular use of OAs requires behavioral modification—adoption of the therapy into the patient’s nightly sleep regimen. Similar to other health behaviors, factors such as lack of knowledge about OSA and OAs, negative attitude towards OAs, low expectations for treatment, lack of social support, and low selfconfidence for using OAs are possibly barriers to adherence. ³

Human factors (i.e., usability of equipment or other treatment methods) may also contribute to nonadherence to OSA therapies, including OAs. Human factors is “an applied science that takes research about human abilities, limitations, behaviors, and processes and uses this knowledge as a basis for the design of tools, products, and systems.” ⁴ These factors represent concepts such as ease of learning, ease of operating the therapeutic device, and ease of remembering how to use it, as well as patients’ overall satisfaction with the device-user interface.^5,6 “Applying human factors principles leads to designs that are safer, more acceptable, more comfortable, and more effective for accomplishing their given tasks.” ⁴

Few studies have assessed human factors in the context of OSA therapy. A recent pilot survey found that 8% of positive airway pressure device users strongly disagreed with the statement, “When I first got my current equipment, I easily learned how to get it ready for use.”⁷ In this same study, 20% of respondents reported difficulty getting their equipment ready for use in the past 30 days. ⁷ Most studies, however, have limited assessments of human factors to asking patients about overall satisfaction with the device.^9–12

In general, assessment of human factors may occur during either of the two stages of a medical device’s approval life cycle: pre-market and post-market. During the pre-market approval process, the Food and Drug Administration requires manufacturers to present human factors data. These data focus primarily on the impact of human factors on the safety and effectiveness of the device.⁵ Post-market assessments of human factors occur ad hoc (e.g., filing incident reports⁸ ), despite the potential broad impact of human factors on adherence to therapy and sales of the device. This may be due to the rudimentary infrastructure and lack of tools to date for conducting wide-scale human factors surveys. Survey instruments to conduct such research are not available. Only survey items to measure overall satisfaction and preference for OA therapy have been tested.^9–12

The purpose of this study was to develop a questionnaire to measure human factors associated with OA use among patients with OSA. This questionnaire may be useful clinically to assess human factors that contribute to nonadherence.

METHODS

USE-OA Questionnaire Development

To develop the USE-OA, we adapted the general usability items from the Usability of Sleep Apnea Equipment–Positive Airway Pressure (USE-PAP), which was developed with rigorous survey instrument methodology consisting of a literature review of human factors survey instruments, in-depth interviews with patients, a technical advisory panel, cognitive interviews, and a pilot survey among PAP users.⁷ Similar to the USE-PAP items, the USE-OA human factors items are aimed at measuring the following constructs: learnability, memorability, effectiveness (ease of patient getting the OA ready for use, not clinical effectiveness), efficiency, feedback from the OA that it is working, and overall satisfaction. For each of the 9 human factors items, respondents are asked to rate their agreement with each statement and are provided the following response options: strongly agree (5), agree (4), neither agree nor disagree (3), disagree (2), and strongly disagree (1). Two additional items were developed to assess the frequency of difficulty getting the OA ready for use and cleaning the OA in the past 30 days. For these items, the response options are the following: no days (4), some days (3), most days (2), everyday (1), or doesn’t apply to me (−6). The last option is provided for respondents who have not used their OA within the past 30 days or who never clean their OA. The questionnaire was revised iteratively (formatting, phrasing) prior to distributing it to clinic patients, based upon feedback from sleep clinical and research staff at our institution. The final USE-OA consisted of 11 items that assessed human factors associated with OAs.

Feasibility Testing

From January 2014 to July 2014, we conducted a feasibility survey at a university dental clinic that treats patients with OSA to test the USE-OA questionnaire items and our research methods for conducting a human factors survey. In addition to the USE-OA, we included in our survey instrument items to collect information about the participant’s age group, the brand name of the OA, the length of use of the OA, and previous attempts at other OSA treatments. We also included an open-ended item that asked participants to describe any experiences using current or past OSA treatments.

Patients who checked in to the clinic were provided a study information sheet, a survey cover sheet (which indicated that the survey would take 2 to 3 minutes to complete), the survey instrument, and a blank envelope. Patients who were interested in participating in the survey self-screened for eligibility by completing 2 items at the top of the questionnaire that confirmed that the participant was aged ≥ 18 years and was a current or previous sleep OA user. Patients who screened eligible completed the questionnaire and returned it in a sealed envelope to their dental provider. Patients who were ineligible or who opted out of participating had the option to return a blank questionnaire in the sealed envelope or not to return any envelope. The envelopes were given to their dental provider, who placed the envelope in a collection box, which was located in an area of the clinic accessible only to clinic staff. Responses were entered into a Microsoft Access 2010 database. The study was approved by the institutional review board at the University of California, Los Angeles.

Analyses

During the feasibility testing, we conducted a formative evaluation of the survey procedures. Descriptive statistics to characterize the completeness (missing values) of the questionnaires and variation in responses were summarized to describe central tendency (Microsoft Excel 2010). We also reviewed questionnaires for handwritten comments and responses, which may be present when questionnaire items are unsatisfactory to participants and can ultimately, inform survey revision.

RESULTS

Based on informal feedback from the dental clinic staff, the questionnaire was easy to distribute and administer in an office setting.

Ten completed surveys were available for review and analysis (response rate 23%). Participant characteristics are summarized in Table 1. Two participants (20%) indicated difficulty getting their OA ready for use (1 individual indicated “some days” and 1 individual “every day”). No participants responded that they have difficulty cleaning their OA. Table 2 provides the responses to the human factors items. Most participants gave favorable usability rating, either strongly agreeing or slightly agreeing with the human factors statements. However, one participant gave unfavorable ratings of the ease of getting the OA ready for use and of knowing when the OA is working properly. The questionnaire did not contain any comments written in the margins.
 

Table 1 Participant characteristics (n = 10). (more ...)

 

Table 2 Mean and median responses to USE-OA items (n = 10). (more ...)

 

DISCUSSION

To our knowledge, the USE-OA is the first questionnaire to broadly assess patient-reported human factors (usability) associated with OA use. This brief report describes the items in the questionnaire and the feasibility of administering it. We found that the USE-OA is easy to administer and because of its short length, has the potential to be distributed to a large number of OA users and may identify the subset of patients with usability challenges. As expected, most respondents gave favorable usability ratings regarding their OAs, but as is the case for PAP usability,⁷ not every respondent gives the highest ratings. Additional surveys conducted in high volume OA clinics are needed to formally assess the psychometric properties of the USE-OA.

As more evidence emerges on the effectiveness of various types of OAs for OSA treatment and more patients turn to OA for OSA treatment, post-market assessments of human factors associated with OA devices are needed. These assessments should be systematic, large-scale, quantifiable, and accessible, and results of these surveys should be available to patients and providers to enable them to identify devices that are most suitable for patients and to provide signals to manufacturers about the types of improvements that are needed to enhance usability. To achieve these assessments, questionnaires such as the USE-OA are needed. We found that the USE-OA could be quickly completed by patients in the waiting area of a clinic.

Comparisons of human factors between PAP and OA are also needed. Clinical trials have examined general satisfaction or general preference for OA versus PAP,² but the human factors literature suggests that other constructs are also important. ^6,13,14 Together, the USE-OA and the USE-PAP have the potential to achieve these comparisons, because the general items from the USE-OA align closely with the general items from the USE-PAP. In addition to the self-administered surveys, which are relatively inexpensive to conduct and can be used in a large population, direct observation of patients performing OA- and PAP-related tasks should be conducted to compare the usability of OAs and PAPs.

The main limitation of this study is the small sample size, which is common in feasibility studies. Patients at a university dental clinic may not be representative of other clinic populations—feasibility issues that were not detected in this study could arise in other samples. Inferences about human factors were not the aim of the study; rather, the study was performed to identify major flaws in the questionnaire and the survey methods and to inform a larger pilot study, which will assess the psychometric properties of the instrument and provide descriptive statistics on the usability of OAs.

In conclusion, we tested the feasibility of adapting items from an existing questionnaire that measures PAP usability to form the USE-OA. Additional testing is needed to assess the reliability and validity of this new instrument. The USE-OA assesses human factors-usability constructs that are commonly queried in other industries, and may prove useful in the evaluation of medical devices used in the treatment of OSA.

REFERENCES

ACKNOWLEDGMENTS

The authors offer special thanks to the clinic staff at the University of California, Los Angeles School of Dentistry.

SUBMISSION & CORRESPONDENCE INFORMATION

Submitted for publication April, 2015
Submitted in final revised form May, 2015
Accepted for publication May, 2015

Address correspondence to: Constance H. Fung, MD, MSHS, 16111 Plummer Street (11E), North Hills, CA 91343; Tel: (818) 891-7711 x36075; Fax: (818) 895-9519; Email: cfung@ucla.edu

DISCLOSURE STATEMENT

Study was funded by the American Sleep Medicine Foundation Physician Scientist Training Award (77-PA-12 to CHF) and National Institute on Aging of the National Institutes of Health under Award Number K23AG045937 (to CHF)/The Beeson Career Development in Aging Research Award Program (supported by NIA, AFAR, The John A. Hartford Foundation, and The Atlantic Philanthropies). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. No conflict of interest declared for any of the manuscript’s authors. This study did not involve any off-label or investigational use. All work was performed at UCLA School of Dentistry and David Geffen School of Medicine at UCLA. The authors have indicated no financial conflicts of interest.



PDF