Design and Psychometric Evaluation of American Sign Language Translations of Usability Questionnaires

This README webpage contains supplemental online information for the following article:

Larwan Berke, Matt Huenerfauth, and Kasmira Patel. 2019. Design and Psychometric Evaluation of American Sign Language Translations of Usability Questionnaires. ACM Transactions on Accessible Computing 12, 2, Article 6 (May 2019), 43 pages. https://doi.org/10.1145/3314205

For more information about our research at RIT, please visit the Center for Accessibility and Inclusion Research (CAIR) website.

ASL-SUS: The ASL Translation of the System Usability Scale (SUS)

The following videos are the "introduction message" and the ten "questions" from the ASL-SUS: The ASL Translation of the System Usability Scale. The original English SUS was created in this study: Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7.

Videos

The first video was played at the beginning of the ASL-SUS, it provides instructions and explains the Likert-type response scale. The remaining videos were played one at a time, and the participant marked their answer to each question on a paper answer sheet.

Paper Answer Sheet

The PDF file below was the paper answer sheet that participants used to mark their response to each question.

ASL-Adj: The ASL Translation of the Adjective Scale (Adj)

The following video and PDF were used in our study to present the Adjective Scale, which had been created in this study: Bangor, A., Miller., J., Kortum, P. 2009. Determining what individual SUS scores mean: adding an adjective rating scale. Journal of Usability Studies 4(3), 114-123.

Videos

This video explained the instructions and presented the single question from the Adjective Scale. The participant marked their answer on a paper answer sheet.

Paper Answer Sheet

The PDF file below was the paper answer sheet that participants used to mark their response to the Adjective Scale question.

ASL-NPS: The ASL Translation of the Net Promoter Score (NPS)

The following video and PDF represent the final version of the ASL translation of the NPS scale which was created in this study: Reichheld, Frederick F., 2003. The one number you need to grow. Harvard business review, 81(12), 46-55.

Videos

This video explained the instructions and presented the single question from the Net Promoter Scale. The participant marked their answer on a paper answer sheet.

Paper Answer Sheet

The PDF file below was the paper answer sheet that participants used to mark their response to the Net Promoter Scale question. Note: This answersheet was used in a study that also included the ASL SUS and the ASL Adjective Scale. So, the sheet begins with areas where the participant would mark their answer for those items, with the NPS area at the bottom of the page.

Technical Specifications of Videos

The video files linked above are MP4 files. These files can be played using VLC Media Player or other tools. The files have following technical specifications:

User Study Data

This section describes the raw data collected in study 2018-US-ASL and study 2018-US-ENG. Since the dataset would be too large to include in the print version, we have included it in the "Online Appendix" for this article in the ACM Digital Library.

Interested readers are encouraged to peruse the CSV files, and the list (below) describes the contents of the dataset. We want to remind readers that the US-ASL studies utilized the ASL versions of the questionnaires (ASL-SUS, ASL-Adj, and ASL-NPS), while US-ENG studies used the original English forms with no ASL instruction. Therefore, the TACCESS_Berke_dataset_ASL.csv file corresponds to study US-ASL while the TACCESS_Berke_dataset_Eng.csv file corresponds to study US-ENG.

Again, we want to remind readers that this dataset is an concatenation of our older dataset (studies 2017-US-ASL and 2017-US-ENG) that was published (the prior study labelled them #1 and #2, respectively) in the ASSETS '17 conference: Matt Huenerfauth, Kasmira Patel, and Larwan Berke. 2017. Design and Psychometric Evaluation of an American Sign Language Translation of the System Usability Scale. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17). ACM, New York, NY, USA, 175-184. DOI: 10.1145/3132525.3132540, along with data from additional participants in the new user studies (2018-US-ASL and 2018-US-ENG). For the reader's convenience, we have labeled the participants' IDs from the new studies at an offset relative to the older studies. For example: 1 corresponds to participant#1 from the old ASL study (2017-US-ASL), 81 corresponds to participant#1 from the new ASL study (2018-US-ASL), 41 corresponds to participant#1 from the old ENG study (2017-US-ENG), and 61 corresponds to participant#1 from the new ENG study (2018-US-ENG). You will also notice that all participants from the older studies (2017-US-ASL and 2017-US-ENG) evaluated only one artifact: the RIT website, and they were not asked to respond to the NPS instrument (thus, empty cells in the CSV files for these individuals).

The CSV files contain the following columns:

Because half of the individual items on SUS have negative polarity, the overall SUS score is calculated according to this formula: