IAT 432: Design Evaluation

Spring 2012 Course Schedule

Instructor: Allen Bevans (alb19@sfu.ca)
Teaching Assistsant: Andrew Milne (andrew_milne@sfu.ca)

Note: This course schedule is subject to change. Specifcs past Week 11 are still being updated.

week week week week week week week week week week week week week
Week 1
Topic
Required Reading and Resources

Tuesday, January 10

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, January 11

Lab D102
8:30 - 10:20 SUR3130

Lecture Slides

Course Outline

Lecture: Introduction to course

  • Review Academic honesty policy
  • Overview of course format
  • Overview of course work
  • Course textbooks
  • Overview of Evaluation
    • Evaluation Frameworks
    • Why evaluate?
    • Analytic vs. Participative evaluation methods
    • Objective vs. subjective evaluation methods
    • Formative vs. summative methods
    • Quantiative vs. qualitiative data

Lab:

 

 

Course Handout: Dix, A. et al. Human Computer Interaction, Pearson, London, England, 2004, Chapter 9: Evaluation

Lecture Reading: Start on Rubin, J. Handbook of Usability Testing, Part I.

Lab: When Good Things Happen to Bad Products - Jokela 2004

 

The two course textbooks are on hold in library (Burnaby and Surrey) and available at bookstore.

Note: January 11th (Wednesday) is the last day for add/drop via go.sfu.ca.

Week 2
Topic
Required Reading and Materials

Tuesday, January 17

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, January 18

Lab D102
8:30 - 10:20 SUR3130

Lecture: Heuristic Evaluation

  • Analytical Methods
  • History
  • Types
  • Pros and Cons
  • Process of applying
Lab: Team Assignment 1: Heuristic Evaluation (5%)
  • Introduction to Assignment 1
  • Team formation
  • Exercise: Applying heuristics and comparing evaluators

Lecture Reading: Read all five bulleted entries for Useit's Heuristic Evaluations.

Skim through Dix, Chapter 9: Evaluation.

Lab: Review Team Assignment 1 & review the Project Briefing document.

Reading for Assignment 1: Nielsen's Ten Usability Heuristics.

Reading for Assignment 1: AskTog, 2001: How to Deliver a Report Without Getting Lynched.

TA will email team lists for d101 and d102 for Assignment 1 before lab.

 

Week 3
Topic
Required Reading and Materials

Tuesday, January 24

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, January 25

Lab D102
8:30 - 10:20 SUR3130

 

 

Lecture : Usability Testing I

  • What is Usability?
  • Usability testing
  • Usability test plans
  • Issues, goals and audience
  • Team and roles
  • User profiles & tasks
  • Usability metrics, data & measurement.

Lab: Team Assignment 2: Usability Testing (15%)

  • Discussion based on individual lab preparation #1: Naive Usability
  • Introduction to Team Assignment 2
  • Exercise observational studies

Team Assignment 1 DUE Friday, January 27 at 2:30 pm in IAT 432 drop box.

Lecture Reading: Rubin, J. Handbook of Usability Testing, Part II.

Lab: Prepare Studio prep exercise #1: Naive Usability.

Lab: Review Team Assignment 2, the Client Briefing and the NDA.

TA will email team lists for d101, d102, d103 for Assignment 2 before lab.

Extra reading: Hornbaek, Current practice in measuring usability:
Challenges to usability studies and research
, Int. J. Human-Computer Studies 64 (2006) 79102.

Week 4
Topic
Required Reading and Materials

Tuesday, January 31

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, February 1

Lab D102
8:30 - 10:20 SUR3130

 

 

Lecture: Usability Testing II

  • Usability Planning cont'
  • 3 Methods: silent, think aloud, constructive interaction
  • Ethics of working with human subjects
  • Test setup
  • Test materials

Lab: Team Assignment 2 continued

  • Usability testing preparation
  • Preparing test tasks
  • Work on preparation tasks (e.g., timeline & scripts, note sheets)
  • Exercise on think aloud technique

 

Lecture Reading: Rubin, J. Handbook of Usability Testing, Part II -- continue.

Team Assignment 2 DUE Tuesday, Feb 21 (extended due to reading week) by 4:00 pm in the IAT 432 drop box.

Lab: Reading for Assignment 2: http://usability.gov/basics/

 

Week 5
Topic
Required Reading and Materials

Tuesday, February 7

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, February 8

Lab D102
8:30 - 10:20 SUR3130

 

 

Lecture: Usability Testing III: Query Methods

  • Query Methods: Asking Users through Interviews and Surveys
  • Data analysis
  • Report Writing 

Lab: Team Assignment 2 continued

  • Exercise on questionnaire vs. interviews
  • Discussion based on individual lab preparation #2: Writing useful reports.
  • Review of report structure for Usability Testing Assignment 2.
  • Time for Q&A Assignment 2.

Lecture Reading: Rubin, J. Handbook of Usability Testing, Part II.

Lecture Reading: Surveys -- Martin, D. Doing Psychology Experiments, Wadsworth, UK, 2004, Chapter 10 (pp. 215-226)

Extra Reading: AskTog, 2001: How to Deliver a Report Without Getting Lynched.

Lab: Prepare Studio prep #2: Writing useful reports.

Week NaN
Topic
Required Reading and Materials

Tuesday February 14

Reading Week

 

 

 

Week 6
Topic
Required Reading and Materials

Tuesday, February 21

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, February 22

Lab D102
8:30 - 10:20 SUR3130

Lecture: Controlled Lab Experiments

  • Scientific Method
  • Hypothesis testing explained
  • Independent & dependent variables

Lab: Team Assignment 3: Controlled Experiment (10%)

  • Team Dynamics Coaching
  • Introduction to Assignment 3
  • Introduction to tcl environment and tcl kb script.
  • Work through data collection with kb script.
  • New team meet & greet.

Lecture Reading: Martin, Chapters 1, 2, 7, 8

Lecture Reading: Dix et al. Chapter 9: Evaluation. Read Section 9.4.2 pp 329 340 Experimental Evaluation

Team Assignment 2 DUE Tuesday, Feb 21 (extended due to reading week) by 4:00 pm in the IAT 432 drop box.

Lab: Review Team Assignment 3. Download and install tcl environment. Download tcl script.

TA will email team lists for d101 and d102 for Assignment 3.

Team Assignment 3 DUE Friday, March 9 at 2:30 pm in IAT 432 drop box.

Week 7
Topic
Required Reading and Materials

Tuesday, February 28

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, February 29

Lab D102
8:30 - 10:20 SUR3130

 

Lecture: Reliability and Validity

Read Reliability & Validity in Martin.

Lab: Workshop on Lab Prep #3, review of data gathering for Assignment 3.

Lecture Reading: Martin, Chapters 7 , 8, 12, 13.

Extra Lecture Reading: Statistics Tutorial
7 common errors in Statistics.

Lab: Prepare Lab Prep #3: Controlled Experiment Design.

Team DATA SHEETS DUE by email to TA by Wednesday, February 29 at noon.

Whole class data sheet for Assignment 3 will be sent out by email Wednesday, February 29 in the evening.

Week 8
Topic
Required Reading and Materials

Tuesday, March 6

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, March 7

Lab D102
8:30 - 10:20 SUR3130

 

Lecture: Controlled Expts & Statistics

  • Statistics for designers
  • Descriptive statistics
  • Basic Inferential tests
  • Graphing basics

Lab: Go Lab Prep #4: statistics in-class exercise.

Complete Reading: Martin, Chapters 1,2 7 , 8, 12, 13.

Lab: Lab Prep 3 from last week AND

Lab Prep #4: Statistics. We will do this in class and it is due at the end of your workshop.

You will need the sample data file.

To complete Assignment 3, you will need the class data file. Note that the data needs to be ordered by keyboard type before analysis is done.

Team Assignment 3 DUE Friday, March 9 at 2:30 pm in IAT 432 drop box.

Week 9
Topic
Required Reading

Tuesday, March 13

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, March 14

Lab D102
8:30 - 10:20 SUR3130

 

Lecture: Field Studies, Emotion & Affect (Assignment 4)

Review Possible Exam Questions.

Labs: Team Assignment 4 Field Evaluation: Fulll-body interaction (15%)

  • IMI survey scoring exercise
  • Discussion of lab prep exercise #5: Enjoyment/Engagement

Lecture Reading: Martin pp 14-19 on Qualitative Designs; Rubin Chapter 14; Dix et al. Chapter 9 section on field studies. Bently et al. Evaluation using cued-recall debrief to elicit information about user's affective experiences, OzCHI, 2005. (Focus on Cued Recall Debrief Method).

Extra Reading: Jarrett, Caroline. Better Reports: How to Communicate the Results of Usability Testing.

Lab: Review Team Assignment 4

Lab: Prepare Lab Prep #5: Enjoyment/Engagement.

TA will ask you to form teams for Assignment 4.

Final exam questions given out.

Team Assignment 4 DUE Friday, March 30 at 2:30 pm in IAT 432 drop box.

Week 10
Topic
Required Reading

Tuesday, March 20

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, March 21

Lab D102
8:30 - 10:20 SUR3130

 

Lecture: Interface Critisism and Aesthetics

Course Evaluations.

Labs: Team Assignment 4 Field Evaluation: Nintendo Wii (15%)

  • Cued Recall Debrief introduction
  • Coding exercise

 

Lecture & lab Reading: Read Bently et al. Evaluation using cued-recall debrief to elicit information about user's affective experiences, OzCHI, 2005. (Focus on Cued Recall Debrief Method).

Inter-rater reliability example for Lab Coding Exercise and Assignment 4.

Team Assignment 4 DUE Friday, March 30 at 2:30 pm in IAT 432 drop box.

Week 11
Topic
Required Reading

Tuesday, March 27

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, March 28

Lab D102
8:30 - 10:20 SUR3130

Lecture: Usability Can Be Harmful

 

Labs

  • Cancelled due to illness

Lecture Reading: Greenberg & Buxton, Usability Evaluation Considered Harmful. CHI 2008, ACM press (2008), 111-120.

Lab: Review Team Assignment 5.

TA will ask students to choose a partner for Assignment 5.

Team Assignment 4 DUE Friday, March 30 at 2:30 pm in IAT 432 drop box.

Team Assignment 5 DUE Wednesday, April 11 by 2:30 pm in the IAT 432 drop box.

Week 12
Topic
Required Reading

Tuesday, April 3

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, April 4

Lab D102
8:30 - 10:20 SUR3130

 

Lecture: Physiological Measures

Vernonica Zammitto, EA

Lab:

  • Team Assignment 5: Usability in Context(5%)
  • Usability in context exercises.

Lab Reading: Greenberg & Buxton, Usability Evaluation Considered Harmful. CHI 2008, ACM press (2008), 111-120.

Team Assignment 5 DUE Wednesday, April 11 by 2:30 pm in the IAT 432 drop box.

Week 13
Topic
Required Reading

Tuesday, April 10

Lecture D100
4:30 - 5:20 SUR2740

Lab D101
6:30 - 8:20 SUR3130

Wednesday, April 11

Lab D102
8:30 - 10:20 SUR3130

 

Lecture: Review of possible exam questions.

Lab: No lab. You shoud use this time to study the exam questions with classmates you get along with. Probably at a pub,

 

Lab Reading: Dix Chap 9 Matrix of methods

Team Assignment 5 DUE Wednesday, April 11 by 2:30 pm in the IAT 432 drop box.

Exam
   

Wednesday, April 18

 

 

Exam D100 Wednesday, April 18, 19:00 - 22:00

 

Hour Offices by appointment this week.

 

 

FINAL EXAM: Closed Book

Room: SUR 5240.

 

Bring your student ID to the exam for sign in/sign out.

Students without ID will not be allowed to take the exam.

Instructor: Allen Bevans Teaching Assistant: Andrew Milne