Project Summary

Current lessons in Cognitive Tutors (built by Carnegie Mellon's PACT Center) on how to solve algebra equations require students to interact with an interface very different than that of solving equations on paper. Although some alternative interfaces have been explored, typically students use menus to perform manipulations such as adding a quantity to both sides of an equation and then type in the result of such manipulations using the keyboard. There are several weaknesses to this style of online tutoring, especially for mathematics.

1. Keyboard-and-mouse-based interfaces may impose extraneous cognitive load on students during problem-solving and distract them from the math concepts of import, by requiring them to learn and account for idiosyncracies of the type-in interface that are irrelevant to the learning event.
2. Typing mathematics on the computer generally involves some degree of linearization of the input (for instance, "2^x" instead of placing the x as a superscript. A more natural interaction experience using the handwriting modality can more directly support the two-dimensional math notations such as fractions and exponents that more constrained, traditional interfaces do not, allowing students to construct their solutions and express their understanding in an unconstrained way.
3. More generally, current standard interfaces for entering mathematical equations on computers are arguably limited and cumbersome. Mathematical notations have evolved to aid visual thinking and yet text-based interfaces relying on keyboard-and-mouse input do not take advantage of the natural two-dimensional aspects of mathematical equations.
4. Choosing solving steps from a menu rather than requiring the students to generate the next step independently can encourage non-learning-oriented behaviors such as "gaming the system" (see Baker et al's work).

Since Cognitive Tutors are not intended to replace teachers or traditional classroom instruction and tests, these lessons could be a weakness in the curriculum with respect to encouraging transfer from the Cognitive Tutor experience to the paper experience. Due to its similarities to paper-based mathematics, pen-based handwriting input may be faster, more efficient, and more preferable for entering mathematics on computers.

Preliminary results have shown that, in addition to more general usability gains for speed and user satisfaction [Anthony et al, 2005], students are able to solve more problems in the same amount of time, while achieving similar learning gains on the post-test [ref coming soon]. Work is ongoing to enhance the accuracy of the handwriting recognition used in the interface in order to enable step-by-step feedback during problem-solving. A more natural interaction experience could lead to decreased cognitive load, better performance, and improved usability in the general case, and learning gains and improved transfer to paper tests in the intelligent tutoring case. Further studies will explore these factors in detail.

Project Status and Findings

This project began in 2004. We have completed three formative user studies and created a prototype system. Major findings include:

1. General users entering calculus-level mathematics equations and expressions on the computer were faster in handwriting than typing (using Microsoft Equation Editor) by a factor of 3! This advantage increased as equation length and complexity increased. They also rated the handwriting modality more highly on a post-sessions Likert scale.
2. In the same study, users also input equations in a multimodal handwriting-plus-speech method, which ended up being faster and better liked than the keyboard-and-mouse method and was not much worse than handwriting alone. Also, users’ speech while writing differed from when speaking alone. Finally, user errors in handwriting and speech were non-overlapping, a fact which a multimodal recognition system could use to improve overall performance. [Anthony et al, 2006a]
3. Students learning or reviewing algebra equation solving who used a handwriting interface experienced similar learning gains as measured by improvement from pre-test to post-test than those who used a keyboard interface. However, the handwriting students finished the lesson in about 2/3 the time of the keyboard students. [ref coming soon]

Current project status: We are continuing to develop our prototype handwriting interface for an intelligent tutoring system lesson in algebra equation solving. Issues to address include enhancement of accuracy of the handwriting recognition engine in order to enable more detailed step-by-step feedback during problem-solving and development of an instructional paradigm to capitalize upon the strengths of handwriting recognition while accounting for its weaknesses. We plan future studies to collect further robust learning evidence comparing our handwriting interface prototypes to traditional intelligent tutoring systems.

Documents

Publications

Presentations

Posters

Selected References

Handwriting, Speech, and Mathematics

• Blostein, D. and Grbavec, A.: Recognition of Mathematical Notation. In Handbook on Optical Character Recognition and Document Analysis, Wang, P.S.P. and Bunke, H. (eds) (1996) 557-582.
• Brown, C.M.L.: Comparison of Typing and Handwriting in "Two-Finger" Typists. Proceedings of the Human Factors Society (1988) 381-385.
• Fateman, R.: How Can We Speak Math? http://www.cs.berkeley.edu/~fateman/papers/speakmath.pdf.
• Kanahori, T., Tabata, K., Cong, W., Tamari, F., and Suzuki, M.: On-Line Recognition of Mathematical Expressions Using Automatic Rewriting Method. Proceedings of the IEEE International Conference on Multimodal Interfaces (ICMI'00) (2000) 394-401.
• Matsakis, N.E.: Recognition of Handwritten Mathematical Expressions. Master's theses, Massachusetts Institute of Technology (1999) Cambridge, MA.
• Miller, E.G., Matsakis, N.E., and Viola, P.A.: Learning from One Example Through Shared Densities on Transforms. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'00) (2000) 464-471.
• Microsoft.: Microsoft Word User's Guide Version 6.0 (1993), Microsoft Press.
• Smithies, S., Novins, K., and Arvo, J.: Equation Entry and Editing via Handwriting and Gesture Recognition. Behaviour and Information Technology 20 (2001) 53-67.
• xThink.: Math Journal (2003) http://www.xthink.com/MathJournal.html.

Multimodal and Recognition Technologies

• Blum, A. and Mitchell, T.: Combining Labeled and Unlabeled Data with Co-Training. Proceedings of the Workshop on Computational Learning Theory (COLT'98) (1998) 92-100.
• Clark, H. and Brennan, S.: Grounding in Communication. In Perspective on Socially Shared Cognition (eds. Resnic, L.B., Levine, J., and Sag, S.). APA, Washington, D.C. (1991) 127-149.
• Oviatt, S.: Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. Proceedings of the CHI Conference (1999) 576-583.
• Oviatt, S., Coulston, R., and Lundsford, R.: When Do We Interact Multimodally? Cognitive Load and Multimodal Communication Patterns. Proceedings of IEEE International Conference on Multimodal Interfaces (ICMI'04) (2004).
• Oviatt, S., DeAngeli, A., and Kuhn, K.: Integration and Synchronization of Input Modes During Multimodal Human-Computer Interaction. Proceedings of the ACM Conference on Human Factors in Computing (CHI'97) (1997) 415-422.

Education, Cognitive Tutors, and Psychology

• Aleven, V.A.W.M.M. and Koedinger, K.R.: An Effective Metacognitive Strategy: Learning by Doing and Explaining with a Computer-Based Cognitive Tutor. Cognitive Science 26 (2002) 147-149.
• Anderson, J.R., Corbett, A.T., Koedinger, K.R., and Pelletier, R.: Cognitive Tutors: Lessons Learned. The Journal of the Learning Sciences 4 (1995) 167-207.
• Hausmann, R.G.M. and Chi, M.T.H.: Can a Computer Interface Support Self-explaining? Cognitive Technology 7 (2002) 4-14.
• Locke, J.L. and Fehr, F.S.: Subvocalization of Heard or Seen Words Prior to Spoken or Written Recall. American Journal of Psychology 85 (1972) 63-68.
• Sweller, J.: Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12 (1988) 257-285.

Contact

Main contact for this project is Lisa Anthony.
email address: lanthony [at] cs [dot] cmu [dot] edu
homepage: http://www.cs.cmu.edu/~lanthony
thesis proposal document: [coming soon]