My advisees are what makes my job worth all the sweat and sleepless nights...
What ties together everything that excites me most in research from a purely intellectual curiousity standpoint is a focus on conversational interactions. As a Computational linguistics student, I was always most interested in the areas of sociolinguistics, discourse analysis, and pragmatics. However, stretching back even into my undergrad days, I have always had a fascination with artificial intelligence and machine learning. As a faculty member here, I am strongly interdisciplinary. I'm interested in conversation from multiple perspectives, including linguistics (both core linguistics and computational linguistics, including machine learning approaches), social sciences, cognitive science, education, and especially the connections between these fields. I'm not satisfied with a focus on what seems interesting to me. Instead, it is my strong desire to see my research make an impact in the world, and my chosen area of impact is mainly the area of education. Within the field of language technologies, I most strongly align these days with work in dialogue systems, sentiment analysis and summarization, especially conversation summarization. I have a growing interest in the emerging work related to computational models of entailment. Within the field of human-computer interaction, I identify most with the area of computer-supported collaborative learning, especially where the language technologies my students and others are developing can have an impact on conversational interactions in learning contexts (and are doing so!), and ultimately assist people in learning more during these interactions. Recently I have become more and more interested in work related to information access for low literacy populations. And I'm excited about new international partnerships I have recently formed in India and other places.
The driving question behind my research is how to develop technology capable of both shaping conversation and supporting conversation to achieve a positive impact on human learning. If technology is to be maximally successful in this mission, two things must be true. First, the technology must be capable of processing, generating, and participating in conversation. And second, it must do so with insight. In other words, its behavior should be designed with an understanding of what properties of conversation add to or detract from its positive impact. Furthermore, its design should be based on knowledge of what stimuli manipulate these properties and in what ways. Thus, this question is both fundamentally a language technologies and a human-computer interaction question.
Conversation is the cornerstone of my research because of its pivotal role in learning. Conversation builds identification with a group and commitment to a group. Conversation facilitates collaboration. Through conversation, communities offer their members a channel through which they can learn from one another and support one another. Through the offering of social support, conversation can even lead to health benefits. When students exchange and build on one another’s ideas, conversation may facilitate conceptual change. Even explaining to oneself can reveal knowledge gaps and stimulate a search for new knowledge and new understanding. Innovation emerges from the productive exchange and mingling of alternative viewpoints. However, we know from the social psychology of group work that conversation may also result in negative effects referred to as process losses. Dysfunctional communication patterns can harm relationships and hinder the effective exchange of perspectives. Success in my research can be measured in terms of how successfully the technology I create can increase the positive effects of conversation while decreasing the negative ones.
My recent work demonstrates that one answer to my question can be found in the design, development, and evaluation of a new form of collaborative learning support that advances the state-of-the-art in computer supported collaborative learning (CSCL). The state-of-the-art in CSCL consists of static forms of support, such as structured interfaces, prompts, and assignment of students to scripted roles, all of which typically treat students in a one-size-fits-all fashion. In contrast, this new form of support “listens in” on student conversations in search of important events that present opportunities for discouraging negative behavior or encouraging positive behavior using a form of text classification I refer to as automatic collaborative learning process analysis. Interactive support agents that can participate with students in the collaborative discussion are then triggered as a way of interactively offering support. Evaluations of this technology demonstrate its pedagogical effectiveness (Kumar et al., submitted; Kumar et al., 2007; Wang et al., 2007).
Leading up to this recent work, I have made contributions over the past decade and a half both to the field of language technologies and human-computer interaction. First, in connection with the language technologies aspects of my driving question, my main research contributions have been:
Early evaluations of tutorial dialogue technology showed promise but revealed that more insight was needed to be able to wield that technology to a successful end. While the technical feasibility of conducting tutorial dialogue interactions with students was clearly demonstrated, the same studies raised serious questions when even human tutoring was not always significantly more effective than carefully crafted expository text (VanLehn et al., 2006). Thus, in connection with the human-computer interaction aspects of my driving question, I have conducted a series of 13 experimental studies over the past four years to investigate more deeply the connection between conversation and learning in the domains of engineering design, calculus, middle school math, earth sciences, thermodynamics, and physics with students at the middle school, high school, and college levels, both with students in the US and in Taiwan. These studies have yielded new insights into
One important lesson learned from this work has been the observation that effects of certain manipulations can vary over time (Wang & Rosé, 2007). For example, in an analysis of idea generation behavior comparing students with the support of a feedback agent with students who did not have that support, the effect of a support agent intervening in the conversation had a negative impact on productivity during the first five minutes of the 30 minute interactions where students produced the most intensive idea generation on their own (effect size 1 standard deviation). The pattern was different during the remainder of the interaction where the productivity of students was in decline. In that case, the effect of the feedback agents on productivity was positive (effect size 1.4 standard deviations). This finding underlines the importance of dynamic support agents that can behave in a way that is sensitive to what is happening in a conversation and can adjust their behavior over time in response to changes in student behavior. Another important lesson learned from this series of studies was that tutorial dialogue has its greatest value when students are in danger of missing an opportunity to learn by failing to stop and reflect or failing to find the most important material to focus on. This explains the great success of tutorial dialogue with simulation based learning in my work (Kumar et al., 2006; Kumar et al., 2007), in contrast to the earlier results with human tutors not out-performing carefully crafted text under circumstances where this was not the case (VanLehn et al., 2007). It also gives insight into when in an interaction support agents are needed to intervene.
Insights gained from the experimental studies I have run can have an impact on student learning when they are implemented in actual collaborative learning environments that students use. This would not be possible without the language technologies work that I have been involved in for the past fifteen years. A consistent thread that has run throughout this research is a search for characteristics of text that have predictive value in connection with an interpretation goal and can be identified reliably. This approach has been manifest in the wide variety of language technologies work I have done over the past decade and a half, whether in the context of symbolic language processing approaches of the early 90s or machine learning oriented approaches of today.
When I began my graduate work in 1992, the state-of-the-art in language technologies was viewed as too brittle to play the transformative and supportive role that I envisioned. The biggest road blocks were the amount of time required to author necessary knowledge sources, the impossibility of creating a rule set that could fully cover language the way people use it conversationally, and the computationally intensity of interpretation algorithms. In my dissertation research and in my work immediately following, I challenged these limitations at two levels with a single, simple idea that lead to increases both in robustness and efficiency: use the most reliable available partial information to eliminate as many interpretations as possible, thereby increasing the likelihood of piecing together an interpretation that maximizes completeness and correctness. At the sentence level, this idea was embodied in a two stage interpretation process consisting of a partial parsing stage using the LCFLex robust parser to construct partial analyses from grammatical islands of text (Rosé, 2000; Rosé & Lavie, 2001; Lavie & Rosé, 2002; Rosé et al., 2002) and a recombination stage using genetic programming to create partial interpretations (Rosé, 2000; Rosé & Levin, 1998). The LCFlex robust parser increased the efficiency by two orders of magnitude compared to its predecessor. Furthermore, splitting the interpretation process into two stages increased efficiency by roughly a factor of two. A similar approach was equally successful at the discourse level. In that work, I created an algorithm for computing the structure of a running dialogue relying mainly on computed relationships between temporal expressions. This line of inquiry lead to the first plan-based discourse processor that was robust enough to process spontaneously generated dialogues, albeit in a limited domain (Rosé et al., 1995; Qu et al., 1997).
After completing my dissertation, I saw the potential of using language technologies in a transformative role in the field of education in the form of tutorial dialogue technology (Rosé et al., 1999; Freedman et al., 2000; Jordan et al., 2001; VanLehn et al., 2002). Early attempts had failed due to limitations of language technologies to support natural language interaction, but the state-of-the-art in language technologies of the late 90s was robust enough to meet the challenge. My first evaluation of a tutorial dialogue system, the help agent in the Andes physics problem solving environment, demonstrated a significant learning advantage to problem solving with the support of tutorial dialogue agents in comparison with problem solving with the support of hints (Rosé et al., 2001). In order to push the envelope of what was possible in conversational interactions between computer agents and students, I worked to develop an approach for processing short student essays (Rosé & VanLehn, 2005), which improved accuracy by including features extracted from a detailed parse of well formed fragments from the sentences along with word level features. Through a series of 8 studies in physics problem solving and qualitative physics, the technical feasibility of conducting tutorial dialogue interactions with students was well demonstrated, even when those interactions included extended student explanations (Rosé & VanLehn, 2005), and even in speech (Litman et al., 2006).
In my recent work, my long term involvement in development of technology for processing conversation has grown into work on automatic collaborative learning process analysis (Rosé et al., under review; Rosé et al., 2007; Joshi & Rosé, submitted; Wang, Joshi, & Rosé, 2007; Wang et al., 2007b; McLaren et al., 2007; Donmez et al., 2005). The goal here is to be able to construct a model of the collaborative processes that are visible in a conversation between collaborative learners. In the TagHelper project, we have developed a collection of text classification techniques that are effective for processing an on-going collaborative learning discussion either as it is happening, or off-line, for the purpose of detecting important conversational events that indicate the quality and instructional value of the interaction. These investigations have revealed new challenges for text classification research specifically and machine learning more generally. In my work with English, German, and Chinese corpus data, I have directly addressed some of these challenges related to algorithms for increasing reliability on data sets with highly skewed class distributions (Donmez et al., 2005; Donmez et al., submitted), with data sets where class distinctions are subtle and may rely to some extent on the surrounding context for correct interpretation (Rosé et al., under review) and data sets that are limited in size (Arguello et al., 2006; Wang et al., 2007d). Other work is in progress related to avoiding over-fitting to idiosyncratic habits of particular learners due to the non-independence of multiple data points extracted from the same conversation in a relatively small set of conversations. One key to success in this work has remained the search for meaningful features of text that can be extracted reliably and efficiently.
Collaborative learning process analysis has significance in the broader language technologies community in that it is a supporting technology for the emerging area of conversation summarization. Furthermore, this technology enables a different form of dynamic support for collaborative learning conversations: making it possible to alert an instructor when an event occurs that requires the instructor’s attention. Conversation summarization holds the potential to support instructors or group facilitators by distilling from a massive amount of communication data, an indication of the location within that stream of instances of communication that are of particular interest. For example, in an NSF-funded project related to project based learning where I am a Co-PI, we have built a prototype conversation summarization system that processes conversational data posted to a groupware system that is used to coordinate group work in an engineering design project course (Rosé et al., 2007). The system processes a week’s worth of posts per student at a time in order to assign a prediction about how productively that student has contributed to his or her group that week. Using this predictive model, an instructor can view a student’s productivity trajectory over time and identify students who show signs of sub-standard productivity. An interesting finding from this work is that the conversational features that were most predictive were social in nature, such as greetings and thank-yous, which underlines the importance of relationship building through conversation in group work.
What is strikingly different about conversation summarization in contrast to summarization of expository text is that the summary may include more than just a reduced version of the content communicated during a conversation. It may also include notable features related to the style and structure of the conversation. In a separate effort in collaboration with research collaborators in Tuebingen (now in Munich), I have worked to automate a collaborative learning process analysis that assesses the quality and character of a collaborative discussion at multiple levels of abstraction. In that work, I have compared the use of state-of-the-art sequential learning techniques with a novel feature-based approach, which reflects the structure of the interaction at multiple levels. This evaluation demonstrates for three separate dimensions of a context oriented annotation scheme that these novel thread based features have a greater and more consistent impact on classification performance on this data set than state-of-the-art sequential learning techniques. (Wang et al., 2007b).
The language technologies and human-computer interaction threads of my research have come together in the past year to produce the dynamic form of collaborative learning support involving dialogue agents, which was introduced earlier in this statement. Two successful evaluations of this new technology in the past year have demonstrated that students learn more from their collaborative learning interactions when this form of support is present than when it is absent (Kumar et al., 2007; Wang et al., 2007). For example, in a simulation based learning study in the domain of thermodynamics study (Kumar et al., 2007), the finding was that in a simulation based learning task, students learned significantly more when they worked in pairs than when they worked alone. Furthermore, while intelligent tutoring style hints reduced learning as evidenced in an earlier study (Kumar et al., 2006), dynamic support implemented with tutorial dialogue agents lead to significantly more learning than no support, while static support consistent with the state-of-the-art in computer supported collaborative learning was not statistically distinguishable from the no support condition. The largest effect size in comparison with the control condition where students worked alone with no support was a condition where students worked together in pairs with the support of the dynamic support agent, with an effect size of 1.24 standard deviations. The most important finding was that because the effect size achieved by combining the two treatments was greater than that of either of the two treatments alone, we conjecture that each of these factors are contributing something different to student learning rather than being redundant.
Moving forward, as I continue to iteratively refine the design of dynamic collaborative learning support through insights gained from experimental studies, I will also work to increase the fidelity of the model of the collaborative process that can be constructed automatically from collaborative learning conversations. Up until now my research developing this dynamic form of collaborative learning support has been evaluated in the context of short-term lab and classroom studies. As I am moving into distance education in my own teaching, I am also beginning to do research related to using these support technologies in a distance education environment. Ultimately, my long term vision is for this technology to operate continuously over semesters or years in a distance education environment in order to make free educational resources such as CMU’s Open Learning Initiative or MIT’s OpenCourseware Initiative more inviting for students and more effective for providing students with continuous, community based learning support. Through a new NSF-funded collaborative grant with Gerry Stahl at Drexel university, I am taking advantage of the opportunity to increase the potential impact of this technology by integrating it with his virtual math teams on-line learning environment, which is housed in the Math Forum service that reaches millions of kids each month.
Other opportunities for increasing the impact of my work are on the horizon. I am working to build an industrial affiliate relationship with a large textbook publishing company to develop and distribute interactive educational materials with their text books as well as on-line assessment materials to distribute to instructors, leveraging text classification and tutorial dialogue technologies. To further increase the impact of the TagHelper tools work, I have made it publicly available to support related research in other labs at Carnegie Mellon and elsewhere. I will continue to offer training for using these tools through the annual Pittsburgh Science of Learning center summer school as well as in tutorials and workshops both locally and at international conferences.
In summary, my research thus far has benefited from intense involvement both in the language technologies community and in the human-computer interaction community. Because what drives my research is the goal of developing technology capable of both shaping conversation and supporting conversation to achieve a positive impact on human learning, my long term plan is to remain active in both of these communities. Only through an intense integration of these two disciplines is it possible for technology work to be guided by a deep understanding of what is needed for impact. Furthermore, only with a deep understanding of what is and is not possible with technology can experimental work be focused on questions that are most likely to lead to an important technological advance. Thus, only through a continued synergy between fields can my vision be fully realized.
What fascinates me most about studying the role of conversation in learning is that new ideas may be created when exchanging alternative viewpoints. The new ideas that emerge through conversation may draw from the differing perspectives of the participants but nevertheless be distinct from the ideas that existed in any of their minds prior to the interaction. The research literature on group learning provides strong evidence that the success of such interactions between students depends upon the ability of the instructor to facilitate this process. The instructor creates opportunities for learning by meeting the students on their own path and offering the support necessary to draw out the students’ differing perspectives and ideas. In the midst of this conversation, the instructor is well situated to present the content of the course in a way that is seen by students as relevant to meeting their own goals. In creating an environment where students see their involvement in a course as a means to move forward on their own path, the instructor has the opportunity to play the role of a mentor who comes along side students to offer experience and wisdom and to help them navigate the maze that is before them. That investment of the instructor in individual students yields the greatest increase when it is internalized by the students and then brought back into small group activities and the whole group discussion. Thus, my philosophy of teaching is to strive for a personal connection through conversation with and between students.
An essential ingredient in this learning conversation is the differing perspectives of the participants who are involved. The School of Computer Science at Carnegie Mellon is made up of distinct, tight knit communities of specialization that are situated in such a way as to provide many opportunities for exchanging views. This is an ideal environment in which this philosophy of teaching can flourish. Thus, in my position with appointments in both the Language Technologies Institute and the Human-Computer Interaction Institute, I have taken advantage of the opportunity to create and teach three bridge courses designed to promote understanding and strengthen interactions between departments and to keep the conversation active.
More information about the courses I teach can be found on my teaching page.
In conclusion, just as my research interests in supporting and shaping learning through collaborative conversation informs my teaching, my teaching also informs my research. My conversations with students and observations of their interactions with each other in my courses give me insight into their learning processes, which I can then take back into my research.