1. Introduction

The amount of on-line grammar exercises has been increasing steadily since the inception of the World Wide Web. Over the past years, a number of useful authoring tools have been developed that allow instructors to design Web-based student tasks quickly and without extensive computer programming knowledge.

Commonly, publishers of language learning material also provide Web sites that contain vocabulary and grammar practice for second language learners. Mainstream course offerings on the Web, however, often resemble the traditional workbook exercises from which they were adapted, making little use, for instance, of the multimedia capabilities of the Web. For example, a commonly used course package for introductory German⁽¹⁾ contains a companion Web site with vocabulary and grammar exercises for each of the 12 chapters of the textbook. None of these exercises on the Web site, however, contain any graphics, sound or videos which would enhance the static material character of workbooks.

From a pedagogical perspective, the definition of acceptable answers to exercises is also highly constrained. For instance, for the exercises mentioned above the system does not accept the reasonable answer provided in (1b) for the task given in (1a). Instead (1c) is the only possible answer from a system's point of view.

(1a) Heute ist der 10. November.                        Today is November 10.
Den Wievielten haben wir morgen?                    What date is it tomorrow?
Morgen haben wir _____________________     Tomorrow is ___________________
(1b) den 11. November.                                       November 11.
(1c) den Elften.                                                     The eleventh.

(2a) Erwin arbeitet in Leipzig, aber seine Familie wohnt in Bad Harzburg. Am Wochenende fährt er nach Hause. Erwin fährt mit dem ____________. …
Erwin works in Leipzig but his family lives in Bad Harzburg. On the weekend he drives back home. Erwin takes his _____________. …
(2b) Wagen.                                                            car.
(2c) Auto.                                                                car.

Finally, error feedback commonly does not address the source of an error. For instance, for grammar exercise 1a) the system displays the correct answer without any explanation of the student's mistake. For vocabulary exercises, the student is referred to the corresponding page in the textbook which displays the word in question in a word list. In addition to the pedagogical limitations, the student has to consult the textbook which is an unnecessary inconvenience given the potential of the Web. Similarly, for the student answer given in (2b), the system responds with "Einige Ihrer Antworten sind nicht richtig oder fehlen. Bitte versuchen Sie's nochmal." ("Some of your answers are incorrect or are missing. Please try again."). In this case, in addition to rejecting a valid answer the system's feedback is a generic, catch-all response.

The present paper discusses building a more flexible Web-based grammar practice environment around an Intelligent Language Tutoring System (ILTS). While ILTSs employ Natural Language Processing (NLP) and thus require programming and linguistic expertise, they provide error-specific feedback and flexibility in handling student answers. Sound, graphics and/or videos can also be implemented to achieve a more varied, authentic and contextualized learning environment.

In the following, we discuss the pedagogical underpinnings and computational requirements of systems which employ NLP in their error checking mechanism. We introduce the German Tutor, an ILTS for German, and discuss the exercise types currently available in the system. We further describe a system trial conducted in the spring semester 2000. The study identifies an error typology of 1,906 ungrammatical sentences submitted by 33 students of two introductory German classes. We conclude with a brief discussion of the research capabilities of our ILTS.
 

2. Intelligent Language Tutoring Systems

In comparing different error checking mechanisms and levels of feedback specificity, Garrett [1987] describes four kinds of systems (see also James, 1998):

1. systems which present only the correct answer,
2. systems which pinpoint the location of an error on the basis of the computer's letter-by letter comparison of the student's input with the machine-stored correct version,
3. systems which base their error analysis on anticipated wrong answers (error messages associated with possible errors are stored in the computer and are presented if the student's response matches those possible errors), and
4. systems which use Natural Language Processing (NLP) and provide a linguistic analysis of the student's response.

The pedagogical goal behind an ILTS is to provide error-specific feedback. For example, if a student chooses an incorrect article in German the error might be due to incorrect inflection for gender, number, or case. In such an instance and for cognitive learning to occur it is desirable to distinguish between the three error types (Rumelhart & Norman, 1975; Venezky & Osin, 1991). The error-checking mechanisms described in (1) - (3) cannot achieve this goal because they are based on a letter-for-letter comparison against an answer key. However, one obviously cannot enter the arbitrarily numerous sentences required for meaningful practice into memory for purposes of comparison. NLP provides the analytical complexity underpinning an ILTS.

Over the past decade, a number of NLP systems have been implemented (Labrie & Singh, 1991; Levin & Evans, 1995; Loritz, 1995; Hagen, 1994; Holland et. al., 1995; Sanders, 1991; Schwind, 1995; Wang & Garigliano, 1992; Yang & Akahori, 1997, 1999; Heift & Nicholson, 2000). Additionally, a number of studies have focused on comparisons of CALL programs. For example, Nagata (1993, 1995, 1996) compared the effectiveness of error-specific, or metalinguistic, vs. traditional feedback with students learning Japanese. In all studies, Nagata found that intelligent computer feedback that is based on Natural Language Processing -- and thus can explain the source of an error -- is more effective than traditional feedback (see also Yang and Akahori, 1997, 1999; Brandl, 1995). Heift (2001, 2002) found that students attend predominantly to error-specific feedback and correct their errors rather than relying on system help options.

    2.1. An Intelligent Language Tutoring System for German

The goal of the German Tutor is to provide meaningful and interactive vocabulary and grammar practice for learners of German. Meaningful tasks and interactivity require intelligence on the part of the computer program. Unlike existing course-support systems which use simpler grammar practice and feedback mechanisms, the German Tutor emulates two significant aspects of a student-teacher interaction: it provides error-specific feedback and it allows for individualization of the learning process (Heift & Nicholson, 2000). In example (3) the student provided an incorrect German sentence:

(3a) *Familie Braun sind in den Urlaub gefahren.
(3b)  Familie Braun ist in den Urlaub gefahren.
(3c)  Das Subjekt und das Verb stimmen nicht überein.
        There is an error in subject-verb agreement.

There are three learner levels considered in the system: beginner, intermediate, and advanced. For example (3) given above, the beginner will receive the most detailed feedback:

"FAMILIE und SIND passen nicht zusammen. FAMILIE ist singular.
(FAMILY and ARE do not agree. FAMILY is singular.)",

1. an NLP component which can analyze ill-formed sentences, and
2. a Student Model which keeps a record of the learner's past performance.

The Student Model is a representation of the current skill level of the student across different grammatical constructs and vocabulary. For instance, the grammar nodes contain detailed information on the student's performance on subject-verb agreement, case assignment, prepositions, etc. The score for each node increases and decreases depending on the grammar's analysis of the student's performance. The amount by which the score of each node is adjusted is specified in a master file and may be weighted to reflect different pedagogical purposes.

The Student Model has two main functions:

First, the current state of the Student Model determines the specificity of the feedback message displayed to the student. A feedback message is selected according to the current score at a particular node. A student might be advanced with regard to vocabulary but a beginner with passive voice constructions. Hence, a feedback message about vocabulary would be less detailed than a feedback message about passive-voice constructions.

Second, the system provides remedial exercises depending on the current state of the Student Model. For example, if a student is rated as a beginner with respect to dative case, then the additional exercises will focus on this particular construction. A student, however, who is at the advanced level with the grammatical constructions covered up to this point will not receive any remedial exercises at all.

         2.1.1. Exercise Types

The German Tutor is the grammar component of Pilot (Roche et.al, forthcoming), a comprehensive Web-based learning environment for German for specific purposes. Currently, there are six exercise types implemented in the system: Dictation, Build a Phrase, Which Word is Different, Word Order Practice, Fill-in-the-Blank, and Build a Sentence. They all differ with respect to the exercise focus and task, and with respect to their use of media and the NLP capabilities of the German Tutor.

Dictation

The exercise type given in Figure 1 displays a dictation task which focuses on listening comprehension and spelling. Students can first listen to the entire dictation by clicking the "Diktat" (dictation) button, or they can listen to each individual sentence by accessing the "Satz" (sentence) button. Once they type in a sentence and it is correct, it will appear above the input box. For instance, the dictation given in Figure 1 consists of two parts (Satz 2 von 2). The student correctly typed the first part (Guten Tag! Mein Name ist Fumiko Kanno) which is displayed above the input box. The student now proceeds to the next part of the dictation.
 
 

In the event of an error, students have a number of additional options which are consistent for all exercise types. The student can either correct the error and resubmit the sentence by clicking the "Prüfen" (check) button, or peek at the correct answer(s) with the "Lösung" (answer) button, and/or go on to the next exercise with the "Weiter" (next) button. If the student chooses to correct the sentence it will be checked for further errors. The iterative correction process continues until the sentence is correct or the student decides to peek at the correct answer(s).

Finally, in the case of multiple errors, the system prioritizes student errors and displays one message at a time so as not to overwhelm the student with excessive error reports. Previous studies (van der Linden, 1993) have found that lengthy error messages tend to distract the student from the task. Error prioritization follows pedagogical principles by considering the salience of an error and/or the focus of a particular exercise (Heift & McFetridge, 1999).

Build a Phrase

In the Build-a-Phrase exercise type students practice vocabulary and grammar with a focus on individual phrases rather than complete sentences. For example, in Figure 2 the student is asked to provide a noun phrase which corresponds to the picture provided. The system accepts alternative answers such as the singular das Croissant or the plural die Croissants.
 
 

Which Word is Different

The exercise type given in Figure 3 displays a number of words all except one of which belong to the same category. The student task is to identify the one that differs from the others. The divergent word may differ syntactically, semantically or pragmatically from the remaining words.
 
 

Word Order Practice

In this exercise type, students practice German word order with a 'drag and drop' task: words have to be arranged in an appropriate order to form a grammatical German sentence. Figure 4 displays a task in which some of the words provided have already been dragged into the input box. Given the flexibility of German word order, there is commonly more than one acceptable answer.
 
 

Fill-in-the-Blank

The student's task here is to complete sentences by filling in any blanks that appear in the example. For instance, in Figure 5 we display an example task with one blank. For a higher skill level and to make the task more challenging, more than one blank can be contained in the sentence.
 
 

Build a Sentence

Finally, in this exercise type students are provided with a set of words. Their task is to create a grammatical German sentence using all the provided words. As mentioned, the system modulates feedback messages to reflect the current state of the learner's expertise. For instance, Figure 6 displays the feedback for an advanced learner ("There is an error in gender with the subject"). At the advanced level, students receive less elaborate feedback and are also exposed to linguistic terminology while practicing grammar and vocabulary.
 
 

In contrast, the feedback for the beginning learner is more specific, and linguistic terminology is avoided ("No, DER of DER ZEIT is incorrect. ZEIT is not masculine"). An example is given in Figure 7.
 
 

            2.1.2. Remediation

In addition to tailoring feedback messages suited to learner expertise, the system also recommends remedial tasks. At the end of each chapter, the system displays learner results and suggests additional exercises according to the number and kind of mistakes that have occurred. For example, the summary page in Figure 7 states that the student John made one spelling mistake and ten errors in subject-verb agreement with the Build a Sentence exercise set. Due to the number of errors, the system suggests further exercises on subject-verb agreement. The student will receive an individually tailored set of remedial exercises addressing the mistakes s/he made during previous practice. The results can also be sent to the instructor.

3. Error Analysis in Intelligent Language Tutoring Systems

The German Tutor contains a spelling checker and a syntactic parser which detect spelling, syntactical, and morphological errors. While the semantic error detection component is limited, the system nonetheless recognizes the violation of semantic restrictions on verbs and their complements. For example, while "schmecken" is [+food], "gefallen" is marked as [-food] in the lexicon. Thus the sentence "Die Bohnen gefallen mir" (I like the beans), an error commonly made by English speakers due to native language interference, will be corrected by the system.

A number of studies have proven the usefulness of primarily syntactic parsers in language learning. For example, a study conducted by Juozulynas (1994) at Miami University showed that only 20% of errors in the essays of second-year students of German are of a semantic nature. Juozulynas collected 349 students compositions. In all, 360 pages (313 essays) were included in the study. The error distribution in his study was:

syntax: 28.6%
morphology: 24.4%
punctuation: 12.3%
spelling: 14.7%
semantics: 20%

A study by Rogers (1984) who collected 26 German essays with an average length of 559 words revealed similar results. Her distribution of errors is:

syntax: 35%
morphology: 34.5%
lexical: 15.6%
orthography: 9.5%
complete transfer of English expression: 5.4%

Given the outcomes of these studies, syntactic parsers can treat a high percentage of student errors. In the German Tutor, we also largely control the errors that can occur because the system displays tasks in which students select from a given pool of vocabulary and grammatical structures. Thus errors of semantic nature are less likely to occur. Constraining the input domain, however, also results in higher accuracy of the system. In an open domain, any NLP system tends to become less accurate which can pose problems, in particular, for the beginning learner. Our studies show (Heift & Nicholson, 2000) that, after extensive beta-testing, the performance of the system rivals a human analysis by constraining the domain in the ways described.

In the following section we discuss a recent system trial.

    3.1 A System Trial

During the spring semester 2000, 33 students from two introductory German classes spent three one-hour sessions using the Build a Sentence exercise (Figures 6 and 7). During each of the three sessions, students practised vocabulary and grammar from two chapters for a total of 6 chapters and 120 exercises. The linguistic structures themselves had been practised in communicative class activities prior to the computer sessions. Students were also already familiar with the grammatical terminology used in the system feedback. For data collection, we implemented a computer log to collect detailed information on the student-computer interaction (see Heift & Nicholson, 2000).

When analyzing the data, we were interested in the types of errors that occurred during practice and their distribution with respect to the three learner levels: beginner, intermediate and advanced. Table 1 shows the error break-down by error type and learner level.

	Beginner	Intermediate	Advanced	TOTAL	%
Punctuation	2	41	5	48	2.5%
Task	3	126	21	150	7.9%
Spelling	72	278	6	356	18.7%
Grammar	260	944	148	1352	70.9%
TOTAL	337 (17.7%)	1,389 (72.9%)	180 (9.4%)	1906	100%

Table 1 indicates that roughly two thirds of the errors were grammatical (70.9%). Spelling errors were the second most frequent errors (18.7%), followed by mistakes where students did not follow the task (7.9%). An example of the latter error type is given in example (4).

(4a) Task: Sie (singular) / nach / Deutschland / fahren
(4b) Student Answer:             Sie fahren nach Deutschland.
(4c) System Feedback:         Der Satz ist richtig, aber das war nicht die Aufgabe.
                                               The sentence is correct but that was not the task.

Our data, however, emphasize the importance of an adaptive language learning system. Approximately one fourth, or 27%, of the time, students either required more elaborate feedback suited to the beginning learner, or, in the case of the advanced learner, less detailed feedback was sufficient to correct the errors. Moreover, and although not illustrated in Table 1, ten students or 30.3% of all participants received remedial exercises for at least one of the six chapters.

Considering the types of grammar errors, Table 2 provides the error-breakdown for our study participants:
 

Table 2 indicates that most errors occurred with direct objects (21.5%) and subject-verb agreement (20.6%). However, these were the most frequent constructions contained in the 120 exercises of this study. Table 3 displays the grammar topics for the six chapters. For instance, chapters 5 and 6 focus on the present perfect and modals. These constructions are not contained in any of the previous chapters, thus there is less opportunity for errors with these grammar topics than, for example, subject-verb agreement.
 
 

Chapter	Grammar topics
Chapter 1 and 2	Gender and number agreement of noun phrases, subject-verb agreement, present tense of regular verbs, verb conjugations of "to be" and "to have", word order of finite verbs
Chapter 3 and 4	Present tense of irregular verbs, accusative and dative objects, accusative and dative prepositions, two-way prepositions
Chapter 5 and 6	Present perfect, auxiliaries, word order of nonfinite verbs, modals

It is interesting, however, to consider the number of grammar errors made by each learner level as given in Table 2. As mentioned, the system places students into one of the three learner levels, beginner, intermediate, or advanced, depending on their performance during practice. Students are assessed over time, that is, they will most likely be at different levels for each grammatical construction over time. In case of an error, the student moves towards the beginner level. To improve the score, the student needs to submit sentences in which the grammatical construction in question has been applied correctly.

From a learning perspective, the data in Table 2 indicate three distinct groups:

1. those grammar topics where the error distribution for the beginner and advanced levels is fairly balanced (missing words, extra words, word order, auxiliaries, verb complements),
2. those grammar points where students are far more often at the beginner than the advanced level (direct and indirect objects, accusative, dative, two-way prepositions), and
3. those grammatical constructions where students are far more often at the advanced than the beginner level (subject-verb agreement, subjects).

The data support the need for an individualized system which makes subtle distinctions between error types. The main strength of the Student Model in the German Tutor is that a single erroneous message will not drastically change the overall assessment of the student. The Student Model indicates precisely which grammatical violations occurred, allowing for a finely tuned assessment of student competency. As a consequence, a student can be at a different level for each given grammar constraint reflecting her performance of each particular grammatical skill. This subtlety of evaluation is desirable in a language teaching environment because as the student progresses through a language course a single measure is not sufficient to capture the knowledge attained and to distinguish among learners. The Student Model aids in directing each student toward error-specific and individualized remediation.
 

4. Conclusions

In this paper, we discussed the strengths of Web-based computer systems that incorporate NLP techniques in analyzing student input. By way of example, we introduced the German Tutor, an ILTS that provides individualized, error-specific feedback and flexibility in answer processing compared with more generic drill-and-practice programs. ILTSs are labour-intensive and require computer programming and linguistic expertise. However, their error analysis can be highly accurate if the domain is appropriately constrained.

A recent study conducted with two introductory German classes provides an error typology and emphasizes the importance of an adaptive language learning system that considers user diversity. A Student Model keeps a record of each user and makes system decisions accordingly.

The German Tutor, in addition to the pedagogical strengths discussed in this paper, maintains a computer log that records student computer interaction. For each student, we log learner input and system response. In addition to determining the accuracy of our system, this research tool allows us to study how students use computer programs for grammar practice, the kinds of errors they make and their language progress over time. The results may also assist in the development of effective language learning systems.
 
 

1. Widmeier, E. & Widmeier, F. (1999). Treffpunkt Deutsch (third edition). Prentice Hall. Available: http://cw.prenhall.com/treffpunkt/ (April 2001).

2. Terrell, T., Tschirner, E., & Nikolai, B. (2000). Kontakte (fourth edition). McGraw Hill. Available: http://www.mhhe.com/socscience/german/kontakte/ (April 2001).
 
 

Brandl, K. K. (1995). Strong and Weak Students' Preference for Error Feedback Options and Responses. Modern Language Journal, 79, 194-211.

Elsom-Cook, M. (1988). Guided Discovery Tutoring and Bounded User Modelling. In Self, J. (Ed.). Artificial Intelligence and Human Learning. Bristol: J. W. Arrowsmith Ltd., pp. 165-178.

Garrett, N. (1987). A Psycholinguistic Perspective on Grammar and CALL. Smith, W. F. (Ed.). Modern Media in Foreign Language Education: Theory and Implementation. Lincolnwood, IL: National Textbook, pp. 169-196.

Hagen, L. K. (1994). Unification-Based Parsing Applications for Intelligent Foreign Language Tutoring Systems." CALICO, 12 (2), 5-31.

Heift, T. (2002). Learner Control and Error Correction in ICALL: Browsers, Peekers and Adamants." CALICO, 19 (3), in press.

Heift, T. (2001). Error-specific and Individualized Feedback in a Web-based Language Tutoring System: Do they Read it? ReCALL, 13 (2), 129-142.

Heift, T. & Nicholson, D. S. (2000). Theoretical and Practical Considerations for Web-based Intelligent Language Tutoring Systems. In Gauthier, G., Frasson, C., & VanLehn, K. (Eds). Intelligent Tutoring Systems. 5th International Conference, ITS 2000, Montreal, Canada, June 2000, pp. 354-62.

Heift, T. & McFetridge, P. (1999). Exploiting the Student Model to Emphasize Language Teaching Pedagogy in Natural Language Processing. Proceedings of the Workshop on Computer-Mediated Language Assessment and Evaluation in Natural Language Processing, ACL/IALL: 1999, Maryland: 55-62.

Holland, M. V., Kaplan, J. D., & Sama, M. R. (Eds.). (1995). Intelligent Language Tutors: Theory Shaping Technology. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc.

James, C. (1998). Errors in Language Learning and Use: Exploring Error Analysis. In C. N. Candlin (Ed.). Applied Linguistics and Language Study. London: Addison Wesley Longman Limited.

Juozulynas, V. (1994). Errors in the Compositions of Second-Year German Students: An Empirical Study for Parser-Based ICALI. CALICO Journal, 12 (1), 5-17.

Labrie, G. & Singh, L. P. S. (1991). Parsing, Error Diagnostics, and Instruction in a French Tutor. CALICO, 9, 9-25.

Levin, L. S. & Evans, D. A. (1995). ALICE-chan: A Case Study in ICALL Theory and Practice. In Holland, M. V., Kaplan, J.D., & Sama, M.R. (Eds.). Intelligent Language Tutors: Theory Shaping Technology. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc., pp. 77-99.

Nagata, N. (1996). Computer vs. Workbook Instruction in Second Language Acquisition. CALICO, 14 (1), 53-75.

Nagata, N. (1995). An Effective Application of Natural Language Processing in Second Language Instruction. CALICO, 13 (1), 47-67.

Nagata, N. (1993). Intelligent Computer Feedback for Second Language Instruction." Modern Language Journal, 77, 330-338.

Rogers, M. (1984). On Major Types of Written Error in Advanced Students of German. International Review of Applied Linguistics in Language Teaching, 22 (1), 1-39.

Rumelhart, D. & Norman, D. A. (1975). Explorations in Cognition. San Francisco: Freeman.

Sanders, R. (1991). Error Analysis in Purely Syntactic Parsing of Free Input: The Example of German. CALICO, 9 (1), 72- 89.

Schwind, C. B. (1995). Error Analysis and Explanation in Knowledge Based Language Tutoring. Computer Assisted Language Learning, 8 (4), 295-325.

Terrell, T., Tschirner, E., & Nikolai, B. (2000). Kontakte (fourth edition). McGraw Hill. Available: http://www.mhhe.com/socscience/german/kontakte/ (April 2001).

Van der Linden, E. (1993). Does Feedback Enhance Computer-Assisted Language Learning. Computers & Education, 21 (1-2), 61-65.

Venetzky, R. & Osin, L. (1991). The Intelligent Design of Computer-Assisted Instruction. New York: Longman Publishing Group.

Wang, Y. & Garigliano, R. (1992). An Intelligent Language Tutoring System for Handling Errors Caused by Transfer. In Frasson, C., Gauthier, G., & McCalla, G. I. (Eds.). Intelligent Tutoring Systems. Lecture Notes in Computer Science. Berlin, Heidelberg, New York: Springer Verlag: 395-404.

Widmeier, E. & Widmeier, F. (1999). Treffpunkt Deutsch (third edition). Prentice Hall. Available: http://cw.prenhall.com/treffpunkt/ (April 2001).

Yang, J. & Akahori, K. (1999). An Evaluation of Japanese CALL Systems on the WWW. Comparing a Freely Input Approach with Multiple Selection. Computer Assisted Language Learning, 12 (1), 59-79.

Yang, J. & Akahori, K. (1997). Development of Computer Assisted Language Learning System for Japanese Writing Using Natural Language Processing Techniques: A Study on Passive Voice. Proceedings of the Workshop "Intelligent Educational Systems on the World Wide Web". 8^thConference of the AIED-Society, Kobe, Japan.