ESL EFL Writing

The Pragmatic Tests

Cloze

The Cloze procedure was first applied as a reliability measure with native speakers by Taylor (1953).  It has since been demonstrated in many studies to have substantial concurrent validity as an integrative test of overall proficiency, in English as a second language.  In the studies carried out by Oller & Conrad (1971), Oller (1972), Stubbs & Tucker (1974), Irvine, Atai & Oller (1974) Hinofotis (1980) and Edith Hannia & May Shikhani (1986) high correlations scores on an established measure of language proficiency.

Cloze is considered an integrative rather than a discrete point test because it draws at once on the overall grammatical, semantic and rhetorical knowledge of the language.

To reconstruct the textual message, learners have to understand key ideas and perceive inter – within a stretch of continuous discourse and they have to produce, rather than simply recognize an appropriate word for each blank.  The focus of the task involved is more communicative than formal in nature and it is therefore considered to reflect a person’s ability to function in the language.

According to Moller (1981), both cloze procedure and paragraph writing are integrative in nature.  They require the learner to draw upon several language skills simultaneously and involve complex processing of language while the focus is on content.  Both require the production of language rather than mere recognition of correct items, although writing may be considered to include the communicative dimension more directly.

There is some disagreement among researchers about the suitability of cloze to test high – order language abilities.  According to Alderson (1979) cloze provides a measure of core linguistic skills of a relatively low order.  According to a number of other researchers like, Chihara, Oller, Weaver and Chavez (1977), Bachman (1982) and Brown (1983), cloze procedures can test not only lower – order linguistic skills, but also higher – level ability involving discourse constraints across sentences.

Types of Cloze

Some cloze use rational deletion procedure to ensure the inclusion of cohesive items.  Bachman (1982) in his study used the rational deletion procedure.  Some other studies use systematic deletion procedure where the blanks occur at regular intervals.  In a study conducted by the American University of Beirut cloze passages were carefully selected so that blanks systematically include both syntactic and cohesive factors and therefore covered higher – order skills. Edith Hanania and May Shikani (1986) in their article in TESOL Quarterly (Vol.20) report a recent research carried out at the American University of Beirut where the inter – relationship of three types of measures – A standardize ESL test, a cloze and a written composition test – was studied.  The results indicate that there is a high correlation between cloze and written composition test.

Oller (1979) gives detailed instruction for the selection of cloze passage and for scoring procedure.

Selecting the material

The difficulty level of the passage should be suited to the level of the learners. The passage should be of some general topic which does not require technical knowledge.  The standard test construction technique known as the fixed ratio method involves deleting every ‘n’ the word (where ‘n’ usually varies from five to ten) and replacing each one with a standardized blank (usually about fifteen typed spaces).  The standard length of cloze is fifty items – thus the passage length is approximately fifty times ‘n’.  Another procedure is to delete words on some variable ratio usually decided by a rational selection procedure – e.g.  delete only content words.  Taylor (1957) demonstrated that the every nth word deletion technique yielded cloze items of somewhat greater reliability and validity than deliberate selection of words to be deleted, i.e. deleting only function or content words.

Scoring criteria

There are two methods of scoring a cloze passage.

The exact word method – The word in the original passage is considered as the correct one and any other word which the testee might supply is not accepted.

Contextual appropriateness – Any word supplied by the testee which is appropriate to the context is accepted.

General guidelines given by Oller for contextual appropriateness are as follows:

If it is the exact word as in the original passage it is marked correct.

If not, it should be checked to see if it fits the immediate surrounding context; whether or not it violates any local constraints in the same sentence or surrounding phrases.

It should also be checked to see if it is consistent with all of the preceding and subsequent text (this includes previous and subsequent responses in other blanks as filled in by the examinee).

If the response passes all these checks it is marked correct, otherwise it is considered incorrect.

Naccarate and Gilmore (1976) have shown that contextually appropriate scoring  can be done either by a single rater working alone or by several raters, with equivalent results.

Having discussed in detail the pragmatic tests, it is time to compare them with discrete point and integrative tests.

Comparison - Discrete point / Integrative / Pragmatic Tests

John Carroll (1961) first proposed the distinction between discrete point and integrative language tests. Though these two are not always different for practical purposes there are two basic differences.

?              The theoretical bases of the two approaches contrast markedly.

?              Predictions concerning the effects and relative validity of the two procedures differ.

Discrete tests take language skills apart and attempt to test the knowledge of language one bit at a time.  Integrative tests put the skills back together and attempt to assess learner’s capacity to use many bits all at the same time and possibly while exercising several presumed components of a grammatical system and perhaps more than one of the skills or aspects of skills.

Pragmatic tests go a little beyond the integrative tests.  The pragmatic tests meet the pragmatic naturalness criteria while the integrative tests do not reach up to that point.  So unlike the integrative tests, pragmatic tests invoke and challenge the developing grammatical system of the learner.  This requires processing sequences of elements in the target language subject to temporal contextual constraints.  In addition to this, pragmatic tests are such that learners need to relate the linguistic sequences to extralinguistic contexts in meaningful ways.

Implications

$               The above discussions indicate that pragmatic tests like dictation and cloze can be used effectively to test language proficiency.  These two tests are valid and reliable measures of language use.  They are practical because it is not very difficult to prepare the material, administer the test, score the test and interpret the results.  They also have diagnostic value, and error analysis based on the results could offer suggestions for further instructional programmes.

$               Scoring of cloze passages could be based on contextual appropriateness and that of dictation could be of the correct word-in-sequence method.

        (See 2.3.6 biii)

$               Since objective, multiple-choice items do have some advantages, some vocabulary and grammar items could be tested using this method.

$               To test the communicative efficiency, a letter could be given for which they are to write the reply.  The reply could also be given in part thereby structuring the learners’ response partly so as to make the evaluation more effective. This could again be scored using both contextual appropriateness criteria and correct word-in-sequence method.

If you have any questions, comments or suggestions post them on my blog and I will respond to you: What to Pursue Blog