Utilization-focused evaluation for program improvement : a needs analysis in a Colombian government entity

Abstract

The primary purpose of this evaluation was (a) to identify the needs of the professionals enrolled in a General English course in one government entity in Colombia, (b) to determine whether the current language training program meets those needs, and (c) to provide a sound basis for decision-making regarding future language training program. To this end, four evaluation questions were addressed in four phases, following the utilization-focused evaluation (UFE) framework. Data were collected from the professionals working in this entity (n=66), high-management officials in charge making decisions (n=2) and the program administrator, using the following instruments: focus groups, interviews, observations and a survey questionnaire. In phase one, 38 domain experts participated in focus groups and structured interviews. In phase two, a questionnaire survey was decided to be used as the methods for further data collection on needs. In phase three a task-based needs analysis was conducted to determine the frequency of tasks that require using English at the workplace. Following the data collection, the professionals were tested for academic reading and general proficiency (92% response rate and 100% respectively). In phase four, a meta-evaluation was performed. Data were analyzed using thematic coding and descriptive statistics. The findings revealed the discrepancies between the participants' subjective and objective needs. Moreover, on the basis of the findings, this evaluation proposes an ESP course as an answer to the complexity of needs in this government entity