Evaluation of the NCIH Initial Deployment

Education Needs Assessment

An Evaluation of Perceived Needs Versus Current Applications of the NCIH

A report presented to the OSC by the Needs Assessment Focus Group of the NCIH Evaluation Advisory Steering Committee

Focus Group Members: Mr. Theodore G. Kummer - Chair, Mrs. Linda H. Phillips - Reporter, Mr. Bill Randall, Mr. Mike Parrish

Introduction

In October 1995, this focus group was created by the OSC to assist with the development and implementation of an NCIH evaluation model. Since there was no formal evaluation of the project regarding effectiveness of delivery, student satisfaction, use, and similar issues, this focus group wishes to state at the outset that data collected and analyzed for this report should be considered as baseline data. Furthermore, although methodology and procedures are assumed to be valid, it is expected that, as follow-up evaluations occur, there will be improvement in methods, procedures, and instruments.

One of the critical questions in evaluating the effectiveness of a specific project is: Does the project fulfill the needs of the population which it is designed to serve? In considering the need for a statewide fiber-optic network, the decision was made by the North Carolina Legislature to utilize a grants process to identify and fiscally support initial sites for the NCIH. The first portion of the narrative section of the grant application addressed the need for NCIH service. More specifically, it stated: "Describe the intended use for a NCIH connection for your institution, including: What kinds of applications have been identified or are being developed for use on the NCIH?" (State of North Carolina Office of State Controller, 1994, October 12).

These grant applications provided the baseline data for evaluating two basic research questions:

1. What are the uses and applications of the NCIH as identified by potential users?

2. To what extent have those applications been utilized by sites that are operational?

The primary focus of the research questions was not to determine whether educational delivery or student satisfaction is better with the technology, but whether the system’s uses and value justify its existence. In other words, are educational opportunities for the citizens of North Carolina improved both qualitatively and quantitatively by the NCIH and, if so, what promise does this hold for the future?

Procedure

To address the first research question, needs/applications identified by potential users, a frequency rating was utilized (count of anticipated uses of, or need for, NCIH). All grant applications submitted in November of 1994 were reviewed with special attention to applications identified for each specific site. The identified needs/applications were grouped into predetermined general categories (distance education, economic development, health care, library services, government services, etc.). Sub- categories were identified for each of the general groups and frequency of identified applications in each category were counted. Categories were added or modified as needed and totals were computed for each category.

Recognizing that a number of the NCIH sites had only been operational a short time, and consequently would not have implemented all anticipated applications, the needs assessment group turned to the second research question regarding the extent of use. To determine the extent of use of the identified applications, a similar procedure was employed.

A survey instrument was developed based on the categories identified in the frequency rating since the research question has to do with perceived need versus actual use. The purpose of the survey was to identify whether or not specific applications are underway, the average number of send/receive hours per month for each application, and the total participants to date. Other factors the survey addresses which are "extent issues" include room capacity and total hours per month. Length of time "on the highway" was also addressed in the final analysis. The instrument was tested at two sites, one a community college and the other a high school. Modifications were made based on feedback. Surveys were mailed to the 76 facilitators/site managers, representing a total of 116 individual sites. The survey included an explanatory letter from the OSC and an instruction sheet for the purpose of clarification of the specific requests for information. Follow-up requests were made by phone for any surveys not returned within two weeks. Raw data was then collected, applications ranked according to those most frequently utilized, and by total number of hours for each application.

Although a quantitative methodology may have fewer limitations than a qualitative "opinion" survey, follow-up information assists in the validation of the results. There are two reasons for this. First, there is the possibility that the quantitative question may be misinterpreted or unclear. Second, surveys often elicit comments or information which is of a qualitative nature. Third, first time use of an instrument to generate baseline data should involve a process for appropriate revision of the instrument for future use. To assure reliability of the survey, follow-up phone calls were made to a randomly selected sample (every tenth site from an alphabetical list of those surveys returned) for the purpose of verifying information and to determine what, if any, other information might be useful. The results were tabulated and analyzed. A quantitative evaluation of perceived need versus actual use was then conducted by comparing and contrasting the results of the identified applications and the current uses of the highway. Resulting data provided information about what applications had been implemented as well as the number of North Carolina citizens who have benefited to date from the availability of NCIH offerings.


Home Page	Table of Contents	Previous Page	Next Page