Scriven borrows from Bob Stake in offering a less formal but perhaps more enlightening distinction between formative and summative: “When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.” From an appreciative perspective, formative evaluation can be said to be an exercise in fully understanding the complex dynamics and causal factors influencing the operation of a program and taking corrective action if needed. By contrast, a summative evaluation allows one to identify and build on the specific successes and strong features of a specific program unit. Both formative and summative evaluations can be appreciative, and the comprehensive appreciation of any program unit involves both formative and summative evaluation processes.

Program Planning and Evaluation

Ed Kelly, an experienced program evaluator who served on the faculty of Syracuse University for many years, further differentiated judgments concerning the extent to which the intentions of the program were satisfied and judgments concerning whether or not the program was any good. Concern for judgment necessarily involves issues of values, criteria, goals, customers and audience. Concern for evaluative process necessarily involves issues of method, instrumentation and resources. Both approaches to evaluation require a clear definition of clientship, a precise sense of the role of evaluation and an explicit understanding of the way the judgment or process will be used by the program staff and others.

In essence, program evaluation involves the development of a process whereby program activities can be interrelated and compared to program expectations, goals and values. The nature of this interrelationship will vary considerably. In some instances, external assistance will be required to establish the process, while in other instances the external assistance will be used to provide the interrelationship judgments once the process has been defined. In yet other instances, the external assistant (evaluator) both identifies the process and provides the judgments.

Regardless of the process being used, an effective program evaluation effort will commence with the initial planning of the program. In planning for any program, or in deciding on the initiation of a proposed program, the processes of evaluation are inevitably engaged. Those who plan the program will be concerned with the validity of their assumptions about needs, strategies and resources. Those who review their proposal will ask questions about feasibility, attractiveness and probable success.  Others will ask how program achievement is to be measured.  Program evaluation is not a topic to be addressed at the end of a planning process; rather, program evaluation should be a vital and influential element that is discussed at the beginning and given serious consideration throughout the process–especially if the evaluation is being engaged in the assessment of a complex process such as professional coaching.

The Four Ds of Program Evaluation

There are four basic types of program evaluation: (1) description, (2) documentation, (3) determination of outcomes, and (4) diagnosis. I identify these as the “four Ds.” An outcome determination evaluation is conducted primarily for the purpose of judging the degree to which a program achieved its intended goals and outcomes. This “summative” approach aids decision-making about the continuation of the program. Ongoing decision making concerning the nature, content and scope of a program are best addressed through use of diagnostic evaluation. This type of evaluation is “formative” in nature, since it is conducted while a program is in progress and is used to continually or intermittently refine and improve the program. Program evaluations often are of greatest value when they aid the dissemination of program results. Descriptive and documentary approaches to program evaluation are most often employed when dissemination is critical. Descriptive evaluation tells other people about the nature and scope of a program. Documentary evaluation provides evidence for the existence of the program and its outcomes. It illustrates the nature of the program and its impact. Following is a more detailed description of each of these four types of program evaluation.

Description of Program

The first feature in any program evaluation, according to Scriven, is the identification of the program unit(s) being evaluated. He suggests that this identification should be based in a comprehensive description of the program being evaluated. Thus, program description is always the first element of a program evaluation. It is also one of the final elements, for any final evaluation report will typically contain a description of the program being evaluated. Consequently, there is little need to spend much time advocating the importance of or identifying procedures for the description of a program. Nevertheless, most program descriptions can be improved. Given the importance of dissemination, one must be certain not only that information about the program is accurate and complete, but also that other people understand the program description.