How to Evaluate a Research Program

Start with what the program is supposed to accomplish

Before examining projects or performance measures, establish what the program exists to accomplish. That may sound straightforward, but research programs often operate with several layers of objectives. There may be a scientific objective, a technology objective, an economic or social objective, a capability objective, and an organizational objective. Different participants may also understand the purpose differently.

A useful evaluation reconstructs the logic connecting those objectives. What problem is the program addressing? What needs to be demonstrated, developed, understood or changed? What outcomes are expected to follow from the work, and what assumptions connect the activities to those outcomes? A program can produce substantial activity and many tangible outputs while still failing to address the problem it was established to solve.

This distinction matters across disciplines. An engineering program may be intended to move a technology toward a performance threshold. A computing program may be intended to establish whether an approach can operate reliably with real-world data and infrastructure. An environmental program may be designed to reduce uncertainty about a system sufficiently to support policy or operational decisions.

A biotechnology program may be intended to establish a path from scientific rationale through technical validation and eventual application. The evaluation needs to understand the intended path before judging whether the individual pieces are performing well.

Look at program architecture, not just project quality

The quality of individual projects is important, but it is not enough. A program can contain technically strong projects and still be poorly designed if those projects do not form a coherent response to the program's objectives. Program evaluation therefore needs to examine the architecture connecting the work.

That means asking why each project exists, what role it plays, what evidence or capability it is expected to produce, and how its results affect other parts of the program. Some projects may generate knowledge needed by several others. Some may develop shared infrastructure or methods. Others may test high-risk propositions, provide alternatives to a preferred approach, or address a bottleneck that would otherwise constrain the portfolio.

The important question is not whether every project can be justified in isolation. It is whether the projects make sense together. A useful way to examine this is to trace the program from objective to required evidence, from evidence to decisions, and from those decisions to the intended outcome. This often exposes gaps, duplication or activities that have become disconnected from the program's purpose.

Eight questions that reveal how the program works

A practical program evaluation can be organized around a small number of questions. The answers should connect to one another rather than being treated as independent checklist items.

1. 01 What is the program trying to accomplish?

   Define the underlying problem, intended outcomes and decisions the program is ultimately expected to influence.

2. 02 What must be learned to achieve that objective?

   Identify the scientific, technical, operational or other uncertainties that could materially affect the path forward.

3. 03 Which projects or activities produce that information?

   Determine what each part of the program is intended to establish, demonstrate, develop or rule out.

4. 04 What depends on what?

   Map dependencies among projects, capabilities, infrastructure, partners, data, approvals and other conditions required for progress.

5. 05 What are the critical milestones and decision points?

   Define what evidence should be available at each milestone, what standard it must meet, and what decision follows.

6. 06 What happens if evidence is negative, ambiguous, delayed or contradictory?

   Examine whether the program has defined responses to findings that challenge the original plan.

7. 07 Where are resources being concentrated, and why?

   Assess whether funding, people, infrastructure and management attention are aligned with the uncertainties and opportunities that matter most.

8. 08 What would cause the program to change direction?

   Determine whether there are meaningful conditions under which the program would redirect, expand, pause or stop work.

Dependencies are where program logic becomes real

Dependencies are often where the difference between a collection of projects and a functioning program becomes most apparent. One project may depend on data generated elsewhere. Another may depend on a specialized platform, field access, manufacturing capability, regulatory approval, a partner organization or the successful completion of an earlier technical step.

These relationships can be easy to miss when projects are evaluated independently. Several projects may appear to be progressing in parallel while actually depending on the same unresolved bottleneck. Conversely, work that appears sequential may be unnecessarily constrained if independent questions could be investigated at the same time.

A program evaluation should therefore examine whether the sequencing of work reflects the actual dependency structure. It should also consider what happens when a dependency fails or is delayed. Strong program architecture does not eliminate uncertainty. It makes the consequences of uncertainty visible and gives the program options for responding to it.

Milestones should mark evidence or decisions, not just dates

Milestones are useful only when they establish something that matters. A date on a project plan is not, by itself, a meaningful milestone. Completing an experiment, producing a prototype or submitting a report may represent important activity, but the evaluation question is what that activity is expected to establish.

Stronger milestones define an evidence threshold or a decision condition. The milestone might require a particular performance level, a defined set of experimental results, successful validation under specified conditions, or enough information to resolve a particular uncertainty. The important feature is that the program knows what constitutes sufficient evidence and what decision should follow.

A milestone also needs a consequence. If the required result has not been produced by the agreed time, or the evidence collected is insufficient to meet the defined criterion, the appropriate decision may be to stop the project. That does not necessarily mean the project was poorly conceived or that the people conducting it performed poorly. Research and development operate under uncertainty, and some projects will produce evidence that a proposed path should not continue.

Stopping that work releases resources for other opportunities, preserves the ability to pursue better approaches, and can increase the probability that the overall program succeeds. A program that never stops projects has removed one of its most important mechanisms for managing uncertainty. The purpose of a decision point is not to protect every project from failure. It is to make sure that resources continue to be used where they have a sufficient purpose.

Portfolio decisions matter more than individual project scores

Once projects are viewed as parts of a portfolio, evaluation needs to examine how resources are distributed across them. The question is not simply which project is strongest. It is why the program is funding these projects, at these levels, at this point in time.

A useful portfolio may contain projects with different levels of risk, different time horizons and different roles. Some may address established priorities. Others may explore uncertain opportunities or develop capabilities needed for future work. Some may provide redundancy because the consequences of failure are high. Others may create alternatives so that the program is not dependent on one technical path.

Portfolio evaluation therefore looks for concentration, duplication, gaps and unnecessary commitments. A portfolio can tolerate individual project failure when the failures are informative and the remaining work retains meaningful options. Conversely, a portfolio containing many low-risk projects may consume substantial resources without materially improving the program's strategic position.

Portfolio evaluation also requires the willingness to reconsider existing work. Programs evolve, and a project that was highly relevant when the program began may become less important after new evidence changes the landscape. Continuing it simply because resources have already been committed can make the portfolio less effective over time.

This is where milestones, decision gates and portfolio management connect. When a project fails to produce the evidence required to justify its next stage, resources should not automatically follow it forward. They may be redirected to another project, held for a later opportunity, or retained as a deliberate reserve. Ending a project can therefore be a successful program decision rather than a program failure.

What happens when evidence changes the original plan?

Research programs begin with assumptions about what is important, what is technically feasible, which questions can be answered, and how work should proceed. Those assumptions are expected to change as evidence accumulates. A useful evaluation therefore examines not only whether the original plan was sensible, but how the program responded when reality differed from the plan.

This can be seen in decision histories, changes in project priorities, revised milestones, shifts in resource allocation and changes in the portfolio itself. If new evidence invalidated an important assumption, did the program respond? If a project became less promising, was the decision to continue or stop explicit? If an unexpected result opened a better opportunity, could resources be redirected toward it?

Adaptation should not be confused with drift. A program that repeatedly changes direction without a clear rationale may be poorly governed. A program that changes direction because evidence, external conditions or the understanding of the problem has changed may be functioning exactly as intended. The difference is whether the change is deliberate, evidence-informed and connected to the program's objectives.

This is one reason retrospective evaluation should examine the decisions made during the life of a program rather than judging the original plan against the final outcome alone. A program may have been well designed even though some of its original assumptions proved wrong. The more important question is whether the program recognized those changes and responded appropriately.

Resources and capability have to match the program's needs

Program design can be strong on paper and still fail because the capabilities required to execute it are not available when they are needed. Evaluation should therefore consider people, equipment, infrastructure, data, funding, partnerships, management capacity and other constraints as part of the program itself.

This is more than a question of whether the team has the right expertise. A capability may exist somewhere within the broader organization but not be available at the point in the program when it becomes critical. Specialized equipment may create a bottleneck. Access to a particular dataset may determine whether several projects can proceed. A partner may be essential to translation or deployment. Management capacity may become the limiting factor as the portfolio grows.

These constraints should be considered alongside the program's dependencies and milestones. Otherwise, apparent project delays may be treated as execution problems when the underlying issue is a structural limitation in the program.

Evaluate outputs without mistaking them for outcomes

Research programs generate many things that are easy to count: publications, patents, prototypes, datasets, software, trained personnel, partnerships, reports and completed projects. These outputs can be important, but they do not necessarily demonstrate that the program is achieving its intended purpose.

Evaluation needs to follow the chain further. What did the output enable? Did it resolve an important uncertainty, establish a capability, change a technical decision, support adoption, influence policy, enable commercialization or create the basis for another stage of work? Different programs will produce different forms of meaningful outcome, and a useful evaluation should not force them into a single metric.

This distinction is reflected in established evaluation practice. Program evaluation frameworks commonly distinguish inputs, activities, outputs, outcomes and longer-term impacts, while recognizing that the relationships among them depend on assumptions and external factors.

The same principle applies to research programs. A publication may be evidence of an active research effort, but the evaluation should ask what the work established and whether that knowledge affected the program's objectives. A prototype may be an important technical achievement, but its significance depends on what it demonstrates and what remains unresolved.

Collaborative research programs need an additional layer of evaluation

Many research programs operate across organizational boundaries. The work may connect universities, companies, government organizations, hospitals, not-for-profits or other partners. In these settings, the program architecture has to account not only for the technical work but also for how responsibilities, incentives, resources, intellectual property, timelines and decision-making are distributed.

The same evaluation logic applies well beyond one funding model or one discipline. Collaborative programs in engineering, computing, environmental research, biotechnology, manufacturing and other fields can all be examined by asking whether the structure of the collaboration supports the evidence, capabilities and decisions required to achieve the program's objectives.

Evaluate evidence as a program-level asset

Individual projects produce individual findings, but the value of a program often depends on what can be concluded when those findings are considered together. Evidence may confirm an assumption, challenge it, narrow an uncertainty or reveal that a different question matters more than originally expected.

This is where project-level scientific assessment and program-level evaluation intersect. The quality of the underlying evidence still matters. Weak measurements, poorly designed studies, inappropriate comparisons or unsupported interpretations can undermine the conclusions drawn from the work. But program evaluation adds another question: is the portfolio generating the kinds of evidence needed to make the program's important decisions?

Negative or inconclusive results can therefore have substantial value. If they rule out an approach, expose a dependency, identify a technical limit or prevent resources from being committed to an unproductive path, they have changed the decision landscape. The value of the work is not determined solely by whether it produced the result originally hoped for.

A strong program accumulates evidence in a way that progressively clarifies what should happen next. That does not require every project to succeed. It requires the portfolio to produce information that is sufficiently reliable and decision-relevant to guide the program.

Evaluate the program over time, not only at the end

Program evaluation is often most useful when it is treated as part of the program lifecycle rather than as a final accounting exercise. Evaluation at the design stage can test whether the architecture and intended outcomes are credible. Evaluation during implementation can identify dependencies, emerging risks and opportunities to redirect resources. Retrospective evaluation can examine what was achieved, what changed and what should be carried forward.

This is consistent with established evaluation practice. The NIH Fogarty program evaluation framework, for example, describes evaluation across the program lifecycle and uses quantitative and qualitative evidence to examine outputs, outcomes and impacts. NSF guidance likewise emphasizes credible evidence, appropriate evaluation design and attention to limitations and risks.

A mid-program evaluation can be particularly valuable because it creates an opportunity to act on what the evidence shows. If the evaluation identifies a structural problem but the program has no mechanism for changing priorities, reallocating resources or stopping work, the evaluation has described the problem without giving the program a way to respond.

A practical framework for research program evaluation

A useful evaluation can be organized around eight connected dimensions:

These dimensions should not be scored independently and added together mechanically. A weakness in one area may alter the significance of another. A technically strong project may have limited program value if it addresses a low-priority question. A modest project may be highly valuable if it resolves a critical dependency. A strong portfolio may still be constrained by a missing capability or an unrealistic milestone.

The purpose of the framework is therefore not to produce a single number. It is to reveal how the program actually works, where its assumptions are concentrated, how resources move through the portfolio, and whether the evidence generated is sufficient to support the decisions the program will eventually need to make.

The central question

The most useful research program evaluation ultimately asks whether the program is increasing the probability of achieving its objectives by directing resources toward the work that matters most, generating meaningful evidence, responding to what that evidence shows, and making deliberate decisions about what should continue.

That requires looking beyond activity and beyond the performance of individual projects. The evaluator needs to understand the structure of the whole program, the dependencies that connect its parts, the milestones that govern progression, the portfolio decisions that allocate scarce resources, and the points where evidence should change the plan.

The practical question is not whether a program is busy. It is whether the work is producing meaningful evidence and whether that evidence is being used to make better decisions.