NSF Merit Review Criteria Comments

Since I posted several times on the subject of NSF’s proposed revisions to its merit review criteria, it seems proper to: submit comments to NSF, and post them here.  Let’s get to it.


I wish to provide comments on the proposed revisions to the National Science Foundation’s merit review criteria.

It is encouraging to see more explicit guidelines outlining the kinds of things that would fall under the broad category of ‘broader impacts.’  Please extend this additional detail to the specific guidance that will be developed for various stakeholder communities.

With particular relevance for the broader impacts criterion, it would benefit NSF to engage more with the public to solicit input on the criteria and how to implement them.  The process so far has been relatively silent on how it has engaged with the public to this point in the process.

Review Process for Broader Impacts

Part of the challenge in successfully implementing the broader impacts criteria is an effective review process.  It does not necessarily follow that the same reviewers that can effectively judge the intellectual merit of a proposal can effectively judge the broader impacts of the projects.  But the logistical challenges of empaneling review panels and processes applications in a timely fashion makes adding a separate panel process for reviewing broader impacts impractical.

The review process becomes a bigger challenge given how broader impacts might be addressed in the proposed revisions.  With NSF research “collectively” helping to advance a number of national goals, there will need to be some means of cross-discipline and cross-directorate assessment of how NSF grants are working to achieve this collective goal.  This would be particularly true for broader impacts work that is not achieved through the research, but through activities related or ‘ancillary’ to the research.  Does the Foundation have sufficient staff expertise to achieve this goal?  That capacity (or plans to address it) needs to be a part of the new merit review process.

While the specific guidance is welcome, it would be helpful – for both grant applicants and the public – to better understand what is meant by the different kinds of activities outlined in Principle 3 that qualify as addressing broader impacts.  A (non-exhaustive) list of possible activities that would qualify as, for example, “activities that are supported by the project but ancillary to the research.”  This suggests that activities that tenure committees might consider as service or outreach might qualify for NSF funding.  To the extent this reflects the intention of the Foundation, I find it encouraging.  But tinkering with tenure expectations is fraught with consequences.  More detail here would help make sure all affected parties have a common understanding of any new expectations.

What follows are some comments on the specific language in the proposed revisions.

Merit Review Principles and Criteria 

Principle 2:

The use of the word collectively in reference to how NSF projects are to address national goals suggests that broader impacts criteria will be assessed in a wider pool than a single program solicitation.  What NSF has in mind for this kind of review needs to be explained and communicated to stakeholders and the public.

The phrase “NSF projects should help to advance a broad set of important national goals, including…” could be interpreted to mean that other possible goals could be considered relevant to broader impacts. I would encourage that this language explicitly indicate that the list of goals is suggestive, and not exhaustive.

Principle 3:

The meaning of ‘ancillary’ in the phrase “activities that are supported by the project, but ancillary to the research.”  A list of possible (but not exhaustive) activities that would fall into this category would be useful.  Comparing that list to a list of activities considered “directly related to specific research projects.”

Thank you for the opportunity to comment on the proposed revisions to the merit review criteria.

David Bruggeman


2 thoughts on “NSF Merit Review Criteria Comments

  1. Gavin,

    I don’t know what plans there are to publish the comments. Your point about the publishing requirements of Federal Register published notices is well-taken (and I should have included it in earlier posts). If nothing is forthcoming, inquiring with agency makes sense. This shouldn’t require a Freedom of Information Act request.

    Daniel Sarewitz’s recent Nature comments (available at CSPO’s website – or should be) are also required reading on this topic. Where I avoided diving into the tangle of having a review panel make judgments on national goals, Sarewitz swam quite well.

    If, and this is perhaps a big if, the comments received reflect the kinds of concerns brought out by you, Frodeman and Holbrook, Sarewitz and others, there will need to be a lot more clarification/guidance on the meaning and implementation of this criteria.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.