I wish to provide comments on the proposed revisions to the National Science Foundation’s merit review criteria.
It is encouraging to see more explicit guidelines outlining the kinds of things that would fall under the broad category of ‘broader impacts.’ Please extend this additional detail to the specific guidance that will be developed for various stakeholder communities.
With particular relevance for the broader impacts criterion, it would benefit NSF to engage more with the public to solicit input on the criteria and how to implement them. The process so far has been relatively silent on how it has engaged with the public to this point in the process.
Review Process for Broader Impacts
Part of the challenge in successfully implementing the broader impacts criteria is an effective review process. It does not necessarily follow that the same reviewers that can effectively judge the intellectual merit of a proposal can effectively judge the broader impacts of the projects. But the logistical challenges of empaneling review panels and processes applications in a timely fashion makes adding a separate panel process for reviewing broader impacts impractical.
The review process becomes a bigger challenge given how broader impacts might be addressed in the proposed revisions. With NSF research “collectively” helping to advance a number of national goals, there will need to be some means of cross-discipline and cross-directorate assessment of how NSF grants are working to achieve this collective goal. This would be particularly true for broader impacts work that is not achieved through the research, but through activities related or ‘ancillary’ to the research. Does the Foundation have sufficient staff expertise to achieve this goal? That capacity (or plans to address it) needs to be a part of the new merit review process.
While the specific guidance is welcome, it would be helpful – for both grant applicants and the public – to better understand what is meant by the different kinds of activities outlined in Principle 3 that qualify as addressing broader impacts. A (non-exhaustive) list of possible activities that would qualify as, for example, “activities that are supported by the project but ancillary to the research.” This suggests that activities that tenure committees might consider as service or outreach might qualify for NSF funding. To the extent this reflects the intention of the Foundation, I find it encouraging. But tinkering with tenure expectations is fraught with consequences. More detail here would help make sure all affected parties have a common understanding of any new expectations.
What follows are some comments on the specific language in the proposed revisions.
Merit Review Principles and Criteria
The use of the word collectively in reference to how NSF projects are to address national goals suggests that broader impacts criteria will be assessed in a wider pool than a single program solicitation. What NSF has in mind for this kind of review needs to be explained and communicated to stakeholders and the public.
The phrase “NSF projects should help to advance a broad set of important national goals, including…” could be interpreted to mean that other possible goals could be considered relevant to broader impacts. I would encourage that this language explicitly indicate that the list of goals is suggestive, and not exhaustive.
The meaning of ‘ancillary’ in the phrase “activities that are supported by the project, but ancillary to the research.” A list of possible (but not exhaustive) activities that would fall into this category would be useful. Comparing that list to a list of activities considered “directly related to specific research projects.”
Thank you for the opportunity to comment on the proposed revisions to the merit review criteria.