Centers for Ocean Sciences Education Excellence (COSEE) FAQ's
BACKGROUND
The COSEE National Advisory Committee met in November, 2007 and concluded that the tasks outlined in the recent NSF program solicitation (NSF 08-509) for the COSEE National Network Evaluator would require more than the budgeted $75,000/year. It recommended that NSF increase the amount of funding available for the evaluation activities. The intent of this FAQ is to clarify the scope of the required evaluation by specifying that COSEE Network activities that affect the NSF-funded oceanographic research community are of highest importance and by describing the constraints on how to respond to the Academic Competitive Council expectations.
-
Are there any specific goals that should take priority in the evaluation?
- The ACC identifies a hierarchy of evaluation strategies with randomized control trials being the “strongest” method. Does the COSEE evaluation need to be based on randomized control trials?
Answer: NSF is interested in the impact and effectiveness of COSEE on all audiences, as stated in the program solicitation. Background documents for the planning and implementing the National Network Evaluation include the COSEE Blueprint (http://www.vims.edu/adv/cosee/COSEEBlueprintJan2007.pdf). The five Blueprint goals should be evaluated primarily from the viewpoint of COSEE’s impact on the NSF-supported oceanographic research, education, and public outreach community.
Evaluation of the internal functions of the COSEE Network, i.e. the ability of the Centers to function as a network, cross-network collaborations, etc. is not considered a priority for this study.
- What should the evaluator focus on with relation to the Academic Competitiveness Council (ACC) Report (U.S. Department of Education, Report of the Academic Competitiveness Council, Washington, D.C., 2007 at http://www.ed.gov/about/inits/ed/competitiveness/acc-mathscience/report.pdf)?
Answer: For the purposes of reporting to the Academic Competitiveness Council, COSEE has been identified as an “Outreach and Informal Education Program” by the NSF. The Informal Education and Outreach working group of the ACC identified two national and program-level goals: National Goal 1: Public Awareness—In the context of informal education and outreach, increase the awareness, interest, engagement, and understanding of STEM concepts, processes, and careers of the general public and other target populations. National Goal 2: Professional Audiences—Improve practice and build professional and institutional capacity by funding efforts that generate, develop, and apply innovative ideas and models for the informal science education field. These two goals should be used to develop evaluations of COSEE. Metrics for these two goals are identified on page 44 of the report.
- The ACC identifies a hierarchy of evaluation strategies with randomized control trials being the “strongest” method. Does the COSEE evaluation need to be based on randomized control trials?
Answer: The report states that “The hierarchy does not include study designs appropriate for education research in areas other than measuring the impact of an educational activity on student outcomes.” (p. 15). The report also notes, “The hierarchy is a statement of general principles and does not address all contingencies that may affect a study’s ability to produce valid estimates of a project’s impact.” (p.16). The ACC report concludes that the nature of informal education and outreach makes it difficult to conduct rigorous evaluation (p. 26). Evaluators should consider the hierarchy as guidance for evaluation development, but should propose evaluation methods that are most appropriate for the COSEE goals and the stated ACC goals for Informal Education and Outreach.