NSF CERP REU Evaluation

About

The NSF CISE Research Experiences for Undergraduates (REU) Survey, managed by the CRA Center for Evaluating the Research Pipeline (CERP), is an evaluation resource that provides critical information to the community on how participation in undergraduate research opportunities influences student pathways to computing graduate education and computing careers.

Benefits of Working with CERP

By partnering with CERP, you’ll work with a trusted, NSF-supported partner, receive a custom evaluation report for your REU site, contribute to a national study of computing REUs, and fulfill your NSF REU evaluation requirement.

Your custom evaluation report will help you:

Demonstrate student learning gains across key areas
Understand student experiences and mentoring effectiveness
Identify opportunities to improve your REU program
See how your site compares to other REUs through CERP’s national dataset
Strengthen your NSF annual reports and future proposals.

Participation Requirements and Eligibility

Participation is simple:

Share basic project information
Complete short PI surveys before and after your REU
Distribute student surveys at the start and end of your REU

That’s it — we handle the analysis and reporting.

Sign-up for CERP’s REU Evaluation

Eligible projects:

Any REU project with a computing-related focus, regardless of funding source, may sign-up for CERP’s free evaluation.

Submitting a proposal?

PIs submitting a proposal to NSF can use the following paragraph about our evaluation process in their grant submission:

The project team will work with the Computing Research Association’s Center for Evaluating the Research Pipeline (CERP) to conduct an evaluation of NSF CISE REUs. CERP conducts longitudinal quantitative surveys of REU students across three time points: a pre-test survey at the beginning of the REU, a post-test survey at the end of the REU, and a follow-up survey one year later. These online surveys gather data on students’ experiences and outcomes, measuring changes in students self-perceptions (e.g., self-efficacy, computing identity), academic development (e.g., research skill proficiency), and professional aspirations (e.g., intentions to pursue graduate school, career goals) over the course of the REU and longer-term changes. The REU PI is responsible for disseminating the pre-test and post-test surveys to their students. Students can opt in to provide their contact information to participate in the follow-up survey, which will be directly sent to them.

A few months after the REU ends, CERP will provide REU Principal Investigators with a report that examines changes in their REU students from the pre-test and post-test surveys. This report will also contain comparative data to other REU students. Collectively, this evaluation plan, data collection, and associated reporting provide evidence of student learning and changes in their knowledge and skills regarding computing research, attitudes, and intentions to pursue further computing-related education and career pathways.

Meet the team

Kari George, Senior Research Associate, Project Lead

Brendan Kane, Research Associate, Communications and Survey Administrator

Burçin Campbell, Director of Data and Evaluation, Project Oversight

Acknowledgements

The CISE REU Evaluation Project is currently supported through a U.S. National Science Foundation (NSF) cooperative agreement #2335072. The project began as a contract (49100421C0010) with the NSF CISE directorate in 2021 and was approved by the Office of Management and Budget (OMB control number 3145-0265).

Evaluation Process

CERP primarily conducts a pre-test and post-test survey evaluation model for REU evaluation, though we can also work with your site to determine the most appropriate evaluation method for your project and timeline. While the majority of projects working with us conduct their REU during the summer, we also support evaluation for REUs that are conducted during the academic year.

Data Collection

Four weeks prior to your REU start date, we will send PI instructions and a PI pre-program form.
Two weeks prior to your REU start date, we will send a suggested participant recruitment email and the pre-test survey link to the PI to distribute the survey to the REU participants. We ask that participants complete the pre-test survey before or within the first 2 weeks of the REU. CERP staff will monitor that data collection and send updates on response rates and reminders for survey distribution as appropriate.
About a week before the end of your REU, we will send the second PI post-program form for PIs to complete. Two days later, we will send the PI post-test survey link and suggested participant recruitment email to be sent to all REU participants. All forms and surveys must be completed within 3 weeks after your REU ends. CERP staff will also monitor that data collection and send updates on response rates and reminders for survey distribution as appropriate.’

See the PI Checklist reference guide of the key steps, action items, and deadlines necessary for conducting the evaluation.

Reporting

By partnering with CERP, Principal Investigators (PIs) will receive a custom evaluation report, contribute to a national study of computing REUs, and meet this NSF evaluation requirement.

The custom REU Evaluation Site Report allows PIs to:

Demonstrate learning gains across key areas
Understand student experiences and mentoring effectiveness
Identify opportunities to improve their REU program
See how their site compares to other REUs through CERP’s national dataset.

The REU Evaluation Site Report includes:

A key findings page – a one-page report summarizing key findings from your REU participants.
A site report – summarizing your REU participants’ responses alongside other similar REU sites.

To be eligible for a custom site report, PIs need at least 5 respondents on both the pre-test and post-test student surveys. For sites or supplements with fewer than 5 respondents, we produce an REU Small Sites and Supplements report which aggregates those students and projects together.

Reports are typically available 2-3 months after data collection is complete for each term (e.g., summer, fall, and spring).

Click here to view a sample comparison report

Measures

Our REU Evaluation surveys include various measures tailored towards different experiences of students that participate in NSF CISE REUs. Below is a non-exhaustive list of measures included.

Click here to request a copy of our participant surveys.

Previous REU experiences

Year, setting, and details of previous REUs that students have participated in

General attitudes and beliefs regarding computing

Perceptions of abilities, skill sets, and accomplishments
Levels of identification with computing and sense of belonging
Perceived support from professional network and mentors

Future plans and career aspirations

Highest degree intentions
Employment and graduate school plans
Future career interests
Career values and perceptions of computing careers

Characteristics and Experiences

Academic background (e.g. pre-college CS and Math courses)
Enrollment information and reasons for matriculation and persistence
Perceptions of degree program and institution
Degree of involvement in curricular, co-curricular and extracurricular activities
Demographics

Results

As part of our ongoing dedication to being a resource for the computing community, CERP makes it a priority to provide timely access to our research findings as soon as they become available. Explore our latest results to stay up-to-date with the REU evaluation team.

Survey Reports

The CERP team prepares an annual report highlighting key insights from the pre-survey, offering a snapshot of who participated in the formal research experience program for that cohort.

Click here to view the most recent report for the 2023 cohort.

Infographics in CRN

Explore our CERP Infographics, published in Computing Research News, for a visual dive into research findings and key takeaways on formal research experiences. These infographics provide valuable insights that the computing community will find both informative and engaging.

Infographic: Highlights on Faculty Support for REU Students from CERP’s REU Evaluation

Infographic: Student Motivations for Participating in Research Experiences for Undergraduates (REUs)

Infographic: NSF CISE REU 2023 Annual Report: Student Characteristics of REU Participants

Cleveland dot plot showing the seventeen motivating reasons undergraduate computing students decided to participate in a formal research experience. The most frequently cited reason is to build work experience or their resume/CV, with 63% selecting this option.

CERP Infographic: What motivates students to participate in formal research experiences?

Fact sheet presenting a summary of the demographic makeup of CISE REU Past Participants who participated in REU projects between 2013 and 2021.

Who has participated in an NSF CISE REU in the last ten years?

Thumbnail for CERP infographic showing a horizontal bar graph

Compared to Peers, Students from Low-Income Backgrounds Report Less Awareness of – but Greater Interest in – Formal Research Experiences

Text reads “REU Participation” with 3 graduation caps underneath

REU Participation Encourages Students to Pursue Graduate Degrees

One Year Later, CERP Data Still Indicate REU Participation Relates to Graduate School Enrollment

IT Majors are Less Likely to Intend to Pursue Formal Research Before Graduation Compared to CS and CSE Majors

Previous Next

Click here for more

FAQ

Frequently Asked Questions about CERP NSF REU Evaluation

What is the cost of working with CERP evaluation?

With funding from the U.S. National Science Foundation, CERP’s REU evaluation services are free. PIs do not need to include a budget for CERP’s evaluation in their proposals.

If I work with CERP, do I need to hire another evaluator to fulfill NSF’s evaluation requirement?

CERP’s evaluation fulfills NSF’s evaluation requirement, so another evaluator is not necessary for this purpose. If you have other funding sources, please contact your program officer to confirm CERP’s evaluation services alone are sufficient.

Will CERP collaborate with other evaluators?

While CERP is happy to provide other evaluators with the methodology and copies of the survey, CERP is not directly collaborating with other REUevaluators for this project.

When I signed up to work with CERP for evaluation, I received a proposal paragraph. How do I include it in my NSF REU Site/Supplement proposal?

The proposal paragraph that we provide should be inserted directly into the required section of your proposal on evaluation. As it is a general paragraph, please feel free to add minor modifications to the language so that the paragraph includes any project-specific language or institutional information.

As an REU PI working with CERP for evaluation, do I need to have my own IRB approval?

CERP has IRB approval to conduct evaluation through BRANY IRB. As an REU PI, you are encouraged to contact your IRB office to determine whether additional oversight is required by your institution. You can find a copy of our approval letter and templates here that you can use to initiate that conversation.

Can I add specific questions to the CERP survey?

We have some limitations in our ability to change our surveys, as our project requires approval of all study instruments by our IRB. You may contact us if you have suggestions for new measures. We will keep a record of any requested additions and will consider suggestions for future surveys.

Will surveys also be sent to mentors/staff?

CERP collects information about the REU from PIs and other project leadership to better understand the structure of each REU. Surveys are not sent to REU mentors or other staff.

Who is responsible for distributing the evaluation surveys to student participants?

Each REU PI or Co-PI will be responsible for distributing the evaluation surveys to participants for the pre-test and post-test REU student surveys. Prior to data collection, CERP will provide the PIs (or the determined contact person) with all materials necessary for distribution, such as recruitment templates and the survey link. All communications from CERP will come from cerpreu@cra.org. For more information, see Evaluation Process.

Where can I share feedback about my experience working with CERP?

If you have feedback or suggestions for improvements, please contact us or complete our Evaluation Feedback Form.

Provide Feedback