Measuring the Impact of Scientific Outreach and Engagement

Q: How did you first get involved with scientific outreach for young students?

Oliveira Yassa: I studied neuroscience at Johns Hopkins University and got involved with outreach while there in the Baltimore City community. After graduating, I was selected to participate in the Baltimore City Teaching Residency, which is similar to Teach for America, but specifically for Baltimore City schools. I taught middle school science in Baltimore City for a while, which was amazing, wonderful—and also the hardest thing I’ve ever done.

Thanks to the systemic inequities that existed in the school system, many students had decided that science was not for them by the time they entered my classroom. Children and teens are natural scientists—they explore the world, ask questions, and look for answers every day. I quickly learned that to really engage my students, we needed hands-on activities that allowed their curiosity and questions to guide the learning.

When my husband and I later moved to the University of California, Irvine (UCI), I started thinking about how I could marry my passions for neuroscience and education. I became the director of outreach and education at the Center for the Neurobiology of Learning and Memory, a new position at the time that sits squarely at the intersection of neuroscience and education.

My scholarly interests are understanding how we can enhance diversity in the neuroscience workforce and figuring out the factors that impact interest, persistence, and achievement both in neuroscience and science in general. For so many years, university-based outreach has been done by faculty, students, or staff as an afterthought and on a volunteer basis. While many have created great programs, students graduate or move on, and because there is no plan for sustainability, the programs fizzle. A new group then has to pick up the outreach efforts and start all over again.

At UCI, I created the CNLM Ambassador Program, an umbrella program for graduate students and trainees to learn best practices in outreach and get involved in the planning and execution of programs along the entire education and outreach pipeline all year long. Committees within the program focus on different areas within outreach and education and are led by rotating co-chairs. The program ensures continuity and sustainability of our outreach and education programs.

You’ve been involved with a lot of different neuroscience outreach programs, from the Brain Bee to Brain Awareness Week. Why look at the impact of a Live Review?

While I’m interested in studying the impact of all of these various outreach programs, I’m particularly interested in understanding how science outreach impacts the different pieces of the system. Often, we study the impact of programs on the participant as if that individual exists in a vacuum. The reality is much more complex, and I think the impact of these programs lies in that complexity.

The Live Review allows us to see that complexity—and it’s unique for a few reasons: First, it provides an opportunity to meaningfully engage various actors within the neuroscience ecosystem like never before. The iterative and bidirectional engagement that happens between the children and scientists during the review process is quite special, and taking that process in front of a live audience in the middle of a scientific conference takes it to a whole new level because now a broad population of scientists and community members are also engaged.

The second unique piece of the Live Review is that it places power and control in the hands of children. It turns out that children, like adults, love to feel like they are in power. We often give children the perception of having authority, but in this case, the buck really does stop with them—and when they realize this, they get really excited knowing they contributed to something important.

By studying the impact of the Live Review on not only the children who participated but also on their families, their teachers, and the scientist attendees, we can start to uncover the complexities and learn about how these programs are influencing various pieces of the system.

How does the Live Review work?

Our cohort for this Live Review was made up of fourth and fifth grade students, many from Title 1 schools that might not have as many opportunities to engage in neuroscience.

The students participated in a series of five, two-hour sessions on Saturdays at UCI before the conference event where they worked with a group of graduate and undergraduate students who served as science mentors. Together, they participated in various activities including exploring preserved human brains , sheep brain dissections, and activities to learn about how neurons work. Once the children had gained a bit of neuroscience knowledge, we jumped into reviewing the manuscript. Similar to how scientists approach reviews, we began with the positive feedback and strengths of the manuscript and then moved to questions, comments, and suggestions.

The Live Review event took place on April 29^th at the 2023 International Conference on Learning and Memory. Children arrived with their families an hour before the event so they could settle in and prepare a bit with the science mentors before the audience arrived. The event began with a brief presentation by the authors who shared about their inspiration for doing this type of work. The children then had the mic.

They began by sharing with the authors what they liked about the paper and then shared their questions, concerns, and suggestions. Some of the comments that stood out to me were, “I really liked the article, but I suggest you add a fun fact because whenever I see a fun fact, I’m like ‘Let’s go!’” Another was, “Your paper was really interesting, and it gave us a lot of good information…however I do have two suggestions. My first suggestion is that the paper should have more picture—like pictures showing all of the different tools a scientist might want to use. My second suggestion is that you could be more specific with some examples.” They commented on the choice of examples, the complexity of some paragraphs, and the need for examples that are relevant to children. After the comments, the children deliberated and then announced their decision: revise and resubmit. They’re excited to see the final paper published on the Frontiers for Young Minds website.

In addition to the Live Review, the session also included a Brain Festival where children had the opportunity to showcase what they learned during the sessions leading up to the conference. Scientists, families, and children explored preserved human and sheep brains, participated in hands-on activities,  and learned about rare neurological diseases from high schoolers. Watching a fourth grader describe the lobes of the human brain to a neuroscientist at a neuroscience conference is quite an experience.   

You evaluated all the groups who were involved with the Live Review. What was the feedback from the scientists regarding what they could do to more meaningfully engage?

Our evaluation of conference attendees showed us that most of the scientists at the conference were unaware that Frontiers for Young Minds existed. Sixty-four percent said it was their first exposure to the journal, and they didn’t know there was an opportunity for them to write articles for young students about their work. After attending the event, they reported high satisfaction and felt like the session was a good model for community engagement. Many said they could envision themselves being involved with similar efforts in the future.

The scientists who authored the manuscript felt the feedback from the kids was valuable and admitted that some of the suggestions they would have never considered. For example, the manuscript included an example of someone drawing a bowl of fruit. One of the kids told the scientist, “Yes, we draw, but I’ve never seen a fourth grader draw a bowl of fruit. Your example needs to be more relevant. Here’s a list of things kids actually draw—” So often, we don’t think about the importance of a relevant example until we get that kind of direct feedback.

Another concern was about insufficient figures and images. The manuscript included magnetic resonance imaging but no information about the meaning of the colored blobs. The kids really wanted to know. Scientists often underestimate how much kids really care; how curious they are about the details, and how hungry they are for knowledge.

And the kids?

Generally, the kids were very satisfied with the program. We asked about their feelings on interacting with real scientists, what they thought of the neuroscience activities before the Live Review, and what it was like to go to a scientific conference. We also asked whether their feelings about science had changed since participating in the program. I didn’t expect the kids to notice any sort of change, but the data shows their interests increased after participating. They looked for more science-related information, are excited to learn more about neuroscience, and want to participate in neuroscience programs in the future.

You evaluated the Live Review in 2018, too—what new measures did you add this time around?

In 2018, we focused on measuring the impact the program had on the child participants. Since then, we’ve taken the more social-ecological approach to understand how these programs impact the whole neuroscience ecosystem with the inclusion of feedback from parents, teachers, and scientists. We actually had one parent who said they worked in a scientific field in the country they grew up in, but not here in the US. They said coming to the conference was a dream come true and a great family experience.

What is the importance of having an IRB-approved study to collect this data?

Most science outreach programs evaluate outcomes to inform program improvement and not to disseminate findings. Often, they collect data on the number and characteristics of attendees and general satisfaction with the program. If surveys are sent to adults, they can be considered exempt from IRB approval; however, surveys to minors require approval, which is often a barrier that prevents program evaluation data from being shared with the field. Quantitative and qualitative program evaluation studies are immensely valuable to inform resource allocation and future program design. High-quality program evaluation is becoming more common as funding agencies are beginning to require this component. This is great news as there is so much to be learned from the successes and challenges of past programs.

Were there any specific findings from your evaluations that could help future outreach activities be more engaging for children?

We know that we should include more community engagement in conferences and scientific events, especially those that receive federal funds. But, far too often, this community engagement piece is a keynote lecture that’s tagged onto the conference as an afterthought, intended to check a box. The event is often not advertised appropriately to the target audience and is inaccessible to the communities who often feel like outsiders if they do attend.

We scheduled the Live Review smack dab in the middle of the conference on a Saturday morning with no competition from parallel sessions. It was in a primetime spot and listed as a plenary session. Thanks to a grant from the Dana Foundation, we offered complimentary parking to all community members as well as a lunch for the scientists and families. The families and scientists sat together and had a great time.

We have to start being more intentional about science outreach and treating it with the same kind of rigor with which we approach science. We can use our scientific skills to make our outreach efforts better by collecting data and learning from our own experiences and from each other. If we can build bidirectional bridges between neuroscience and society, making science more accessible to the community and allowing the voices of the community to influence science, we can build a future neuroscience workforce that better reflects our society—only then will we be prepared to tackle the greatest challenges.