The Impact Meter: Assessing Student Mindset

The Impact Meter: Assessing Student Mindset

"The main point of this evaluation effort is that it is not limited to one course or program -- it is a way to measure and evaluate interventions designed to increase the EM of undergraduate engineering students across courses and programs."

- Samantha Brunhaver, Assistant Professor, The Polytechnic School

Case at a glance

Integration goals

This effort focuses not on integrating EM into a class or program but on the development and deployment of an assessment framework and instrument called the Engineering Student Entrepreneurial Mindset Assessment (ESEMA). 

Materials affected

Framework defining EM mindsets and behaviors for each of the 3C's, survey instrument to measure the entrepreneurial mindset (EM) of undergraduate engineering students, and two journal papers

Lessons learned

Through this process, we discovered the variety of ways researchers are choosing to define EM and to use it as a tool for evaluating classroom and program-level activities.  The process also helped us to more clearly define the scope of our framework and instrument, in an effort to both align with the 3C’s and also to differentiate our work from that of others. Sharing this work with the broader engineering entrepreneurship community also provided valuable feedback on how to best articulate and define the constructs underlying the instrument, to facilitate their integration into engineering curricula.

Context

This case study does not focus on integrating EM into a class or program but on the development, implementation, evaluation of, and reporting on an assessment framework and instrument based on the 3C’s. It is called the Engineering Student Entrepreneurial Mindset Assessment (ESEMA), although students see a more generic name (the Engineering Student Mindset Survey) when taking the survey to avoid biasing their responses. It has been administered five times, twice at E2 Camp, once to students enrolled in the second-year design project course (EGR 201), once to all FSE first-year students, and once to all FSE senior students, both to assess and improve survey effectiveness (that is, to ensure its validity and reliability).  Now that scholarship on ESEMA has been published, it will be deployed more broadly. 

Conceived of and developed by members of the Polytechnic School’s KEEN team and based  on  a  review  of  the  literature as well as the team’s own prior research, the assessment framework and instrument were drafted following a rigorous multi-stage process using multiple populations of ASU engineering undergraduates. The instrument is being used to measure and evaluate the impact of interventions designed to increase EM among students. The Engineering Student Mindset Survey comprises 34 items and is designed to be completed in about 20 minutes. The instrument's development organically influenced some courses within FSE's curriculum, as well as FSE’s program-level evaluation of the EM initiative. 


The ESEMA played a small role in the graduate course EGR 572: Quantitative Methods in Engineering Education Research. During the fall 2016 and 2017 semesters, students were instructed that a research team was planning to develop an instrument for measuring the EM of undergraduate engineering students, and asked to draw and write out a process the research team could use to create the instrument. Students worked on this activity for 5-10 minutes before pairing up with a partner and discussing for another 5-10 minutes. Separate from the class activity, a version of ESEMA was used as the basis of a homework assignment around exploratory factor analysis; students analyzed which items on the modified version of the instrument cohered as measurable factors (for example, empathy and persistence). On the final exam, students were also asked to list the steps they would use to develop an instrument to measure entrepreneurial mindset among undergraduate engineering students.

Integration details

Details of the integration effort are summarized in the timeline that follows.

2016
Early spring: Framework development
Late spring: Item writing, administration of survey to FSE 100 faculty to help validate the behaviors in the framework (14  surveys  collected--faculty were asked to rate how often they engaged students in, and how much importance they ascribed to, each behavior )
Summer: Pilot administration of survey instrument to E2 Camp attendees for incoming freshman engineering students and subsequent analysis and item revision; presentation of the framework and plans to measure EM mindsets and behaviors using the framework during the KEEN Assessment Working Group at the ASEE Annual Conference
Fall: Administration of survey instrument to EGR 201 students and subsequent analysis and item revision
 

2017
Early winter: Framework shared as part of a workshop at the 2017 KEEN Winter Conference 
Spring and Summer: Authorship of framework and instrument articles 
Summer: Dissemination of survey instrument to select partner institutions, administration of revised survey instrument to all  and subsequent analyses
Fall: Test-control evaluation of E2 Camp activity around EM using instrument 


2018
Spring: Administration of revised survey instrument to all FSE seniors
Fall: Analysis of final results; publication of two related articles:

  • Brunhaver, Samantha R., Bekki, Jennifer M., Carberry, Adam R., London, Jeremi S., McKenna, Ann F. (accepted, to appear Sept. 2018). Development of the Engineering Student Entrepreneurial Mindset Assessment (ESEMA), Advances in Engineering Education.
  • London, J., Bekki, J.M, Brunhaver, S., Carberry, A., & McKenna, A. (accepted, to appear Sept. 2018)). A framework for entrepreneurial mindsets and behaviors in undergraduate engineering students.  Advances in Engineering Education. 

Summer/Fall: Authorship of publication on EM within FSE

In sum, the core tasks of this integration effort included the development and validation of a survey instrument to measure EM among undergraduate engineering students. Subtasks included establishing a framework for defining EM mindsets and behaviors for each of the 3C's, writing items for each of the EM mindsets, deploying the survey instrument with multiple populations of ASU engineering undergraduates, and performing exploratory and confirmatory factor analyses to evaluate the underlying factor structure of the survey items. The core deliverables of the effort were: (1) a framework defining EM mindsets and behaviors for each of the 3C's, (2) an instrument to measure the entrepreneurial mindset (EM) of undergraduate engineering students, and (3) two journal papers describing the framework and the instrument.

NOTE: Supporting resources for this case study can be found within its companion KEEN card (link below), which is also where the community can discuss the case and its broader topic.

Integration outcomes

Some details of this effort’s outcomes are available in the published articles, which may be accessed from this case study’s corresponding KEEN Card, while others remain forthcoming. This effort dovetails with FSE overall evaluation of the integration effort.

Future plans

We plan to further disseminate the instrument to other institutions and may publish additional research articles. We also plan to work closely with the KEEN Team’s research scientist, Dr. Gary Lichtenstein, who is leading evaluation of the overall EM integration effort.

Considerations

Through E2 camp, EGR 201, and the FSE first-year and senior programs, we have been fortunate to find courses and programs that we can leverage for our instrument development process, although this effort included the usual challenges of instrument development such as the ability to pull in sample sizes large enough to conduct analyses, the ability to pull in multiple samples to test generalizability of the instrument, and the need to ground items in theory and prior work. Having a team of researchers (Samantha Brunhaver, Jennifer Bekki, Ann McKenna, Adam Carberry, and Jeremi London) made the challenges more manageable.