Institutional Learning: Evaluating the Initiative
Institutional Learning: Evaluating the Initiative
"It's not as hard as it may seem to define EM and devise credible, elegant, non-intrusive program-level assessments. If there's the will, there's a way."
- Gary Lichtenstein, Assistant Research Scientist, KEEN Team
Case at a glance
Integration goals
The goal of this effort is not to integrate EM but to assess the effectiveness of FSE’s overall entrepreneurial mindset initiative. During the initial phases (2016-2018), we focused on the following questions:
To what extent have faculty and students been exposed to EM?
How do faculty, staff, and students understand entrepreneurial mindset?
How important do faculty and students believe that EM is to an engineering degree?
What are the customer segments and value propositions of EM to FSE faculty and students?
Materials affected
Surveys and related assessment instruments, data analysis tools, conference presentations, thought leadership publications
Lessons learned
Although we have barely begun, we have already learned several lessons: EM provides a context within which to teach user-inspired design; EM value propositions appeal to many faculty and students; some resistance emerges when faculty and students understand “entrepreneurial mindset” as venture creation; and truly institutionalizing EM throughout FSE will first require a commonly adopted framework that defines EM and EM outcomes that will be consistent across programs. Collecting data from students and faculty can be extremely challenging.
Context
Dr. Gary Lichtenstein was hired by FSE in December 2016 to assess the EM initiative. Working with the University Office of Evaluation and Education Effectiveness (UOEEE), Dr. Lichtenstein now heads an evaluation team dedicated to assessing the initiative.
In the early phases of the initiative, assessment was predominately head counts of participants, as well as some brief evaluation surveys of specific activities. Although EM was presented as the KEEN 3Cs, no specific or common EM outcomes were specified across curricular and co-curricular activities, nor often within them. Instead, the initiative was characterized by several innovative pilot initiatives, predominantely located in the Polytechnic School and the School of Biological and Health Systems Engineering, although students and faculty participated from across FSE.
Reflecting an important shift, the team’s evaluation research now focuses on identifying faculty and students’ exposure to, understanding of, and perceived importance of EM in engineering, with the goal of establishing a baseline for institutionalizing EM across FSE beginning academic year 2019. The research questions were designed to assess not only current coverage of EM but also to explore customer segments and value propositions that might facilitate or impede widespread adoption. Surveys were collected from over 300 students and 60 faculty, and 20 faculty and 60 students were interviewed. The evaluation team also conducted a syllabus study to assess EM coverage in EM-focused courses.
The table below shows EM initiative activities throughout 2017- 2018:
| Curricular Activities of Interest | Faculty EM Development | Co-Curricular Activities of Interest |
|---|---|---|
| Intro to Engineering (FSE 100, EGR 101, EGR 102, EGR 201) | KEEN Professorships | E2 Camp |
| BME Design Spine | Pilot Faculty Mentoring Program | Devils Invent |
| Entrepreneurship and Value Creation (FSE 301) | Engineering Faculty Impact Collaborative (EFIC) | Engineering Student Organizations |
| Engineering Futures/FUEL (FSE 194) | Fulton Undergraduate Research Initiative (FURI) | |
| Graduate Research (EGR 535, EGR 572, EGR 673) | Venture Devils | |
| Engineering Projects in Community Service (EPICS - FSE 104) | Student Mini-Grants | |
| Grand Challenge Scholars Program (FSE 150) |
Integration details
The first stage of this effort had seven milestones on a timeline that started in spring 2017 and concluded at the end of August 2018, with some milestones overlapping as shown in the chart below:
The table below provides a summary of the questions being used to establish this effort’s baseline:
| Faculty | Undergrads & Grads | Institution |
|---|---|---|
|
|
|
Academic Year (AY) 2017-2018 Data Collection
The AY 2017-18 evaluation comprised five streams of data:
- EM General Survey. This survey asked FSE students: a) have you heard about EM?, b) to what extent have you participated in EM-related activities? c) do you believe that EM is important to an engineering degree? This survey was initially deployed fall term through sophomore level courses across FSE. However, the response rate was very low—fewer than 30 surveys were collected using this approach. In spring, members of the evaluation team approached student groups and students one-on-one. In this way, over 200 surveys were collected.
- EM Interviews. All faculty and staff interviews were conducted in the fall term. Participants were recruited using a snowball technique. These faculty (n=17) and staff (n=9) were predominately (though not exclusively) involved in EM initiatives, skewing the data positively. Although the evaluation team gathered the email addresses of 145 students who expressed a willingness to be interviewed, only 13 actually participated. In spring term, the IRB was modified and students were offered a financial incentive ($25) for sitting for interviews. This strategy was effective, and 50 more interviews were conducted. Students represented a range of exposure to and understanding of EM.
- Course Syllabi Study. This study examined evidence of inclusion of EM as learning objectives, course activities, and graded assignments in the syllabi of EM-focused courses and KEEN Professorship courses. The evaluation team received 23/38 syllabi from faculty—a 60% response rate.
- EM Course Coverage Surveys. This survey was deployed to all faculty teaching EM-related courses in fall and spring, including KEEN Professors. The fall survey asked faculty to identify the extent to which EM, EM concepts, and soft-skills were integrated into out-of-class activities, in-class activities, and graded assignments. Based on fall results, a modified survey was distributed to faculty in the spring. This time, a parallel survey went to students of those faculty. Of 67 faculty who received surveys, 24 responded (a 36% response rate). We had useable surveys from students in 10 of those courses. The Evaluation Team analyzed surveys from those 10 courses that provided student as well as faculty data on EM coverage.
2018
We closed the following surveys and interviews and spent the summer analyzing and reporting:
- Student survey regarding exposure to EM in courses and extracurricular activities (n=~215)
- Student survey to students in EM-related courses regarding their perception of EM coverage in the course (n=~146)
- Faculty survey to those teaching EM-related courses regarding their perception of EM coverage (n=~26)
- Interviews of FSE students regarding their understanding of and attitudes about EM (n=~63)
NOTE: Supporting resources for this case study can be found within its companion KEEN card (link below), which is also where the community can discuss the case and its broader topic.
Integration outcomes
The data we have collected will be used to establish a baseline of EM coverage in EM-focused courses, exposure of students and faculty to EM, and perceptions of the importance of EM to an engineering degree. As we move towards institutionalizing EM across 17 Accreditation Board for Engineering and Technology (ABET)-accredited programs, our data will facilitate the following activities:
- Creating a framework for defining EM and EM outcome competencies across FSE
- Articulating guidelines for accurately representing EM in course syllabi
- Promoting graded EM assignments
- Emphasizing to faculty and students the value that EM adds to engineering work and industry and society
Future plans
At the time of this case’s publication, we are still analyzing survey and interview data collected during 2017-2018 and compiling a comprehensive report that summarizes
faculty and student exposure to and attitudes towards EM, as well as coverage of EM in EM-related courses. Data will be used to create an Entrepreneurial Mindset at Fulton Schools of Engineering (EM @ FSE) Framework, which 1) defines Entrepreneurial Mindset (EM) at Ira A. Fulton Schools of Engineering (FSE); 2) outlines a rationale for our unique approach to EM @ FSE, and 3) articulates FSE-wide outcome expectations and metrics to be implemented across FSE’s six schools and 17 ABET-accredited programs.
Within FSE, the EM @ FSE Framework will be communicated to faculty through an adaptation of the Just in Time Faculty Development (JTFD) model. Designed by engineering faculty (Drs. Stephen Krause and Keven Hjelmstad) through a three-year NSF grant, JTFD has been highly rated by two cohorts of FSE faculty. In the institutionalization phase, all new FSE faculty, and those teaching EM-focused courses (Introduction to Engineering, Senior Capstone, and one other required course) will attend JTFD workshops and learn the EM framework, including Outcome Competencies and the common assessment rubric.
Our continued involvement in the KEEN Assessment Working Group (AWG) informs our process and at the same time becomes a means by which to share our approach throughout the network. Ultimately, this next phase of EM institutionalization will rely on an evaluation team comprising FSE leadership and the UOEEE. Using the common definition of EM and program-wide outcome competencies, the evaluation team anticipates being able to assess the effectiveness of implementation of EM as well as the outcomes.
Considerations
We anticipate some challenges in getting broad-based buy-in to EM @ FSE. Even among those most disposed towards integrating EM into FSE degree programs, building consensus around a definition of EM, outcome expectations, and metrics, will raise differing points of view. These differences are to be expected with any major reform in a large, higher education institution, and our AY 2018 data have helped us anticipate the resulting challenges. Key features of EM @ FSE are designed to mitigate the expected challenges and include 1) focusing on Design Stage innovation development (from problem recognition to prototype/proof-of concept), which aligns with FSE’s use-inspired engineering design model, 2) emphasizing that EM is a means of adding value to industry and/or society and does not necessarily mean venture creation, and 3) allowing faculty discretion in designing assignments that meet outcome expectations (although a common assessment rubric will be devised). Finally, the Evaluation Team will need to work closely with FSE leadership and course faculty to promote completion of evaluation instruments—targeted to both faculty and students. Without high response rates to surveys, the effectiveness of implementation and outcomes can not be reliably assessed.