Extending the Student Qualitative Undertaking Involvement Risk Model

ABSTRACT:

The Student Qualitative Undertaking Involvement Risk Model (SQUIRM) was designed to facilitate the determination of the impact of using student (or inexperienced) workers, on a project. The model identifies several prospective categories of risk. It, then, discusses the risk potential and source and provides a limited consideration of how to mitigate this risk. The risk sources considered included those specific to student (inexperienced worker) involvement, standard risks and standard risk sources which are enhanced by the use of student (inexperienced) workers. This paper presents a qualitative assessment framework and begins the process of quantifying the model. The difference between the use of students (in an academic or industrial setting) versus inexperienced workers is also considered. The base model is presented and extended by further tracing the risk sources back, using root cause analysis techniques. The application of the base and extended models to various projects is discussed. Considerations in choosing which model to use for a given application are also presented. The paper concludes by presenting a value model for considering student (inexperienced worker) involvement benefits versus associated risks, and the differences in the risk reward ratio between academic, internship and junior worker scenarios.

KEYWORDS:
Risk model; Student workers; Small spacecraft; Aerospace education

INTRODUCTION

Student involvement in research and other projects is common at universities around the world. Through internships, part-time work and other mechanisms, students also perform limited work for commercial, governmental and other employers. Despite the prevalence of student involvement in the development of key technologies and their performance of numerous duties, the management literature contains little consideration of the specific risk elements introduced by student workers. Inexperienced workers (including students, interns and junior employees) have particular characteristics that may create new risk sources and alter the likelihood and magnitude of typical risks.

An understanding of the impact of using student (and other inexperienced) student workers is particularly important in the case of aerospace projects due to the low defect tolerance, inaccessibility and criticality of many projects. Small spacecraft, for example, are commonly integrated as secondary payloads on rockets carrying other orders-ofmagnitude more expensive hardware. They must meet the same (or perhaps even more stringent) integration standards as the primary payload. Some small spacecraft have also been launched via the International Space Station, necessitating their compliance with human safety standards. Once they are in orbit, they are also on their own, with no practical servicing capability. Design and implementation failures can, thus, cause a spacecraft to fail integration testing and not get launched, to fail subsequent to integration and damage expensive equipment or pose a threat to astronauts or fail on orbit, impairing mission performance. The training and the research provided by these efforts is integral to developing new technologies as well as training the next generation of aerospace professionals. Given this, a better understanding of the risks posed by student and inexperienced staff involvement is necessary.

In prior work (Straub et al. , 2013b), a risk model specifically targeted at students (and to some extent, at all inexperienced workers) was presented, called the Student Qualitative Undertaking Involvement Risk Model (SQUIRM). This paper, while discussing and classifying numerous types of risk, did not (due to space limitations) present an evaluation of the model nor demonstrate its application to any particular scenarios.

This paper picks up where the previous work left off. It expands on the prior work in several ways. First, it presents an enhanced model that augments the base SQUIRM framework with root cause analysis, resulting in a more detailed consideration of student status on typical (non-student) risk factors. The use of this model can provide a more robust evaluation of the impact of student participation, as compared to the base model. However, it is not a panacea, and prospective tradeoffs between the use of the two approaches are discussed. Second, it begins the process of quantifying the SQUIRM and extended SQUIRM frameworks, discussing how the models can be used in order to assess risks (considering likelihood, impact and the mitigation techniques employed) on a single project basis or across multiple projects. Third, it presents a value model for evaluating the participation of student (and other inexperienced) workers. This model facilitates the determination of the value proposition of using this type of staff, which can be compared to increased risks and other associated costs. Finally, the differences between types of inexperienced workers are briefly discussed, before concluding.

BACKGROUND

This section provides an overview of areas that the current work benefits from a wealth of prior work in. Despite a growing contemporary interest, the tasking of trainee or inexperienced workers to real-world projects is certainly not a new phenomenon. Apprenticeship-style training has been used throughout history (Elbaum, 1989; Snell, 1996). Modern approaches, however, combine formal and experiential techniques. One relevant technique is project-based learning.

In the remainder of this section, the benefits of project-based learning are, first, discussed. Next, prior work, regarding assessment of the value of students to faculty efforts, is briefly considered. Finally, a brief discussion about risk perception is presented.

PROJECT-BASED LEARNING

With project-based learning (PBL), students are involved in hands-on projects that could be developed specifically for a course or which might feature student involvement in faculty research or other real-world projects. PBL has been shown to be an effective instructional tool at all levels of education: from collegiate graduate-level to primary school level (Brodeur et al. , 2002; Hall et al. , 2002; Mathers et al. , 2012; Mountrakis and Triantakonstantis, 2012; Nordlie and Fevig, 2011; Straub et al. , 2013a). It has also been demonstrated across a wide variety of subject disciplines, including project management (Pollard, 2012), psychology (Dahlgren and Dahlgren, 2002), physics (Duch, 1996), computer science (Broman et al. , 2012; Correll et al. , 2013), mathematics (Roh, 2003), engineering entrepreneurship (Okudan and Rzasa, 2006) and aerospace (Jayaram et al. , 2010; Saunders-Smits et al. , 2012), computer (Qidwai, 2011), electrical (Bütün, 2005; Ribeiro, 2008) and mechanical (Coller and Scott, 2009; Robson et al. , 2012) engineering.

In addition to teaching subject-specific skills, PBL projects can teach students how to work with those outside their specific discipline, as is required in the vast majority (Hayne et al. , 2012) of workplaces. Gaining a shared prior knowledge base (such as through PBL techniques) can improve team efficiency (Hayne et al ., 2012). Workers with interdisciplinary skills are in demand (Sulaiman et al. , 2010); PBL also provides students with an opportunity to learn "soft" skills which are required for workplace success (Jackson and Hancock, 2010).

PBL has also been shown to have a beneficial impact on student motivation (Doppelt, 2003), self-image and creativity (Ayob et al. , 2012) and material retention (Bauerle and Park, 2012). Field-based/realistic-environment PBL has been shown to increase students' understanding of course materials (Simons et al ., 2012). Nagda et al . (1998) show that one type of PBL, research participation, can also improve student retention, particularly for at-risk students. The benefits of PBL to student placement, after graduation, have been demonstrated by Hotaling et al ., (2012) and Fasse et al., (2012). Gilmore (2013) even argues that techniques such as PBL, for teaching STEM disciplines, are critical to national prosperity.

In aerospace engineering and related disciplines, many students are gaining practical experience working on small spacecraft and high altitude ballooning projects. The SQUIRM framework (Straub et al. , 2013b) was created, initially, to assess the risks applicable to student involvement in a small spacecraft project; however, it is useful for many applications beyond this. The utility of PBL for teaching aerospace engineering (Straub et al. , 2013a; Straub and Whalen, 2013), software development for aerospace applications (Straub et al. , in press) and providing other benefits (Swartwout, 2004; Swartwout, 2011) has been demonstrated. CubeSat projects have been demonstrated to be an effective pedagogical approach (Larsen and Nielsen, 2011; Larsen et al. , 2013; Straub, 2013).

The level of the aforementioned benefits, Zydney et al. (2002a) proffer, increases with the duration of participation. However, not all students reach these higher levels of benefit; while numerous reasons for premature termination of student participation in a research project exist, manifestation of the risk factors discussed in a subsequent section may explain some of the incomplete experiences.

VALUE OF STUDENT INVOLVEMENT TO FACULTY RESEARCH

If student involvement's benefit was solely student education, the need to characterize and mitigate risks would be dramatically reduced. The impact of a student/ inexperience-specific or general risk factor's occurrence can have impact to the student participant's success; it can also have a pronounced effect on the project as well. While students may gain (possibly even enhanced) benefit from risk actualization, the project stands to suffer. To characterize the magnitude of impact, it is important to consider faculty perceptions of student involvement on research projects. Zydney et al . (2002b) proffer that faculty see students' participation as valuable, with over half of them indicating that students' contribution to their work was "important" or "very important". Thus, the failure of a student to make progress is a risk that may be comparable to causing damage or other types of impact on prior work. While student participation is valuable to faculty, it appears that project completion may be less important to students, as Prince et al . (2007) demonstrated a lack of correlation between the research productivity level of faculty and students' educational benefits.

RISK PERCEPTION

One reason that student workers may be more riskoccurrence prone is a failure to properly assess risk likelihood and impact. However, despite a significant correlation between youth and inexperience, it is important to note the potentially confounding impact of risk perception. Because of this, there may be a performance difference between younger and older individuals with similar experience levels in a field. A full exploration of the topic of risk perception is far beyond the scope of this paper; however, reviews of areas of this topic are readily available. Botterill and Mazur (2004) provide a general overview of the topic, while Slovic et al . (1982) consider the value of studying it. Boholm (1998) reviews and compares risk perception research over a twenty-year period and Mitchell (1995) considers risk perception and risk reduction in the context of an organization.

The crux of the risk perception problem is that younger individuals may fail to appreciate the applicability of risk to them and its impact (Weinstein, 1984). This has been documented across multiple areas, including driving (Deery, 1999), sexual (Levinson et al. , 1995) and other "healththreatening" (Cohn et al. , 1995) behaviors. Steinberg (2004) attributes the greater risk-taking tolerance of youth to "age differences in psychological factors that influence self-regulation". Thus, age may confound the experience/ risk correlation, and intensify certain risk factors when both young age and inexperience are the case. Given this, traditional-age undergraduates may have a higher propensity to fail to see how their actions, behaviors or inaction may create risks, or the impact that these risks may have on them or others.

Risk perception, however, is not only affected by age. Correlation has been shown with gender (DeJoy, 1992), culture (Rippl, 2002) and other factors (Sjöberg, 2000; Wildavsky and Dake, 1990). The impact of education in correcting risk perceptions has been demonstrated by Ronan and Johnston (2001). Weber and Milliman's (1997) work suggests that "risk preference" may be a stable aspect of an individual's personality, highlighting the importance of risk perception on the acceptance or rejection of the risk in a given circumstance. Renn (1998) discusses the importance of risk perception in relation to the management of risks.

THE STUDENT QUALITATIVE UNDERTAKING INVOLVEMENT RISK MODEL

The following subsections, reprinted with minimal modification from Straub et al . (2013b), provide an overview of the risk categories of the SQUIRM framework (which is depicted in Fig.1). First, technical, schedule and other standard risks will be discussed. Then, the risks posed by student worker involvement will be considered.

Figure 1
SQUIRM Model Diagram (Straub et al., 2013b).

TECHNICAL, SCHEDULE AND OTHER STANDARD RISKS

Every project, including those involving students, must deal with numerous possible risk factors. Project managers attempt to control many of these risk factors, assume others, and they are, ultimately, forced to ignore a large set of risks that they have no insight into or control over. Numerous standard risks are well documented in the literature and will not be reviewed in detail here. The impact of student participation on these standard risks is considered. For each risk factor, a brief description of its nature is provided. This is followed by a discussion of how the risk factor is influenced by or may influence student project involvement.

TECHNICAL RISK

The technical risk category is comprised of the set of risks that could result from a failure of hardware and software or its integration and operations to perform as required to meet project's objectives. Three aspects are considered: construction/fabrication of assemblies, failure of purchased components and their integration.

Construction/Fabrication

Construction and fabrication risks are inherent to any manufacturing process. Quality control processes, including those designed to prevent defects as well as those to detect and remediate defects, are generally included to mitigate these risks. In a student project, which generally doesn't involve mass-production, one is confronted with two primary risks. First, standards-based quality control may be cost-prohibitive to implement. Second, students who lack knowledge and understanding of the characteristics of the product may be poorly equipped to detect and evaluate the significance of errors.

Component

Components obtained from suppliers will occasionally be defective, either due to manufacturing or shipping issues. Production processes generally incorporate an acceptance testing procedure or supplier process validation procedure. A student-involved project, generally, suffers from two risk factors regarding components. First, the limited production (in many cases, producing only a single or small quantity of units) precludes the implementation of a standard quality process. Second, student inexperience may result in a failure to properly design acceptance tests or to detect latent issues.

Integration

The process of combining components together introduces risks due to design and implementation failures. Design failures may result in a system, which, regardless of how well it is assembled, cannot perform the desired task. Implementation issues may result in degraded performance, non-operation, or failure after a period of time operating. Student designers and workers generally have traits that significantly increase the probability of these risks happening. Having an incomplete or largely untested understanding of the design process or specific design elements may result in wholly unworkable designs or designs with latent and hard-to-detect flaws. Limited time and resources will generally result in a comparatively lower level of testing being conducted. The fact that this testing will likely be performed by inexperienced (student) testers further exacerbates the problem. Even if a perfect design is produced, inexperience in the techniques required for construction may result in sub-par construction, component attachment and solder connection issues, and so forth. These may cause the assembly to not work initially or to be prone to failure.

SCHEDULE RISK

Every project faces the possibility that its schedule will not be met. External factors, such as the unavailability of key components, and internal factors, such as staff absences or equipment failure, may result in delays. When these delays impact the critical path, the project schedule is impaired. Key areas of consideration for projects involving students include schedule estimation error, critical path risks and schedule creep.

Schedule Estimation Error

Estimation error occurs when the time projected for task completion is different than actual task completion. A certain amount of error is to be expected; however, when tasks are consistently taking longer than projected, the project's schedule is at risk. Estimation error is common, even for experienced estimators. Students, who do not have significant experience, may fail to consider anything other than the best-case scenario. Alternately, they may not completely understand the process that they are estimating and, thus, omit the time required for overlooked process components. Either of these may result in (possibly dramatic) under estimation. On the other hand, students may be overwhelmed and wildly overestimate, (so as to avoid the pitfalls of underestimation). This is, however, problematic, as it may result in the project's momentum being lost, if materials, tools or staff for subsequent phases are not available when a previous phase is completed early.

Critical Path Risk

Critical path risk is a set of risk factors that impact the chain of tasks, which, in succession, take the longest amount of time. As the project is not complete until all of these tasks are done, anything that elongates the schedule of a task on the critical path (or another task, which becomes a critical path task due to schedule overrun) affects the project's overall schedule. Critical path risk can be created by factors that are both external and internal to the project. External factors may include impairment to the availability of supplies, unavailability of key equipment at the needed time, changes in laws or regulations and many other factors. Internal factors, however, are the primary area where projects with student involvement differ from conventional projects. Internal issues that may be exacerbated by student involvement include staff availability issues, delays caused by quality failures - and, thus, the need to repair or recreate the improperly produced items -, and delays caused by poor scheduling. Staff availability and quality issues are discussed in other areas of the model. Poor scheduling may be the result of a failure to identify precursor and successor tasks due to failing to identify required task inputs and outputs or, more simply, error in the actual creation of the schedule. Either of these can easily occur when a schedule is produced by an inexperienced scheduler.

Schedule Creep

Schedule creep is the schedule component of scope creep. Scope creep occurs when changes or documentation issues result in a more robust product being produced than the one called for by planning. The involvement of students, who are generally eager to please and may not understand the impact of accepting changes (or not understand that they are implicitly accepting a change), increases the risk of schedule creep. The fact that most academic projects are run by professors who are trained as researchers - not project managers - and may have limited documentation further exacerbates this risk.

COST RISK

With tight budgets and long-duration funding cycles, cost overrun is a significant risk to student-involved projects. Cost overruns can lead to reduced deliverable utility and/or quality. If severe enough (and supplemental funding cannot be sourced), they can even lead to project termination and failure. Risks that must be considered relative to student involvement include estimation error, cost creep, damage and rework costs, and costs associated with meeting schedule requirements.

Cost Estimation Error

Cost estimation error closely mirrors schedule estimation error. It occurs when the level of cost required to be incurred for a given activity is different from the level forecast. While variation is expected, proper estimation should result in some tasks concluding with small overruns and others being completed under budget. Generally, an allowance for unexpected costs is included in the budget as a separate line item to allow the absorption of additional costs, should the project average out to a slight overrun. As with schedule estimation error, students who may be estimating costs for the first time (or may have limited domain experience, even if they have performed cost estimating before) may be prone to underestimate, due to ignoring complexity or inadvertently omitting various types of costs or specific costs.

Cost Creep

Cost creep is the cost component of scope creep. Scope creep occurs when changes are accepted without commensurate changes in budget and schedule. Due to student inexperience and other factors, scope creep is likely on student projects. If scope creep occurs, it is likely that cost creep will occur.

Damage and Rework

Damage and rework costs are incurred when hardware, facilities, supplies or the item being created are damaged due to carelessness, accident, misuse or otherwise. Damage and rework costs are likely on a student-involved project. First, the lack of a production environment designed for the repetitive production of an item means that construction and integration jigs will be setup on the fly. This may result in inadvertent loss of control, dropping, or the application of unwanted torques or pressure to parts or assemblies. Second, the lack of a repetitive production environment means that there is not a set of well-tested task instructions that can be followed. Third, supply and equipment limitations may result in jury-rigging of various jig-elements, making damage more likely. Forth, horseplay or carelessness may result in damage. All of the aforementioned are exacerbated by having young and/or inexperienced individuals working on the project.

Buying Time

Costs can be incurred to resolve schedule issues. For example, a component could be purchased, at additional expense, to return the project to schedule or an external consultant could be hired to expedite a process. Due to this, schedule issues can become cost issues. Given how student involvement can exacerbate schedule risk, it would seem that student involvement would heighten the possibility of transferring schedule overruns to cost in order to hit a key deadline.

RISKS POSED BY STUDENT WORKER INVOLVEMENT

Several risk factors are impacted so dramatically by student involvement as to deserve separate consideration from their standard counterparts. Each is now discussed in detail.

Scheduled Turnover

Scheduled turnover has a dramatic impact, but can be planned for. It is attributable to the fact that students only participate in a given effort for a period of time. When this participation ends the student may be unavailable to provide documentation or assistance related to their work on the project. As students become task-experts, if documentation is not stressed, understanding can be lost - or a key component of an integrated system can become unserviceable. Compounding this issue is the fact that many students are not adept in documenting their work and lack an understanding of the need for documentation and what needs to be documented. Mitigation strategies for this risk include knowledge distribution, stressing documentation throughout a project's lifecycle, and validating the usefulness of documentation, by requiring its use prior to a student-worker's departure.

Unscheduled Turnover

Unscheduled turnover is a risk factor present in all types of organizations. As in corporate work environments, medical, personal and other factors may necessitate a worker's immediate departure from the workplace. Mitigation techniques for this class of risk include duplication (or responsibility distribution) of key roles, wide knowledge distribution, and stressing documentation and documentation validation.

Miss-commitment

Students' miss-commitment can be more problematic than the occurrence of turnover. With turnover, the project leader has knowledge of the current status of the team member. With miss-commitment, the individual is still present and ostensibly working on their assigned tasks; however, due to conflicting demands for limited time resources (and the academic-trumping of most project duties) the student worker may not have time to make the requisite level of project progress. This is compounded by the cramming-centric work styles learned by many students, which lead to the belief that everything can be 'made up' at the last moment. With student miss-commitment, project leaders may not become aware of the issue, until investigating the cause of a key deadline being missed. Mitigation techniques for this class of risks include defining tasks to have demonstrable milestones, creating an environment where challenges are reported instead of obfuscated, and involving multiple individuals in key tasks.

Inexperience

Inexperience is, of course, a problem that is faced by numerous projects in every sphere. A team member may be new to the workforce, or may lack experience in the specific areas required by a project. However, inexperience is a particular issue in student-centric projects as many students lack practical experience. This translates into misestimating and a lack of experience in problem resolution techniques. This class of risks can be mitigated by training students in the desired behaviors (e.g., how to estimate in a given sphere, how to deal with problems, etc.). This mitigation not only benefits the project, but also prepares the students for workplace entry.

EXTENDING THE MODEL WITH ROOT CAUSE ANALYSIS TECHNIQUES

The original SQUIRM model, presented in prior work (Straub et al ., 2013b), expanded upon the causal factors for standard risks, which could be exacerbated by student/ inexperienced workers' involvement. While some discussion of the causality of the student-worker-specific risks was included, these were not incorporated into the formal model. The SQUIRM-Extended Model (SQUIRM-E) adds these causal factors to the model, as shown in Fig. 2. This addition is necessary to begin quantitative assessment using the model (which is discussed in a subsequent section). This section begins with a discussion of the value of the use of root cause analysis and then the rest of this section discusses the new elements of the SQUIRM-E model and expands upon the types of risks posed by them and their causes.

Figure 2
SQUIRM-E Model Diagram.

ROOT CAUSE ANALYSIS

The premise of root cause analysis (RCA) is that a better understanding of the underlying factors of an exceptional occurrence (either positive or negative) facilitate a better understanding of how negative occurrences can be avoided in the future and positive occurrences brought about. Significant prior work exists in this area; a high-level overview is provided by Rooney and Heuvel (2004). RCA has been used for process analysis (Weidl et al. , 2005), investigating medical error (Iedema et al ., 2006), improving patient safety (Neily et al. , 2003), as well as in analyzing and improving industrial safety and performance (Carroll et al. , 2002). A discussion of several tools for RCA was presented by Doggett (2004).

In the context of this work, RCA was used to assess why student-involved projects and student workers could have higher levels of risk actualization than a similar project not incorporating inexperienced workers. In prior work (Straub et al ., 2013b), this was applied to seek out causes that were specific to student (and other inexperienced) workers. In this paper, RCA is used to decompose standard risk factors to assess the prospective contribution of inexperience and related factors on these risk areas.

RCA is not the only technique that could be used to assess these types of risks. However, it has several benefits. Unlike some other approaches, for example, it uses a bottom-up approach which makes it suitable for projecting risks instead of analyzing actualized risks. This is particularly valuable in the context of non-operations risk analysis, where prior occurrences in a recurring process cannot be analyzed to project future risk factors and their likelihood. With RCA, the individual factors contributing to each type of prospective risk have been identified. These can, then, inform planning (in order to facilitate avoidance and mitigation) as well as be used to arrive at an understanding of the risk level of a project and its areas of particular risk. To perform RCA, prospective sources of the higher-level risk factors previously presented were identified. These are described in greater detail throughout the remainder of this section.

INEXPERIENCE SYMPTOMS OCCUR

The risk categories related to inexperience are a lack of attention to detail, lack of self-motivation, uncertainty as to how to perform a task, overconfidence that causes failure and problems with the work environment. These are now discussed.

Lack of Attention to Detail

Student workers may lack an understanding of the importance of particular details of a task, lack an understanding of the actual details (i.e., what is a correct implementation at a detailed level versus an incorrect one), or may simply fail to pay the level of attention required. This may be exacerbated due to other time commitments (reducing the amount of time that can be devoted to these details and task performance), the level of strain that the student is under (particularly if the student lacks coping mechanisms), and other factors (such as the amount of time available during the semester, etc.).

Lack of Self-Motivation

Students (particularly lower-level undergraduates) may not yet have developed the skills, habits and work ethic required to self-motivate work when tasks seem unexciting or are in support of a longer-term goal. This may translate into unsatisfactory performance in terms of meeting deadlines, unsatisfactory work product or other deficiencies. It may also trigger or contribute to other risk factors (such as misscommitment if work piles up due to not starting things until there is an imminent due date, etc.).

Unsure of How to Perform Task

Students may be unsure of how to perform particular tasks or elements of a task. This may translate into delays waiting for clarification, attempts that result in wasted materials and time, obviously defective products or products with latent defects that may impair progress during later phases (e.g., integration, testing). This lack of understanding may decrease motivation, increase frustration and delays may trigger other issues such as miss-commitment.

Overconfidence Causes Failure

Students may underestimate the difficulty of a task or overestimate their own capabilities. This can have several different symptoms, depending on when it occurs. First, it can cause issues with scheduling and costing. Students may underestimate the amount of time that will be required for learning how to perform a task, experimenting to gain understanding and/or correcting less-than-acceptable products. They may also underestimate the amount of waste material that may be consumed by reattempts to fix defects.

Second, it can result in unsatisfactory performance in terms of meeting deadlines, unsatisfactory work product or other deficiencies due to the aforementioned scheduling and the reality of performance conflicting, or a lack of understanding of what an acceptable product is, triggering a need for significant rework. This may translate into delays waiting for clarification, attempts that result in wasted materials and time, obviously defective products or products with latent defects which may impair progress during later phases (e.g., integration, testing). These issues may trigger other risk factors such as miss-commitment, decreased motivation and increased frustration.

Third, this may result in students responding negatively to feedback, as they think that it is unnecessarily critical (based on their inaccurate assumptions about their own capabilities and what constitutes an acceptable level of performance). This may also increase frustration, decrease motivation and potentially trigger other issues, such as turnover.

Problem with Work Environment

Student workers may lack an understanding of how to cope with difficulties in the workplace environment. For example, they may not understand how to deal with a poor manager (and the, particularly if a student, manager may lack the skills and understanding required to resolve this conflict). They may also lack the skills required to resolve workplace conflict or to collaborate with others in the work environment. This can potentially trigger miss-commitment, if work is left to pile up while issues are being resolved, or if unscheduled turnover occurs.

UNSCHEDULED TURNOVER OCCURS

Unscheduled turnover can be caused by a student transferring between degree programs or colleges/universities, as a result of miss-commitment, because of a student's departure from the university, or even by a student taking an internship or a medical, family or other personal problem. Each is now discussed.

Student Transfers Program/School

In the context of their educational pursuits, students make decisions in light of what they perceive as their own best interests (which may consider short and/or long term goals). The inflexibility of the semester system may limit students' ability to provide notice (even for a paid position), should they decide to transfer between schools or programs. They may also lose interest at the point that they realize that program participation is no longer supporting their goals (framed now in terms of their new school/department). This may result in low or no-notice turnover.

Turnover due to Miss-commitment

Students may miss-commit (reasons for this are discussed subsequently). If this miss-commitment becomes an acute problem, students may terminate their involvement in paid and/or unpaid extracurricular activities in deference to their immediate academic time needs. This may occur with low or no notice or it may simply result in the student failing to show up (without any sort of explanation).

Departure from University

Students may leave (or be dismissed from) the university for a wide variety of reasons. This may also result in low or no-notice turnover.

Student Takes Internship

Students may decide to pursue an internship to increase their skills and/or post-graduation employment opportunities. Internships may pay more than on-campus employment and generally offer work experience benefits and prospective employer contact that on-campus employment cannot. Students may begin an internship with little or no notice (as employers may offer internships at the last minute to meet their needs and funding capabilities); in many cases, however, internships can be a planned absence and a student may be able to/decide to return to the project after its completion.

Medical/Family/Personal Problem

Like any worker, students may suffer from medical family or other personal problems. These may be intensified by students' lack of coping skills and/or the lack of a need to maintain an income, even in the face of a major medical condition. Notice levels, the potential for students to return to the project upon the resolution of the issue and the duration of the issue will, obviously, vary significantly based on the nature of the issue.

SCHEDULED TURNOVER OCCURS

Scheduled turnover is an expected occurrence at a college or university. It can be caused by student graduation, the end of a paid (e.g., extramurally funded) work period or the end of a course project period. Each is now considered.

Graduation

Students enroll in a university with their departure planned (unlike a typical work environment where employees may not plan to make a career out of a job, but also look at it as something to pursue for an indeterminate period of time). Graduation, fortunately, will be an occurrence that is known well in advance and can be planned for to ensure proper handover. Students, however may fail to notify project leaders (either due to a presumption that they should be notified by some other means or to avoid less-interesting handover activities) and/or have a declining level of interest (particularly after they have secured a job or admission into another program for graduate studies, etc.), that may reduce the ability to conduct and/or the quality of handover activities.

End of Paid Work Period

Research grant (or other funding source) work may have a definite cut-off point after which no additional funding is available to continue a position. This creates a known dateof-departure for a student from a project (or a transition from a paid role to continuation on a volunteer basis). This should be known to the investigator (and thus not suffer from the aforementioned failure-to-notify problem) and be able to be planned for. Students may lose interest and/or change their final day if they find an alternate position, as they approach their known final days.

End of Course Project Period

Course projects, like paid work periods, have definite (and known-to-the investigator) end dates. A desire to receive a good final grade, however, may keep students motivated until the end of the period.

MISS-COMMITMENT

Miss-commitment is to be expected with students who may be unable to gauge the level of work required both from their academic, paid work and extramural pursuits. Miss-commitment, thus, can occur due to students' underestimation of coursework time commitments, an external work commitment commencing or changing, a change in a student's course load and/or involvement in other university activities. These are now considered.

Underestimation of Coursework

Students may overcommit to extramural projects or paid on-campus project work, based on an underestimation of the level of time required for their coursework. This may result in delays, turnover or impaired quality.

External Work Commitment/Change

Students who are working on a project in either a paid or volunteer basis may have jobs outside the project or may seek/take a job based on the benefits it may provide (e.g., work experience, employer contact) or due to their personal financial situation. This may result in low or no-notice changes in project involvement levels, turnover or a decline in product quality.

Change in Course Load

Students may change the number or selection of courses they are taking during the semester and this may change somewhat from semester to semester. This may result in turnover, delays, or quality impairment.

Involvement in Other University Activities

Students may decide to pursue other university extracurricular activities in addition to or instead of the project, or the level of involvement required for (or desired in) these activities may change, reducing the students' level of involvement in the project and/or causing delays, quality problems or turnover.

DIFFERENCES BETWEEN AND CHOOSING BETWEEN USING SQUIRM AND SQUIRM-E

With both the SQUIRM framework and its extension presented, the two can now be compared. This section reviews the differences between SQUIRM and its extension, SQUIRM-E. It discusses the benefits of using one versus the other across multiple scenarios.

DISCUSSION OF THE DIFFERENCES BETWEEN SQUIRM AND SQUIRM-E

The fundamental difference between SQUIRM and SQUIRM-E is the addition, in SQUIRM-E, of the decomposition of standard risk classes in order to also consider risk sources attributable to student and inexperienced workers. This has resulted in two models, each of which is better suited for certain applications (as compared to the other). The remainder of this section considers specific benefits of using one model over the other. It begins by discussing the comparative simplicity presented by SQUIRM, versus SQUIRM-E, and where this simplicity may be valuable. Next, it discusses how SQUIRM-E leans further towards student workers, making SQUIRM more suitable for use or adaptation to non-student, inexperienced workers (or students in contexts where the student status is less relevant). Finally, logistical considerations such as project size and assessor environment familiarity are discussed before a concluding discussion regarding model selection.

COMPARATIVE SIMPLICITY

The SQUIRM framework, by abstracting the root causes of the student-specific risk types into larger categories, is comparatively easier to work with. This is particularly useful in cases where real numbers for these risk types are unknown and cannot be accurately estimated, or where data has been collected without sufficient granularity for use with the more granular model. Alternately, those estimating without data may prefer the more detailed model, as it allows them to consider the risk, likelihood and impact for specific prospective problems, without having to consider whole categories at one. The use of the SQUIRM-E framework, thus, would correspond to a bottom-up risk identification strategy, while the SQUIRM framework (for student-specific risk types) would correspond to a top-down risk identification and assessment approach.

TYPES OF INEXPERIENCED WORKERS

While the SQUIRM model contains elements that may be useful for all areas of inexperience, the elaborations in SQUIRM-E have been targeted specifically at student workers (with a particular focus towards student workers working in the context of a university environment). The further that the actual situation diverges from this, the less valuable the SQUIRM-E elaborations may be. Alternately, one might use these as a starting point, removing (and/or replacing) irrelevant topics and making changes as needed to relevant ones that have an incorrect focus for the scenario under consideration.

PROJECT SIZE

For smaller projects or projects that are less critical, there may be less need for and resources with which to perform risk management. In these cases, the use of the simpler model (and, in fact, even simplifying the SQUIRM framework to remove the third-level error sources) may be prudent.

FAMILIARITY WITH PARTICULARS OF STUDENT WORK ENVIRONMENT

Those with greater familiarity with the risks and nature of the student-involved work environment may find less need for the additional granularity of the SQUIRM-E model. However, as some risk types occur infrequently, heuristic models based on past experiences may oversimplify actual risk levels. Alternately, non-university employers that are less familiar with the particulars of student worker risks may desire to use a modified version of the full SQUIRM-E model. This adaptation is discussed in a subsequent section.

CHOOSING A MODEL

While the two models are not that dissimilar, the selection of a model should be based on the complexity of the project as well as particular needs related to assessing student-statusattributable risk factors. Choosing the incorrect version of the model to use may result in oversimplification, under or overstatement of risks and/or unnecessary work.

EXAMPLES, APPLICATION AND COMPARISON

This section presents three examples which are used to aid reader understanding and application of the models, and to compare the SQUIRM and SQUIRM-E frameworks. These examples include a small spacecraft project, a surface rover project and a near-space recovery system development project. Each is now briefly presented, followed by discussion and the presentation of steps for model use.

ENGINEERING A SPACECRAFT PROTOTYPE

A project to build a prototype for a small spacecraft included numerous risk factors, as it was an initial effort for the participants. The project team included students working on it for class credit and several more that were participating as an extracurricular activity. Risk factors for this project included technical factors and personnel factors. There were numerous technical issues that could have presented a problem as the prototype was student designed and fabricated. A printed circuit board ended up being the factor that created a significant schedule impact (and a minor cost impact, which was absorbed by contingency funds). Due to the board not working during testing (after the students had mounted components on the supplierfabricated-from-student-design board), the prototype was not able to be launched on a high altitude balloon before the end of the spring semester and incurred a significant delay, having to wait until the project team returned in the fall.

BUILDING A SURFACE ROVER

A project to build a rover model, that is in some ways analogous to one that could be used on the moon or on Mars, suffered from significant personnel issues. Students involved in the project, while eager for it to succeed, lacked the knowledge and experience required to bring the project to fruition. When schedule issues occurred (initially with the mechanical design), no strategy was found to rectify them, and the project's schedule continued to slip, impairing numerous successor tasks. While near-heroic efforts were made to attempt to complete the rover during the final days of the schedule, insufficient time was available for testing. The project suffered a final component failure which it was unable to recover from.

TWO ATTEMPTS AT A NEAR SPACE RECOVERY SYSTEM

The Near Space Recovery Technology (NSRT) was proposed as a senior design project for two consecutive years. With senior design projects, students self-select into groups based on selecting a topic that they are interested in working on. The goal of the NSRT project was to create a method to control the descent of a high altitude balloon (HAB) payload. Various approaches for doing this were considered by the NSRT team including timing the balloon burst to maximize the amount of time spent at altitudes with favorable wind patterns and, possibly, to incorporate a mechanism for controlling the rate of descent through various altitudes (again to maximize the time spent at favorable ones). The teams, for both years of the NSRT project, were static, with no students entering late or leaving the team mid-year. During the first year of the NSRT project, the students worked well together and had natural leaders. The first group was competitively more motivated. However, the goals of the project were overly ambitious and were not able to be completed. During the second year, the team was less enthusiastic, not self-driven and lacked commitment to the project. No natural leaders emerged from this group.

The objectives for the second year project were only a subset of those from the first year project; however, in both cases, the team advanced many areas to an approximately 90%-of-completion threshold before the project ended.

DISCUSSION

The examples discussed are all university projects. They, however, span three categories of participant commitment. The first (spacecraft prototype) was largely a volunteer project, though there were a limited number of students working for class credit. The surface rover project was a mixed-mode project, with a significant number of students participating as volunteers, and a significant number participating for class credit. The third (recovery system) was a project for class credit.

All three projects would have been good candidates for the use of the SQUIRM or SQUIRM-E model. The first, due to its scope, duration and complexity would have been able to support the additional overhead of the SQUIRM-E model and benefit from the greater understanding of student/ inexperienced staff-specific on standard risk classes that it provides. The second, due to its shorter duration, smaller size and more informal nature, would have been better suited to the qualitative use of the SQUIRM framework. The third falls between the two prior examples and would have required a more detailed analysis of application-specific tradeoffs in determining what model to use. Particularly in the second year, however, an analysis of the risk factors, which eventuated during the first year (while not conclusive), could have aided the decision making process.

The risk models for these projects are, thus, very different. This is suggested by the different types of risks that eventuated. In the first case, high turnover and what eventually became an expected high loss rate between intake and ongoing participant numbers as well as turnover due to students feeling pressures from other areas of their academic pursuits, came to define the risk model. The second suffered from conflicts between the commitment levels of the two groups of students (falling into both the miss-commitment group, from an expectation of participation levels from students in the class participant category with regards to the volunteers, and from unscheduled turnover - for different groupspecific reasons - within both groups). Finally, the third suffered largely from technical issues, largely attributable to inexperience and some miss-commitment.

APPLICATION

If similar projects were planned in the future, they could use SQUIRM or SQUIRM-E as appropriate (see above), following a five-step approach.

First, the nature of the project must be defined. A discussion of this is beyond the scope of this article; however, several common frameworks exist, including those by Wertz et al . (2011) and Fortescue et al . (2011). A simplified version for small high altitude ballooning projects (which could be adapted to other aerospace projects) has also been proposed (Straub and Fevig, 2012). These frameworks incorporate risk analysis in different ways; however, this process - using SQUIRM/SQUIRM-E - should involve the following four steps.

Second, areas of student (inexperienced staff) involvement, areas impacted by student involvement, and areas not impacted by or involving students, should be identified. The use of the SQUIRM/SQUIRM-E model is appropriate for the first two areas; the last one should use conventional risk assessment and management techniques.

Third, a granularity level of risk assessment must be determined, based on the scale and nature of the project. Risk could be assessed at the whole-project-level or at any logical division level thereunder. The granularity level need not be consistent; thus, areas of higher risk or risk impact could be assessed at higher levels of granularity than less risky or impactful areas.

Forth, for each unit of assessment, risk factors should be identified. This will involve application/task-specific brainstorming as well as reviewing the student/inexperienced worker-attributable factors presented by the SQUIRM model. For each factor, a likelihood and impact level should be estimated (based on historic data, experience or other technique).

Finally, any summative assessment should be performed. This may include combining risk data from sub-tasks into task-level assessments (or from tasks into project-level assessment), evaluating student/inexperienced worker participation value and comparing project-level assessments.

The foregoing can be performed qualitatively or quantitatively. Quantitative analysis is discussed in greater detail in the subsequent section.

QUANTIFYING THE MODEL

While the discussion up to this point has been qualitative, both the SQUIRM and SQUIRM-E models lend themselves to being used with quantitative data, if it is available. Figure 3 demonstrates how the identified risk areas, along with mitigation/response strategies identified using the SQUIRM/ SQUIRM-E model, can be used to assess the weighted (by likelihood of occurrence) risk impact levels for particular risk sources and for the project overall. The overall project risk levels may serve to facilitate comparison between projects (in conjunction with other metrics such as project importance and cost, etc.).

Figure 3
Quantifying SQUIRM & SQUIRM-E.

RISK ASSESSMENT

Risks are assessed both in terms of their likelihood of occurrence and the magnitude of impact that they may have if they eventuate. Risks may be assessed based on probabilities, if sufficient historical data exists or a probabilistic model is known or can be inferred, or they can be categorized (with approximate average probabilities, used to facilitate quantitative comparisons).

The impact can, similarly, be quantified in terms of time, resource and cost (which may be combined into a single cost metric), if data is available. Alternately, they can be categorized and an average value used.

MITIGATION/RESPONSE ASSESSMENT

The risk effect may be altered by the existence (or development) of mitigation and response strategies. Mitigation strategies may reduce likelihood, impact, or both, while response strategies focus solely on reducing impact. The change created by the existence of one or more of these strategies should be considered. Again, actual numbers or classifications and average values can be used for this assessment.

COMBINING FOR RESULT

The risk effect and mitigation/response change are combined for each risk factor. Then (if multiple risk factors are present), the final weighted effects are combined, to produce an aggregated risk impact value for the project. It is important, when using this approach, that all values use a comparable scale (e.g., combining average and historical cost values should be done carefully to avoid over or understatement of risks). If risk values are being used to compare projects, then the need for a common scale extends to all items being compared. Thus, it is ideal (but often not practical) to use historical data and (inflation and other factor-adjusted) real costs, as this facilitates direct comparison.

DATA FOR MODEL PARAMETERS

One particular challenge in the use of SQUIRM or SQUIRM-E quantitatively is the collection of the parameters which are required in order to perform the quantitative analysis. Problematically, this data likely varies on an application-specific basis (or general data would need to be validated for application-specific use). While, for small satellites, some relevant data has been collected by Brumbaugh and Lightsey (2013), and they are collecting data (Brumbaugh and Lightsey, 2014) to facilitate a more robust analysis, this doesn't cover all areas required by this model, nor does it help those attempting to assess risk in other application areas. For areas and applications where this data is not available, it will need to be estimated based on past experience and other available information. The collection of data specific to particular applications is an area for future work.

VALUE MODEL FOR INEXPERIENCED WORKERS

The foregoing may lead one to question the value of using inexperienced workers (particularly students) on any project of particular importance. Would the students/junior employees not be better served (and better serve others) by gaining experience through non-impactful learning exercises instead of work on real projects (which could be negatively impacted)? This section considers the value of student (and other inexperienced) workers. Figure 4 presents a diagram of the considerations.

Figure 4
Value Proposition for the Use of Inexperienced and Student Workers.

COST OF INEXPERIENCED/STUDENT WORKERS

The cost of inexperienced and student workers is aptly identified by the SQUIRM and (to a greater extent) SQUIRM-E models. Clearly, each prospective risk may impair a project (if it eventuates) incurring time, productivity impairment (including productivity impairment of other more senior workers that may need to help rectify student/inexperienced worker mistakes), material and goodwill costs. Somewhat (in many cases) offsetting, this is the lower wage levels paid to student/inexperienced staff. Thus, for tasks that these individuals can learn to perform effectively and with minimal (or comparable to more experienced staff) oversight, a cost savings may be enjoyed. The assignment of junior staff to these types of tasks, however, may impair their learning process and prevent them from gaining (or decrease the speed of them gaining) skills that could make them more valuable to their current and future prospective employers.

TRAINING BENEFITS

The proverbial adage of "killing two birds with one stone" can be used in an attempt to justify the use of student/unskilled workers on real projects. If students/unskilled workers can be productively contributing to a project while also gaining experience, it would seem that two types of benefit are being gained for a single cost. While this may certainly be true in some (perhaps many) cases, the oversimplification of the cost model (i.e., the consideration of a "single" cost) may be inaccurate. Costs may be higher to facilitate the student/ inexperienced worker participation, which should be taken into account in the comparison.

DISCONTINUOUS INNOVATION BENEFITS

One area where student/inexperienced workers may offer particular benefit is in identifying sources of discontinuous innovation. These workers, who may not fully understand where the proverbial "box" is, may be well-suited to think outside of it. Swartwout (2004; 2011) identifies this, for example, as a key benefit of "university class" small spacecraft programs: the higher level of risk tolerance and the presence of the junior staff make these types of missions well suited to trying innovative ideas and identifying areas for innovation in operations.

DISCUSSION OF THE DIFFERENCES BETWEEN STUDENT VOLUNTEERS, PAID STUDENT WORKERS, INTERNS AND JUNIOR EMPLOYEES

It has been stated, previously, that the SQUIRM and SQUIRM-E frameworks can be used to address risks across several different types of junior employees; however, the risk factor impact posed by these different groups are dissimilar. This section begins the process of considering the differences between the multiple types of workers that the SQUIRM/SQUIRM-E models could be applied to (in some cases with limited modifications). The particulars of each worker type are now discussed, this includes: student volunteers, paid student workers, interns and junior employees.

STUDENT VOLUNTEERS

Student volunteers will (correctly) view their participation as at-will. If they are interested, see benefits being provided and have time, they will continue working on the project. If they lose interest, feel that they are not receiving (or have already received all applicable) benefits or are confronted with other draws on their time, they will stop. Retention of students from semester-to-semester may be difficult, as they may perceive participation as an opt-in activity (like joining a club or taking a class), where a participation decision is made anew each semester. They may fail to realize or understand the impact of their change in participation status on others that have also donated their time to provide benefit to them or the cost of the time committed to their training by paid staff, etc.

PAID STUDENT WORKERS

Paid student workers may be more committed, as they are receiving another source of benefit (pay) over and above what is received by volunteers. However, in the context of the comparatively large amounts of money that they are paying (or which is being paid on their behalf) to attend school, they may see little difference between the paid and unpaid positions in terms of any sense of commitment or longer-term responsibility. Pay, thus, may overcome (or assist in rectifying) lack of interest issues, but may not assist with semester-to-semester turnover issues or commitment in the face of other time draws.

INTERNS

Interns (in the case of non-university employers) may see a multi-faceted benefit which may cause particular (comparative) commitment. The intern may be earning credit for their participation, getting paid, gaining experience and gaining an opportunity to demonstrate their capabilities to a prospective employer. The foregoing (particularly if the intern sees the employer as a desirable place to seek post-graduation employment) may cause interns to place the internship amongst their highest priorities, overcoming most of the common (controllable) risk factors and creating a particularly high level of diligence. Interns may or may not have ongoing coursework during the internship period (the lack thereof reducing another set of risk factors). As a generally fixed-term period of employment, however, scheduled turnover is expected.

JUNIOR EMPLOYEES

Junior employees may see performance as critical to their future livelihood; however, this perception may not always be the case (even if it is accurate, it may not be perceived or employment may be perceived as an entitlement). While most will want to set their careers off on a 'good foot', others may find the change in structure (more or less control, different control structures and a need to be self-starting) problematic and not know how to function effectively under the changed structure. Employees may also be looking for new positions, if they take a position that is not of their liking simply to 'pay the bills' and may lack the professional discipline to continue to perform while in a job they dislike (or which they are not particularly excited about).

CONCLUSIONS AND FUTURE WORK

This paper has expanded the SQUIRM framework into a new SQUIRM-E version that adds additional assessment criteria related to student-specific risk types. It has presented an analytical framework for assessing risk factors, relevant to student and inexperienced workers quantitatively, and evaluating the value of the use of a student/inexperienced worker on a given project. A limited extrapolation to non-student workers has been discussed.

Future work will involve the enhancement of the quantitative models presented as well as the collection of a data set to begin to characterize these common risk areas for various classes of projects. It will also involve the development of a SQUIRM-E-based model for junior employees that replaces student-specific factors with those more appropriate to junior employees.

ACKNOWLEDGEMENTS

Small satellite development work at the University of North Dakota is, or has been, supported by the North Dakota Space Grant Consortium, North Dakota NASA EPSCoR, the University of North Dakota Faculty Research Seed Money Committee, North Dakota EPSCoR (NSF Grant # EPS-814442), the Department of Computer Science, the John D. Odegard School of Aerospace Sciences and the National Aeronautics and Space Administration.

The involvement of the numerous students from multiple disciplines in UND small satellite development is gratefully acknowledged. Also, thanks are given to the numerous faculty mentors who have helped make this possible.

Thanks is given to John Nordlie, who provided a subset of the information used for the examples. Thanks is also due to all of the faculty, students and staff who participated in the examples presented herein.

REFERENCES

• Ayob, A., Majid, R.A., Hussain, A. and Mustaffa, M.M., 2012, "Creativity enhancement through experiential learning" Advances in Natural and Applied Science, Vol. 6, No 2, pp. 94-99.
• Bauerle, T.L. and Park, T.D., 2012, "Experiential learning enhances student knowledge retention in the plant sciences", HortTechnology, Vol. 22, No 5, pp. 715-718.
• Boholm, A., 1998, "Comparative studies of risk perception: A review of twenty years of research" Journal of Risk Research, Vol. 1, No 2, pp. 135-163. doi: 10.1080/136698798377231.
  » https://doi.org/10.1080/136698798377231
• Botterill, L. and Mazur, N., 2004, "Risk and risk perception: A literature review", Project no.BRR-8A, Rural Industries Research and Development Corporation, Barton.
• Brodeur, D.R., Young, P.W. and Blair, K.B., 2002, "Problem-based learning in aerospace engineering education", Proceedings of the 2002 American Society for Engineering Education Annual Conference and Exposition, Montreal, Canada, pp. 16-19.
• Broman, D., Sandahl, K. and Baker, M., 2012, "The company approach to software engineering project courses", Education, IEEE Transactions On, Vol. 55, No 4, pp. 445-452. doi:10.1109/TE.2012.2187208.
  » https://doi.org/10.1109/TE.2012.2187208
• Brumbaugh, K.M. and Lightsey, E.G., 2013, "Application of risk management to university CubeSat missions", Journal of Small Satellites, Vol. 2, No 1, pp. 147-160.
• Brumbaugh, K. and Lightsey, E.G., 2014, "CubeSat mission risk survey - a call for response from the small satellite community", Journal of Small Satellites, Vol. 2, No 1, pp. 83-84.
• Bütün, E., 2005, "Teaching genetic algorithms in electrical engineering education: A problem-based learning approach", International Journal of Electrical Engineering Education, Vol. 42, No 3, pp. 223-233. doi: 10.7227/IJEEE.42.3.2.
  » https://doi.org/10.7227/IJEEE.42.3.2
• Carroll, J.S., Rudolph, J.W. and Hatakenaka, S., 2002, "Lessons learned from non-medical industries: Root cause analysis as culture change at a chemical plant", Quality & Safety in Health Care, Vol. 11, No 3, pp. 266-269. doi: 10.1136/qhc.11.3.266.
  » https://doi.org/10.1136/qhc.11.3.266
• Cohn, L.D., Macfarlane, S., Yanez, C. and Imai, W.K., 1995, Risk-perception: Differences between adolescents and adults, Health Psychology, Vol. 14, No 3, pp. 217-222.
• Coller, B.D. and Scott, M.J., 2009, "Effectiveness of using a video game to teach a course in mechanical engineering", Computers & Education, Vol. 53, No 3, pp. 900-912. doi: 10.1016/j.compedu.2009.05.012.
  » https://doi.org/10.1016/j.compedu.2009.05.012
• Correll, N., Wing, R. and Coleman, D., 2013, "A one-year introductory robotics curriculum for computer science upperclassmen" Education, IEEE Transactions On, Vol. 56, No 1, pp. 54-60. doi:10.1109/TE.2012.2220774.
  » https://doi.org/10.1109/TE.2012.2220774
• Dahlgren, M. A. and Dahlgren, L.O., 2002, "Portraits of PBL: Students' experiences of the characteristics of problem-based learning in physiotherapy, computer engineering and psychology"Instructional Science, Vol. 30, No 2, pp. 111-127. doi: 10.1023/A:1014819418051.
  » https://doi.org/10.1023/A:1014819418051
• Deery, H.A., 1999, "Hazard and risk perception among young novice drivers", Journal of Safety Research, Vol. 30, No 4, pp. 225-236. doi:10.1016/S0022-4375(99)00018-3.
  » https://doi.org/10.1016/S0022-4375(99)00018-3
• DeJoy, D.M., 1992, "An examination of gender differences in traffic accident risk perception" Accident Analysis & Prevention, Vol. 24, No 3, pp. 237-246. doi:10.1016/0001-4575(92)90003-2.
  » https://doi.org/10.1016/0001-4575(92)90003-2
• Doggett, A.M., 2004, "A statistical comparison of three root cause analysis tools", Journal of Industrial Technology, Vol. 20, No 2, pp. 2-9.
• Doppelt, Y., 2003, "Implementation and assessment of project-based learning in a flexible environment" International Journal of Technology and Design Education, Vol. 13, No 3, pp. 255-272. doi: 10.1023/A:1026125427344.
  » https://doi.org/10.1023/A:1026125427344
• Duch, B.J., 1996, "Problem-based learning in physics: The power of students teaching students", Journal of College Science Teaching, Vol. 15, No 5, pp. 326-329.
• Elbaum, B., 1989, "Why apprenticeship persisted in Britain but not in the United States" Journal of Economic History, Vol. 49, No 2, pp. 337-349. doi:10.1017/S002205070000797X.
  » https://doi.org/10.1017/S002205070000797X
• Fasse, B., Hotaling, N., Forest, C., Bost, L. and Hermann, C., 2012, "The case for multi-disciplinary capstone design: A quantitative analysis of impact on job placement and product quality", BMES Conference, Retrieved from http://pbl.gatech.edu/wp-content/uploads/2013/05/fasse_bmes_2012.pdf
  » http://pbl.gatech.edu/wp-content/uploads/2013/05/fasse_bmes_2012.pdf
• Fortescue, P., Swinerd, G. and Stark, J., 2011, Spacecraft systems engineering, 4th ed., Wiley.
• Gilmore, M., 2013, "Improvement of STEM education: Experiential learning is the key", Modern Chemistry & Applications, Vol. 1, No 3, pp. e109. doi:10.4172/2329-6798.1000e109.
  » https://doi.org/10.4172/2329-6798.1000e109
• Hall, S.R., Waitz, I., Brodeur, D.R., Soderholm, D.H. and Nasr, R., 2002, "Adoption of active learning in a lecture-based engineering class", Frontiers in Education, 2002. FIE 2002. 32nd Annual, Vol. 1, pp. T2A-9-T2A-15. doi:10.1109/FIE.2002.1157921.
  » https://doi.org/10.1109/FIE.2002.1157921
• Hayne, S., Pendergast, M. and Smith, C.A.P., 2012 "Team composition, knowledge and collaboration", SAIS 2012 Proceedings, Paper 17.
• Hotaling, N., Fasse, B.B., Bost, L.F., Hermann, C.D. and Forest, C.R., 2012, "A quantitative analysis of the effects of a multidisciplinary engineering capstone design course", Journal of Engineering Education, Vol. 101, No 4, pp. 630-656. doi:10.1002/j.2168-9830.2012.tb01122.xIedema.
  » https://doi.org/10.1002/j.2168-9830.2012.tb01122.xIedema
• R.A., Jorm, C., Long, D., Braithwaite, J., Travaglia, J. and Westbrook, M., 2006, "Turning the medical gaze in upon itself: Root cause analysis and the investigation of clinical error" Social Science & Medicine, Vol. 62,No 7, pp. 1605-1615. doi:10.1016/j.socscimed.2005.08.049.
  » https://doi.org/10.1016/j.socscimed.2005.08.049
• Jackson, D. and Hancock, P., 2010, "Non-technical skills in undergraduate degrees in business: Development and transfer", Education Research and Perspectives, Vol. 37, No 1, pp. 52-84.
• Jayaram, S., Boyer, L., George, J., Ravindra, K. and Mitchell, K., 2010, "Project-based introduction to aerospace engineering course: A model rocket". Acta Astronautica, Vol. 66, No 9-10, pp. 1525-1533. doi:10.1016/j.actaastro.2009.10.038.
  » https://doi.org/10.1016/j.actaastro.2009.10.038
• Larsen, J.A. and Nielsen, J.D., 2011, "Development of cubesats in an educational context", Recent Advances in Space Technologies (RAST), 2011 5th International Conference On, pp. 777-782. doi:10.1109/RAST.2011.5966948.
  » https://doi.org/10.1109/RAST.2011.5966948
• Larsen, J.A., Nielsen, J.F.D. and Zhou, C., 2013, Motivating students to develop satellites in problem and project-based learning (PBL) environment, International Journal of Engineering Pedagogy (iJEP), Vol. 3, No 3, pp. 11-17.
• Levinson, R.A., Jaccard, J. and Beamer, L., 1995, "Older adolescents' engagement in casual sex: Impact of risk perception and psychosocial motivations", Journal of Youth and Adolescence, Vol. 24, No 3, pp. 349-364. doi:10.1007/BF01537601.
  » https://doi.org/10.1007/BF01537601
• Mathers, N., Goktogen, A., Rankin, J. and Anderson, M., 2012, "Robotic mission to mars: Hands-on, minds-on, web-based learning",Acta Astronautica, Vol. 80, pp. 124-131. doi:10.1016/j.actaastro.2012.06.003.
  » https://doi.org/10.1016/j.actaastro.2012.06.003
• Mitchell, V.W., 1995, "Organizational risk perception and reduction: A literature review", British Journal of Management, Vol. 6, No 2, pp. 115-133. doi:10.1111/j.1467-8551.1995.tb00089.x.
  » https://doi.org/10.1111/j.1467-8551.1995.tb00089.x
• Mountrakis, G. and Triantakonstantis, D., 2012, "Inquiry-based learning in remote sensing: A space balloon educational experiment" Journal of Geography in Higher Education, Vol. 00, No 0, pp.1-17. doi: 10.1080/03098265.2011.638707.
  » https://doi.org/10.1080/03098265.2011.638707
• Nagda, B.A., Gregerman, S.R., Jonides, J., von Hippel, W. and Lerner, J.S., 1998, "Undergraduate student-faculty research partnerships affect student retention" The Review of Higher Education, Vol. 22, No 1, pp. 55-72. doi:10.1353/rhe.1998.0016.
  » https://doi.org/10.1353/rhe.1998.0016
• Neily, J., Ogrinc, G., Mills, P., Williams, R., Stalhandske, E., Bagian, J. and Weeks, W.B., 2003, "Using aggregate root cause analysis to improve patient safety", Joint Commission Journal on Quality and Safety, Vol. 29, No 8, pp.434-439.
• Nordlie, J. and Fevig, R., 2011, "Blending research and teaching through high-altitude balloon projects", Proceedings of the 2nd Annual Academic High Altitude Conference.
• Okudan, G.E. and Rzasa, S.E., 2006, "A project-based approach to entrepreneurial leadership education", Technovation, Vol. 26, No 2, pp. 195-210. doi:10.1016/j.technovation.2004.10.012.
  » https://doi.org/10.1016/j.technovation.2004.10.012
• Pollard, C.E., 2012, "Lessons learned from client projects in an undergraduate project management course", Journal of Information Systems Education, Vol. 23, No 3, pp. 271-282.
• Prince, M.J., Felder, R.M. and Brent, R., 2007, "Does faculty research improve undergraduate teaching? An analysis of existing and potential synergies", Journal of Engineering Education, Vol. 96, No 4, pp. 283-294. doi:10.1002/j.2168-9830.2007.tb00939.x.
  » https://doi.org/10.1002/j.2168-9830.2007.tb00939.x
• Qidwai, U., 2011, "Fun to learn: Project-based learning in robotics for computer engineers", ACM Inroads, Vol. 2, No 1, pp. 42-45. doi:10.1145/1929887.1929904.
  » https://doi.org/10.1145/1929887.1929904
• Renn, O., 1998, "The role of risk perception for risk management", Reliability Engineering & System Safety, Vol. 59, No 1, pp. 49-62. doi:10.1016/S0951-8320(97)00119-1.
  » https://doi.org/10.1016/S0951-8320(97)00119-1
• Rippl, S., 2002, "Cultural theory and risk perception: A proposal for a better measurement", Journal of Risk Research, Vol. 5, No 2, pp. 147-165. doi:10.1080/13669870110042598.
  » https://doi.org/10.1080/13669870110042598
• Ribeiro, L.R.C, 2008, "Electrical engineering students evaluate problem-based learning (PBL)" International Journal of Electrical Engineering Education, Vol. 45, No 2, pp. 152-161. doi:10.7227/IJEEE.45.2.7.
  » https://doi.org/10.7227/IJEEE.45.2.7
• Robson, N., Dalmis, I.S. and Trenev, V., 2012, "Discovery learning in mechanical engineering design: Case-based learning or learning by exploring?", 2012 ASEE Annual Conference, American Society for Engineering Education.
• Roh, K.H., 2003, "Problem-based learning in mathematics", ERIC Clearing House for Science Mathematics and Environmental Education.
• Ronan, K.R. and Johnston, D.M., 2001, "Correlates of hazard education programs for youth", Risk Analysis, Vol. 21, No 6, pp. 1055-1064. doi:10.1111/0272-4332.216174.
  » https://doi.org/10.1111/0272-4332.216174
• Rooney, J.J. and Heuvel, L.N.V., 2004, "Root cause analysis for beginners", Quality Progress, Vol. 37, No 7, pp. 45-56.
• Saunders-Smits, G.N., Roling, P., Brügemann, V., Timmer, N. and Melkert, J., 2012, "Using the engineering design cycle to develop integrated project based learning in aerospace engineering", International Conference on Innovation, Practice and Research in Engineering Education.
• Simons, L., Fehr, L., Blank, N., Connell, H., Georganas, D., Fernandez, D. and Peterson, V., 2012 "Lessons learned from experiential learning: What do students learn from a practicum/internship?", International Journal of Teaching and Learning in Higher Education, Vol. 24, No 3, pp. 325-334.
• Sjöberg, L., 2000, "Factors in risk perception", Risk Analysis, Vol. 20, No 1, pp. 1-12.
• Slovic, P., Fischhoff, B. and Lichtenstein, S., 1982, "Why study risk perception?", Risk Analysis, Vol. 2, No 2, pp. 83-93. doi:10.1111/j.1539-6924.1982.tb01369.x.
  » https://doi.org/10.1111/j.1539-6924.1982.tb01369.x
• Snell, K.D., 1996, "The apprenticeship system in British history: The fragmentation of a cultural institution", History of Education, Vol. 25, No 4, pp. 303-321. doi:10.1080/0046760960250401.
  » https://doi.org/10.1080/0046760960250401
• Steinberg, L., 2004, "Risk taking in adolescence: What changes, and why?", Annals of the New York Academy of Sciences, Vol. 1021, No 1, pp. 51-58. doi:10.1196/annals.1308.005.
  » https://doi.org/10.1196/annals.1308.005
• Straub, J., Berk, J., Nervold, A. and Whalen, D., 2013a, "OpenOrbiter: An interdisciplinary, student run space program", Advances in Education, Vol. 2, No 1, pp. 4-10.
• Straub, J., Fevig, R., Casler, J. and Yadav, O., 2013b, "Risk analysis & management in student-centered spacecraft development projects", Proceedings of the 2013 Reliability and Maintainability Symposium, Orlando, FL, USA.
• Straub, J., Whalen, D. and Marsh, R., (in press), "Assessing the value of the OpenOrbiter program's research experience for undergraduates, Sage OPEN.
• Straub, J., 2013, "OpenOrbiter: Analysis of a student-run space program", Proceedings of the 64th International Astronautical Congress, pp. 2-10.
• Straub, J. and Fevig, R., 2012, "Formalizing mission analysis and design techniques for high altitude ballooning", Proceedings of the 3rd Annual High Altitude Conference, Nashville, TN.
• Straub, J. and Whalen, D., 2013, "An assessment of educational benefits from the OpenOrbiter space program", Education Sciences, Vol. 3, No 3, pp. 259-278. doi:10.3390/educsci3030259.
  » https://doi.org/10.3390/educsci3030259
• Sulaiman, S.A., Git, H.M. and Rani, A.M.A., 2010, "Engineering team project course for development of teamwork skills".
• Swartwout, M., 2004, "University-class satellites: From marginal utility to 'disruptive' research platforms", Proceedings of the 18th Annual AIAA/USU Conference on Small Satellites, Logan, UT, USA.
• Swartwout, M., 2011, "AC 2011-1151: Significance of student-built spacecraft design programs it's impact on spacecraft engineering education over the last ten years", Proceedings of the American Society for Engineering Education Annual Conference, Retrieved from http://www.asee.org/file_server/papers/attachment/file/0001/1307/paper-final.pdf
  » http://www.asee.org/file_server/papers/attachment/file/0001/1307/paper-final.pdf
• Weber, E.U. and Milliman, R.A., 1997, "Perceived risk attitudes: Relating risk perception to risky choice", Management Science, Vol. 43, No 2, pp. 123-144.
• Weidl, G., Madsen, A. and Israelson, S., 2005, "Applications of object-oriented Bayesian networks for condition monitoring, root cause analysis and decision support on operation of complex continuous processes", Computers & Chemical Engineering, Vol. 29, No 9, pp. 1996-2009. doi:10.1016/j.compchemeng.2005.05.005.
  » https://doi.org/10.1016/j.compchemeng.2005.05.005
• Weinstein, N.D., 1984, "Why it won't happen to me: Perceptions of risk factors and susceptibility", Health Psychology, Vol. 3, No 5, 431-457. doi:10.1037/0278-6133.3.5.431.
  » https://doi.org/10.1037/0278-6133.3.5.431
• Wertz, J.R., Everett, D.F. and Puschell, J.J., 2011, "Space mission engineering: The new SMAD", Hawthorne, CA: Microcosm Press.
• Wildavsky, A. and Dake, K., 1990, "Theories of risk perception: Who fears what and why?" Daedalus, Vol. 119, No 4, pp. 41-60.
• Zydney, A.L., Bennett, J.S., Shahid, A. and Bauer, K.W., 2002a, "Impact of undergraduate research experience in engineering", Journal of Engineering Education, Vol. 91, No 2, pp. 151-157. doi:10.1002/j.2168-9830.2002.tb00687.x.
  » https://doi.org/10.1002/j.2168-9830.2002.tb00687.x
• Zydney, A.L., Bennett, J.S., Shahid, A. and Bauer, K., 2002b, "Faculty perspectives regarding the undergraduate research experience in science and engineering", Journal of Engineering Education, Vol. 91, No 3, pp. 291-297. doi:10.1002/j.2168-9830.2002.tb00706.x.
  » https://doi.org/10.1002/j.2168-9830.2002.tb00706.x

Publication Dates

• Publication in this collection
  Jul-Sep 2014

History

• Received
  28 Dec 2013
• Accepted
  26 June 2014