An update to a functional modeling scoring rubric with overall and question-level inter-rater reliability

AR Murphy, HE Ingram… - Journal of …, 2019 - asmedigitalcollection.asme.org
AR Murphy, HE Ingram, JT Nelson, MR Bohm, JS Linsey, RL Nagel
Journal of Mechanical Design, 2019asmedigitalcollection.asme.org
Functional modeling allows an engineer or a designer to separate component architecture
from the purpose or the objective of a system. Many engineering design textbooks introduce
functional modeling as a tool to aid with conceptual design, to assist with idea generation, to
allow for conceptual decomposition of complex systems, to convey abstractions of systems
at both high and low levels, and to reduce design fixation. This article presents an update to
a previously published functional modeling rubric that increases the overall inter-rater …
Abstract
Functional modeling allows an engineer or a designer to separate component architecture from the purpose or the objective of a system. Many engineering design textbooks introduce functional modeling as a tool to aid with conceptual design, to assist with idea generation, to allow for conceptual decomposition of complex systems, to convey abstractions of systems at both high and low levels, and to reduce design fixation. This article presents an update to a previously published functional modeling rubric that increases the overall inter-rater reliability and the question-by-question inter-rater reliability through the inclusion of student examples, expert examples, two additional scoring questions, and detailed guidelines. These changes are validated by novice raters rescoring a prior set of data with the newly updated rubric to compare the results for improved inter-rater reliability. These rubric changes also make the assessment tool more accessible to individuals new to functional modeling. The analysis shows that the updated rubric improved the overall Cohen’s kappa from 0.65 to 0.76 and improved the lowest question-level Cohen’s kappa from 0.03 to 0.47, which indicates a substantial increase in reliability at the question level that subsequently improved the overall reliability. These results are discussed in detail followed by an abbreviated presentation of the updated rubric in the Appendix.
The American Society of Mechanical Engineers
以上显示的是最相近的搜索结果。 查看全部搜索结果