User Perception of Numeric Contribution Semantics for Goal Models: an Exploratory Experiment (ER’2017)
Norah Alothman, Mehrnaz Zhian and Sotirios Liaskos
15 July, 2017
Abstract
This page accompanies our paper submission to the International Conference on Conceptual Modeling (ER 2017). A version including the R scripts can be found here. The data, redacted to include information only relevant to this study and remove personally identifying information can be found here and here, availed here in compliance to clause “Anonymized data will be included as an appendix in the PIs major research report and potential publications of this research.” of informed consent. The models used can be found here.
Participant Demographics
## Gender
## FieldOfStudy Female Male Sum
## Business and Econ. 8 5 13
## Education 3 3 6
## Fine Arts (e.g. Music, Theater, Film) 2 3 5
## Health Sciences 1 1 2
## Humanities 8 3 11
## Science and Tech. 3 7 10
## Social Sciences 6 3 9
## Sum 31 25 56Model Characteristics
Model attributes
## Size Depth Goals Contributions
##
## A Small 1 3 2
## B Small 1 4 3
## C Small 1 5 4
## D Medium 2 5 4
## E Medium 2 7 6
## F Medium 2 6 5
## G Large 3 7 6
## H Large 3 9 8
## I Large 3 10 9Number rankings versus theories
Each row in the table below is each of the model. Each column is each theory. The cell represents the ranking of the number resulting from applying the corresponding theory. For example if for a model SerPar gives 0.2, Bayes 0.4, MinMax 0.5 and Linear 0.3, (an unlikely contigence), the rankings will be respectivelly 1,3,4,2.
## Bayes MinMax SerPar Linear
## [1,] 2 3 1 4
## [2,] 2 3 1 4
## [3,] 2 3 1 4
## [4,] 2 3 1 4
## [5,] 2 3 1 4
## [6,] 2 3 1 4
## [7,] 2 3 1 4
## [8,] 2 3 1 4
## [9,] 2 3 1 4
## [10,] 2 4 1 3
## [11,] 2 3 1 4
## [12,] 2 3 1 4
## [13,] 2 4 1 3
## [14,] 2 3 1 4
## [15,] 2 3 1 4
## [16,] 2 3 1 4
## [17,] 2 3 1 4
## [18,] 2 3 1 4
## [19,] 2 3 1 4
## [20,] 2 3 1 4
## [21,] 2 4 1 3
## [22,] 2 3 1 4
## [23,] 2 3 1 4
## [24,] 2 3 1 4
## [25,] 2 3 1 4
## [26,] 2 4 1 3
## [27,] 2 3 1 4
## [28,] 2 3 1 4
## [29,] 2 3 1 4
## [30,] 2 3 1 4
## [31,] 2 3 1 4
## [32,] 2 3 1 4
## [33,] 2 4 1 3
## [34,] 2 3 1 4
## [35,] 2 3 1 4
## [36,] 2 3 1 4So we observe that Linear is always the largest number the vast majority of the times, MinMax is the second largest also for the vast majority of times, Bayes is always the second smallest and Serial-Parallel is aways the smallest.
Analysis
Do participants chose randomly?
Total Deviations from Uniformity by Style and Size.
Deviations from Uniformity by Theory and Style and Size.
Are participants consistent?
Higher than random concentrations
## Bayes Linear MinMax SerPar
## 5 13 27 2Lower than random concentrations
## Bayes Linear MinMax SerPar
## 9 12 4 35No concentrations
## [1] 7Stated Method and Aggrement
At the end of the first version of the instrument the participants are asked what method they follow through an example mode and calculations on the example model representing each of the theories, with the additional option to provide their own method. At the end of the second version of the instrument the participants are instead asked whether “[they] used a specific calculation method”, which they had to describe in their own words or if instead “[they] did not use a specific calculation method, [but] just used [their] intuition”.
Of the total of 30 responses coming from the first version and the 26 responses coming from the second one we identify 35 participants who forcibly (first instrument) or voluntarily (second instrument) identify a specific way of working through the aggregations. Most of those responses were the result of selection from the corresponding multiple choice question (1st version) while a smaller number is described verbally (1st and second version). The latter (8 cases total) are classified under one of the theories or special “Other” and “Don’t Know” categories through qualitative analysis of the response. The table below represents the distribution of the stated methods.
##
## Bayes Don't know Linear MinMax Other SerPar
## 4 2 8 2 8 11There is a surprising perception that the Serial Parallel model is used, when it is not. As a matter of fact a mere 20% of those 35 participants states that they use a method that also happens to be the one they actually used with statistically significant consistency. A 25.71% actually states that they follow a theory they actually use less with statistically significant consistency.
The finding becomes less surprising by focusing on the second version of the instrument in which participants are not forced to choose a calculation method but are asked if they use their intuition rather than performing calculations. Out of the 26 responses 21 actually say they just use their intuition (no specific method).
Mosaics
Effect of Number Style
In aggregate, we can observe below that free style (weights adding up to more than 1.5) there is a tenancy for participants to select Bayesian or MinMax model and not Linear and SerPar. For restricted style models on the other hand, participants tend to choose Linear much more than other theories, whose choice frequency is much less than expected.
##
## Cell Contents
## |-------------------------|
## | Count |
## | Row Percent |
## | Total Percent |
## |-------------------------|
##
## Total Observations in Table: 2016
##
## | Theory
## Style | Bayes | Linear | MinMax | SerPar | Row Total |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Free | 301 | 175 | 430 | 102 | 1008 |
## | 29.861% | 17.361% | 42.659% | 10.119% | 50.000% |
## | 14.931% | 8.681% | 21.329% | 5.060% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Rest | 208 | 358 | 295 | 147 | 1008 |
## | 20.635% | 35.516% | 29.266% | 14.583% | 50.000% |
## | 10.317% | 17.758% | 14.633% | 7.292% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Column Total | 509 | 533 | 725 | 249 | 2016 |
## -------------|-----------|-----------|-----------|-----------|-----------|
##
## Effect of Model Depth
The model depth does not seem to affect the choice.
##
## Cell Contents
## |-------------------------|
## | Count |
## | Row Percent |
## | Total Percent |
## |-------------------------|
##
## Total Observations in Table: 2016
##
## | Theory
## Size | Bayes | Linear | MinMax | SerPar | Row Total |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Small | 174 | 155 | 200 | 143 | 672 |
## | 25.893% | 23.065% | 29.762% | 21.280% | 33.333% |
## | 8.631% | 7.688% | 9.921% | 7.093% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Medium | 190 | 187 | 238 | 57 | 672 |
## | 28.274% | 27.827% | 35.417% | 8.482% | 33.333% |
## | 9.425% | 9.276% | 11.806% | 2.827% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Large | 145 | 191 | 287 | 49 | 672 |
## | 21.577% | 28.423% | 42.708% | 7.292% | 33.333% |
## | 7.192% | 9.474% | 14.236% | 2.431% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Column Total | 509 | 533 | 725 | 249 | 2016 |
## -------------|-----------|-----------|-----------|-----------|-----------|
##
## Free Style Weighting
##
## Cell Contents
## |-------------------------|
## | Count |
## | Row Percent |
## | Total Percent |
## |-------------------------|
##
## Total Observations in Table: 1008
##
## | Model
## Size | Bayes | Linear | MinMax | SerPar | Row Total |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Small | 114 | 61 | 112 | 49 | 336 |
## | 33.929% | 18.155% | 33.333% | 14.583% | 33.333% |
## | 11.310% | 6.052% | 11.111% | 4.861% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Medium | 108 | 54 | 142 | 32 | 336 |
## | 32.143% | 16.071% | 42.262% | 9.524% | 33.333% |
## | 10.714% | 5.357% | 14.087% | 3.175% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Large | 79 | 60 | 176 | 21 | 336 |
## | 23.512% | 17.857% | 52.381% | 6.250% | 33.333% |
## | 7.837% | 5.952% | 17.460% | 2.083% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Column Total | 301 | 175 | 430 | 102 | 1008 |
## -------------|-----------|-----------|-----------|-----------|-----------|
##
## Restricted Weighting
##
## Cell Contents
## |-------------------------|
## | Count |
## | Row Percent |
## | Total Percent |
## |-------------------------|
##
## Total Observations in Table: 1008
##
## | Model
## Size | Bayes | Linear | MinMax | SerPar | Row Total |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Small | 60 | 94 | 88 | 94 | 336 |
## | 17.857% | 27.976% | 26.190% | 27.976% | 33.333% |
## | 5.952% | 9.325% | 8.730% | 9.325% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Medium | 82 | 133 | 96 | 25 | 336 |
## | 24.405% | 39.583% | 28.571% | 7.440% | 33.333% |
## | 8.135% | 13.194% | 9.524% | 2.480% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Large | 66 | 131 | 111 | 28 | 336 |
## | 19.643% | 38.988% | 33.036% | 8.333% | 33.333% |
## | 6.548% | 12.996% | 11.012% | 2.778% | |
## -------------|-----------|-----------|-----------|-----------|-----------|
## Column Total | 208 | 358 | 295 | 147 | 1008 |
## -------------|-----------|-----------|-----------|-----------|-----------|
##
## Effect of Sex
Nothing very substantial. Females prefer MinMax over Linear compared to Men.
Effect of Background
## Model
## FieldOfStudy Bayes Linear MinMax SerPar
## Business and Econ. 103 133 178 54
## Education 70 56 70 20
## Fine Arts (e.g. Music, Theater, Film) 44 57 46 33
## Health Sciences 27 5 31 9
## Humanities 102 89 142 63
## Science and Tech. 79 114 133 34
## Social Sciences 84 79 125 36Effect of Background – Restricted
## Model
## FieldOfStudy Bayes Linear MinMax SerPar
## Business and Econ. 42 86 72 34
## Education 29 41 25 13
## Fine Arts (e.g. Music, Theater, Film) 15 35 28 12
## Health Sciences 16 2 9 9
## Humanities 47 62 51 38
## Science and Tech. 25 78 59 18
## Social Sciences 34 54 51 23Effect of Background – Free Sytle
## Model
## FieldOfStudy Bayes Linear MinMax SerPar
## Business and Econ. 61 47 106 20
## Education 41 15 45 7
## Fine Arts (e.g. Music, Theater, Film) 29 22 18 21
## Health Sciences 11 3 22 0
## Humanities 55 27 91 25
## Science and Tech. 54 36 74 16
## Social Sciences 50 25 74 13… which is due to…
## Gender
## FieldOfStudy Female Male
## Business and Econ. 288 180
## Education 108 108
## Fine Arts (e.g. Music, Theater, Film) 72 108
## Health Sciences 36 36
## Humanities 288 108
## Science and Tech. 108 252
## Social Sciences 216 108Effect of Age
## , , Style = Free
##
## Model
## Age Bayes Linear MinMax SerPar
## 21-29 53 13 61 17
## 30-39 136 110 224 34
## 40-49 76 49 109 36
## 50-59 22 3 33 14
## 60 or older 14 0 3 1
##
## , , Style = Rest
##
## Model
## Age Bayes Linear MinMax SerPar
## 21-29 34 44 50 16
## 30-39 95 207 140 62
## 40-49 56 77 85 52
## 50-59 22 18 15 17
## 60 or older 1 12 5 0