Package 'reproducer'

Description

This function assumes an ABBA crossover experiment has reported means and variances for each technique in each time period. We calculate the weighted mean and pooled within group variance for the observations arising from the two different sets of materials for a specific technique.

Usage

aggregateIndividualDocumentStatistics(D1.M, D1.SD, D1.N, D2.M, D2.SD, D2.N)

Arguments

Value

data frame incl. the overall weighted mean and pooled standard deviation

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

aggregateIndividualDocumentStatistics(10, 2, 20, 15, 2, 20)
#     M SD
# 1 12.5  2

Description

The function calculates sample statistics based on the residuals from a specified experiment

Usage

AnalyseResiduals(Residuals, ExperimentName = "ExpName")

Arguments

Value

A dataframe identifying the ExperimentName and its associated sample parameter: Length, Mean, Median, Variance, Standard deviation, skewness, kurtosis, the outcome of the Shapiro and Anderson-Darling normality test and the number of outliers.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ExpData=rnorm(30,0,1)
set.seed(123)
AnalyseResiduals(Residuals=ExpData,ExperimentName='ExpName')
#  ExperimentName       Mean      Median  Variance   Skewness Kurtosis ShapiroTest AndersonDarling
#1        ExpName -0.1396192 -0.01943395 0.8424521 -0.1964175 4.559587   0.1608315       0.1316835
#  NumOut
#  1

Description

Boxplot and density curve overlaid on histogram

Usage

boxplotAndDensityCurveOnHistogram(df, colName, limLow, limHigh)

Arguments

Value

A figure being a density curve overlaid on histogram

Author(s)

Lech Madeyski

Examples

library(ggplot2)
library(grid)
library(gridExtra)
boxplotAndDensityCurveOnHistogram(Madeyski15EISEJ.PropProjects, "STUD", 0, 100)
boxplotAndDensityCurveOnHistogram(Madeyski15SQJ.NDC, "simple", 0, 100)

Description

Box plot

Usage

boxplotHV(df, colName, limLow, limHigh, isHorizontal)

Arguments

Value

A box plot

Author(s)

Lech Madeyski

Examples

boxplotHV(Madeyski15EISEJ.PropProjects, "STUD", 0, 100, TRUE)
boxplotHV(Madeyski15EISEJ.PropProjects, "STUD", 0, 100, FALSE)
boxplotHV(Madeyski15SQJ.NDC, "simple", 0, 100, FALSE)
boxplotHV(Madeyski15SQJ.NDC, "simple", 0, 100, TRUE)

Description

This function is a helper function that calculates one element of the standardized mean difference effect size variance based on Hedges and Olkin p128-131.

Usage

calc.a(f, A)

Arguments

Value

The value of a.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

reproducer:::calc.a(10,2/10)
# [1] 0.2128649

Description

This function is a helper function that calculates one element of the standardized mean difference effect size variance based on Hedges and Olkin p128-131.

Usage

calc.b(f)

Arguments

Value

The value of b.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

reproducer:::calc.b(8)
#0.08649774

Description

This function does a non-parametric analysis of a randomized blocks experiment assuming 2 blocks and 2 treatment conditions.

Usage

Calc4GroupNPStats(
  x1,
  x2,
  x3,
  x4,
  sigfig = -1,
  alpha = 0.05,
  alternative = "two.sided"
)

Arguments

Value

The function returns the Cliff's d and its variance, the probability of superiority, phat, and its variance for the 4 group experiment experiment.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

set.seed(123)
x <- list()
x[[1]] <- rnorm(10, 0, 1)
x[[2]] <- rnorm(10, 0.8, 1)
x[[3]] <- rnorm(10, 0.5, 1)
x[[4]] <- rnorm(10, 1.3, 1)
as.data.frame(
Calc4GroupNPStats(x[[1]], x[[2]], x[[3]], x[[4]], sigfig = -1, alpha = 0.05)
)
#   N phat    phat.var  phat.df phat.test  phat.pvalue phat.sig phat.ci.upper
# 1 40 0.17 0.004966667 31.00131 -4.682539 5.324252e-05     TRUE     0.3137336
#  phat.ci.lower     d       vard d.sig d.ci.lower d.ci.upper        cor       sqse
# 1    0.02626639 -0.66 0.02060121  TRUE -0.8545073 -0.3031667 -0.3473684 0.01315789
#        ctvar n1 n2 sigCVt
# 1 0.005990797 20 20   TRUE
as.data.frame(
Calc4GroupNPStats(x[[1]], x[[2]], x[[3]], x[[4]], sigfig = -1, alpha = 0.05,
alternative = "less")
)
#   N phat    phat.var  phat.df phat.test  phat.pvalue phat.sig phat.ci.upper
# 1 40 0.17 0.004966667 31.00131 -4.682539 2.662126e-05     TRUE     0.2894908
#  phat.ci.lower     d       vard d.sig d.ci.lower d.ci.upper        cor       sqse
# 1             0 -0.66 0.02060121  TRUE         -1 -0.3677704 -0.3473684 0.01315789
#        ctvar n1 n2 sigCVt
# 0.005990797 20 20   TRUE
as.data.frame(
Calc4GroupNPStats(x[[2]], x[[1]], x[[4]], x[[3]], sigfig = -1, alpha = 0.05,
alternative = "greater")
)
#   N phat    phat.var  phat.df phat.test  phat.pvalue phat.sig phat.ci.upper
# 1 40 0.83 0.004966667 31.00131  4.682539 2.662126e-05     TRUE             1
#  phat.ci.lower    d       vard d.sig d.ci.lower d.ci.upper       cor       sqse
# 1     0.7105092 0.66 0.02060121  TRUE  0.3677704          1 0.3473684 0.01315789
#        ctvar n1 n2 sigCVt
# 1 0.005990797 20 20   TRUE

#as.data.frame(
#Calc4GroupNPStats(x[[1]],x[[2]],x[[3]],x[[4]],sigfig=-1,alpha=0.00))
#Error in testfunctionParameterChecks(alternative = alternative, alpha = alpha,  :
#  Invalid alpha parameter, select alpha in range (0.0001,0.2)

Description

This functions is a helper function. It assesses the significance one-sided and two-sided statistical of Cliff's d based on its confidence interval. The type of test is determined by the parameter One.Sided.Tests, the direction of one-sided tests is determined by the parameter Positive.MD.

Usage

calcCliffdConfidenceIntervals(
  d.value,
  d.variance,
  d.df,
  alpha = 0.05,
  alternative = "two.sided"
)

Arguments

Value

The function returns a Boolean variable identifying whether the effect size is significant and the confidence interval bounds.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

reproducer:::calcCliffdConfidenceIntervals(d.value=0.5, d.variance=0.04,d.df=18)
# A tibble: 1 x 5
#  d.tvalue d.pvalue d.ci.lower d.ci.upper d.sig
#     <dbl>    <dbl>      <dbl>      <dbl> <lgl>
#1      2.5   0.0223     0.0479      0.782 TRUE

reproducer:::calcCliffdConfidenceIntervals(
  d.value=0.5,d.variance=0.04,d.df=18,alternative='greater')
# A tibble: 1 x 5
#  d.tvalue d.pvalue d.ci.lower d.ci.upper d.sig
#   <dbl>    <dbl>      <dbl>      <dbl> <lgl>
#1    2.5   0.0112      0.123          1 TRUE

reproducer:::calcCliffdConfidenceIntervals(
  d.value=0.2,d.variance=0.04,d.df=18,alternative='greater')
# A tibble: 1 x 3
#  d.tvalue d.pvalue d.ci.lower d.ci.upper d.sig
#     <dbl>    <dbl>      <dbl>      <dbl> <lgl>
#1        1    0.165     -0.133          1 FALSE

reproducer:::calcCliffdConfidenceIntervals(
  d.value=-0.5,d.variance=0.04,d.df=18,alternative='less')
# A tibble: 1 x 5
#  d.tvalue d.pvalue d.ci.lower d.ci.upper d.sig
#   <dbl>    <dbl>      <dbl>      <dbl> <lgl>
#1     -2.5   0.0112         -1     -0.123 TRUE

Description

This function is a helper function for meta-analysis of experiments using Cliff's d as an effect size. It returns the 100*(1-alpha/2)

Usage

calcCliffdTestStatistics(
  d.value,
  d.variance,
  d.df = 0,
  alpha = 0.05,
  alternative = "two.sided"
)

Arguments

Value

d.tvalue The value of the t-statistic

d.pvalue The p-value of the t-test if the parameter d.df>0, or the normal probability value if d.df=0

d.ci.lower The lower 100*(1-alpha/2)

d.ci.upper The upper 100*(1-alpha/2)

d.sig The significance of the statistical test of the d.tvalue return value at the alpha level for one sided tests and aplha/2 for two sided tests as specified by the input parameter alternative

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

aveCliffd=mean(c(0.84,0.2,-0.04,0.44,0.76))
aveCliffdvar=sum(c(0.04,0.18,0.21,0.15,0.06))/25
df=45
calcCliffdTestStatistics(d.value=aveCliffd,d.variance=aveCliffdvar,d.df=df)
# A tibble: 1 x 5
#   d.tvalue d.pvalue d.ci.lower d.ci.upper d.sig
#      <dbl>    <dbl>      <dbl>      <dbl> <lgl>
# 1     2.75  0.00855     0.0923      0.692 TRUE

Description

This function provides single-sided and two-sided confidence interval of an effect size (assuming that the null hypothesis value is zero).

Usage

calcEffectSizeConfidenceIntervals(
  effectsize,
  effectsize.variance,
  effectsize.df = 0,
  alpha = 0.05,
  alternative = "two.sided",
  UpperValue = Inf,
  LowerValue = -Inf
)

Arguments

Value

The value of the test statistic, the p.value of test statistic, the upper and lower confidence interval of the effect size, a logical value specifying whether the effect size is significantly different from zero based on the confidence interval and the lower and upper confidence interval bounds.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

reproducer:::calcEffectSizeConfidenceIntervals(
  effectsize=0.37,effectsize.variance=0.00847,effectsize.df=11.1,
  alpha=0.05,alternative='two.sided',UpperValue=0.5,LowerValue=-0.5)
# A tibble: 1 x 5
#  ES.test ES.pvalue ES.sig ES.ci.lower ES.ci.upper
#    <dbl>     <dbl> <lgl>        <dbl>       <dbl>
#1    4.02   0.00198 TRUE         0.168         0.5

Description

This functions is a helper function. It assesses the significance one-sided and two-sided statistical of the probability of superiority based on its confidence interval. The type of test and the direction of the test is determined by the parameter alternative which takes one of the values 'two.sided', 'greater' or 'less'.

Usage

calcPHatConfidenceIntervals(
  phat,
  phat.variance,
  phat.df,
  alpha = 0.05,
  alternative = "two.sided"
)

Arguments

Value

The function returns a Boolean variable identifying whether the effect size is significant and the confidence interval bounds.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

reproducer:::calcPHatConfidenceIntervals(.65,0.005,8)
# A tibble: 1 x 5
#  phat.test pvalue phat.sig phat.ci.lower phat.ci.upper
#     <dbl>  <dbl> <lgl>            <dbl>         <dbl>
# 1      2.12 0.0667 FALSE            0.487         0.813
reproducer:::calcPHatConfidenceIntervals(.65,0.005,8,alternative='greater')
# A tibble: 1 x 5
#  phat.test pvalue phat.sig phat.ci.lower phat.ci.upper
#      <dbl>  <dbl> <lgl>            <dbl>         <dbl>
# 1      2.12 0.0333 TRUE             0.519             1

Description

This function is a helper function for meta-analysis of experiments using PHat as an effect size. It returns the 100*(1-alpha/2)

Usage

calcPHatMATestStatistics(
  effectsize,
  effectsize.variance,
  effectsize.df = 0,
  alpha = 0.05,
  alternative = "two.sided"
)

Arguments

Value

ES.test The value of the t-statistic

ES.pvalue The p-value of the t-test if the parameter d.df>0, or the normal probability value if d.df=0

ES.sig The significance of the statistical test of the d.tvalue return value at the alpha level for one sided tests and aplha/2 for two sided tests as specified by the input parameter alternative.

ES.ci.lower The lower 100*(1-alpha/2)

ES.ci.upper The upper 100*(1-alpha/2)

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

avePHat=mean(c(0.92,0.6,0.48,0.72,0.88))
avePHatvar=sum(c(0.01,0.04,0.05,0.04,0.01))/25
PHatdf=sum(c(6.63,6.63,5.08,5.61,8))
calcPHatMATestStatistics(effectsize=avePHat-0.5,effectsize.variance=avePHatvar,effectsize.df=PHatdf)
# A tibble: 1 x 5
#   ES.test ES.pvalue ES.sig ES.ci.lower ES.ci.upper
#     <dbl>     <dbl> <lgl>        <dbl>       <dbl>
# 1    2.84   0.00778 TRUE        0.0622       0.378

Description

The function simulates two-group experiments and estimates the power, individual estimate error, and the small sample bias obtained obtained from the set of simulated experiments. The set of simulations for a specific mean difference are repeated for three different values of the difference between the treatment and control groups specified by the parameter "diff". The power is estimated as the percentage of experiments for which the mean of the experiment was significantly different from zero. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The output is a table of values identifying the observed values of three effect sizes: Cliff's d, PHat and StdMD, estimate error and their related small sample bias and power for each set of simulated experiments. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculate2GBias(
  mean = 0,
  sd = 1,
  N,
  reps,
  diff = c(0.2, 0.5, 0.8),
  Expected.StdMD = c(0.2, 0.5, 0.8),
  Expected.PHat = c(0.556, 0.638, 0.714),
  type = "n",
  seed = 223,
  StdAdj = 0
)

Arguments

Value

Design. Specifies the type of experiment, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

GrpSize. Specifies the size of each group in the simulated experiments.

Diff. The size of the difference between the control and treatment converted to an ordinal scale (Small, Medium, Large)

NPBias The relative difference between the average of the observed values of either Cliff's d or centralised PHat and the population value

StdMDBias. The relative difference between the average of the observed values of StdMDBias and the theoretical value

NPMdMRE The median of the absolute relative difference between the observed values of either Cliff's d or centralised PHat and the theoretical value for each experiment.

StdMDMdMRE The median of the relative difference between the observed values of StdMD and the population value for each experiment.

ObsPHat. The average of the Phat values found in the set of simulations.

ObsCliffd. The average of the Cliffd values found in the set of simulations.

ObsStdES. The average of StdMD values found in the set of simulations.

PHatPower. The percentage of the simulations, for a specific mean difference, for which the Phat estimate was significantly different from zero at the 0.05 alpha level based on one-sided tests.

CliffdPower. The percentage of the simulations, for a specific mean difference, for which the Cliff's d estimate was significantly different from zero at the 0.05 alpha level based on one-sided tests.

StdMDPower. The percentage of the simulations, for a specific mean difference, for which the StdMD estimate was significantly different from zero at the 0.05 alpha level based on one-sided tests.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

# as.data.frame(calculate2GBias(mean=0,sd=1,diff=c(0.2,0.5,0.8),Expected.StdMD=c(0.157,0.392,0.628),
#  Expected.PHat=c(0.544,0.609,0.671), N=5,reps=50, type="n", seed=523, StdAdj =0.5 ))
# Results for reps=100 (due to NOTE "Examples with CPU (user + system) or elapsed time > 5s"):
#    Design GrpSize   Diff        NPBias  StdMDBias  NPMdMRE StdMDMdMRE ObsPHat ObsCliffd  ObsSt..
# 1 2G_n_het       5  Small -6.308085e-16 0.07088601 3.272727  3.2700082  0.5440    0.0880 0.168..
# 2 2G_n_het       5 Medium  3.486239e-02 0.09914637 1.385321  1.3502057  0.6128    0.2256 0.430..
# 3 2G_n_het       5  Large  2.222222e-02 0.10446123 0.754386  0.8626523  0.6748    0.3496 0.693..
as.data.frame(calculate2GBias(mean=0,sd=1,diff=c(0.283,0.707104,1.131374),
 Expected.StdMD=c(0.157,0.392,0.628),Expected.PHat=c(0.556,0.636,0.705),N=10, reps=20,
 type="lap",seed=1423,StdAdj=0.5 ))
 #Parameter reps changed due to NOTE "Examples with CPU (user + system) or elapsed time > 5s"
 #Results for reps=100:
#      Design GrpSize   Diff      NPBias    StdMDBias   NPMdMRE StdMDMdMRE ObsPHat ObsCliffd  Ob..
#1 2G_lap_het      10  Small -0.11071429 -0.080855612 1.8928571  2.1256888  0.5498    0.0996 0.1..
#2 2G_lap_het      10 Medium -0.07426471  0.003940804 0.6323529  0.8170856  0.6259    0.2518 0.3..
#3 2G_lap_het      10  Large -0.05756098  0.023696619 0.4146341  0.5447941  0.6932    0.3864 0.6..

Description

The function simulates multiple two-group experiments and estimates the Type1 Error rate obtained from the set of simulated experiments. The Type1 Error is estimated as the percentage of experiments for which the mean the experiment was significantly different from zero at the 0.05 significance level using two-sided tests. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The output is a set of values identifying three observed effect size estimates (Cliff's d, PHat and StdMD) and their related type 1 error rates. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculate2GType1Error(
  mean = 0,
  sd = 1,
  N = 10,
  reps,
  type = "n",
  seed = 123,
  StdAdj = 0
)

Arguments

Value

Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

GrpSize. Specifies the size of each group in the individual experiments.

ObsPHat. The average Phat values found in the set of simulations.

ObsCliffd. The average Cliffd values found in the set of simulations.

ObsStdES. The average of StdMD values found in the set of simulations.

PHatType1ER. The proportion of the simulations for which the Phat estimate was significantly different from zero at the nominated alpha level.

CliffdType1ER. The proportion of the simulations for which the Cliff's d estimate was significantly different from zero at the nominated alpha level.

StdMDType1ER. The proportion of the simulations for which the StdMD estimate was significantly different from zero at the nominated 0.05 significance level.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

calculate2GType1Error(mean=1,sd=3,N=10,reps=100,type="g",seed=3256,StdAdj = 0)
# A tibble: 1 x 8
#   Design GrpSize ObsPHat ObsCliffd ObsStdES PHatType1ER CliffdType1ER StdESType1ER
#   <chr>  <chr>     <dbl>     <dbl>    <dbl>       <dbl>         <dbl>        <dbl>
# 1 2G_g   10        0.498   -0.0034 -0.00464        0.02          0.01         0.02

Description

The function simulates four-group experiments and estimates of the power, individual estimate error and small sample bias obtained from a set of simulated experiments. The function produces three set of simulations obtained using three different values of the mean difference between the treatment and control groups as specified by the parameter "diff". The power is estimated as the percentage of simulated experiments for which the mean of the experiment was significantly different from zero using one-sided tests. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The output is a table of values identifying the observed values of three effect sizes: Cliff's d, PHat and StdMD, their relted estimate error, small sample bias and power for each set of simulated experiments. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size" and its Supplementary Material.

Usage

calculate4GBias(
  mean = 0,
  sd = 1,
  N,
  reps,
  diff = c(0.2, 0.5, 0.8),
  Expected.StdMD = c(0.2, 0.5, 0.8),
  Expected.PHat = c(0.556, 0.638, 0.714),
  type = "n",
  seed = 223,
  StdAdj = 0,
  Blockmean = 0
)

Arguments

Value

Design. Specifies the type of experiment 2G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

BEIncluded. Specifies whether or not a block effect was introduced.

GrpSize. Specifies the size of each group in the individual experiments.

Diff. The size of the difference between the control and treatment converted to an ordinal scale (Small, Medium, Large)

NPBias The relative difference between the average of the observed values of either Cliff's d or centralised PHat and the population value

StdMDBias. The relative difference between the average of the observed values of StdMDBias and the theoretical value

NPMdMRE The median of the absolute relative difference between the observed values of either Cliff's d or centralised PHat and the theoretical value for each experiment.

StdMDMdMRE The median of the relative difference between the observed values of StdMD and the population value for each experiment.

ObsPHat. The average Phat value found for each simulation.

ObsCliffd. The average Cliffd value found for each simulation.

ObsStdES. The average of StdMD calculated for each simulation.

PHatPower. The proportion of the simulations, for a given mean difference, for which the Phat estimate was significantly different from zero at the 0.05 alpha level based on one-sided tests.

CliffdPower. The proportion of the simulations, for a given mean difference, for which the Cliff's d estimate was significantly different from zero at the 0.05 alpha level based on one-sided tests.

StdMDPower. The proportion of the simulations, for a given mean difference, for which the StdMD estimate was significantly different from zero at the 0.05 alpha level based on one-sided tests.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

#as.data.frame(calculate4GBias(mean=0,sd=1,diff=c(0.266,0.72375,1.43633),
#  Expected.StdMD=c(0.2,0.5,0.8),Expected.PHat=c(0.575,0.696,0.845),N=10,reps=200,type="l",
#  seed=17+1823,StdAdj=0,Blockmean=0))
#  Design BEIncluded GrpSize   Diff      NPBias StdMDBias   NPMdMRE StdMDMdMRE  ObsPHat ObsCliffd.
#  1 4G_l         No      10  Small -0.05933333 0.1247408 0.8666667  1.2047848 0.570550   0.1411..
#  2 4G_l         No      10 Medium -0.01760204 0.1565643 0.3112245  0.4426859 0.692550   0.3851..
#  3 4G_l         No      10  Large -0.00326087 0.2273638 0.1594203  0.2924361 0.843875   0.6877..
as.data.frame(calculate4GBias(mean=1,sd=3,diff=c(0.1225,0.3415,0.6224),
 Expected.StdMD=c(-0.208,-0.52,-0.833),Expected.PHat=c(0.444,0.360,0.277),N=20,reps=30,type="g",
 seed=17+977,StdAdj=0 ,Blockmean=0.5))
# Results for reps=200:
#  Design BEIncluded GrpSize   Diff     NPBias  StdMDBias   NPMdMRE StdMDMdMRE   ObsPHat  ObsCli..
#1   4G_g        Yes      20  Small 0.04274554 0.02242895 0.8370536  0.7960052 0.4416062 -0.1167..
#2   4G_g        Yes      20 Medium 0.01959821 0.01585829 0.3348214  0.3210435 0.3572562 -0.2854..
#3   4G_g        Yes      20  Large 0.01303251 0.01515967 0.1905830  0.1871956 0.2740938 -0.4518..

Description

The function simulates multiple four-group experiments and estimates the Type1 Error rate obtained from the set of simulated experiments. The Type1 Error is estimated as the percentage of experiments for which the mean the experiment was significantly different from zero at the 0.05 significance level using two-sided tests. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The output is a set of values identifying three observed effect size estimates (Cliff's d, PHat and StdMD) and their related type 1 error rates. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculate4GType1Error(
  mean = 0,
  sd = 1,
  N = 10,
  reps = 10,
  type = "n",
  seed = 123,
  StdAdj = 0,
  Blockmean = 0
)

Arguments

Value

Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

GrpSize. Specifies the size of each group in the simulations.

BEIncluded. Specifies whether or not a block effect was introduced.

ObsPHat. The average of the average Phat values found in the set of simulations.

ObsCliffd. The average of the average Cliffd values found in the set of simulations.

ObsStdES. The average of StdMD values found in the set of simulations.

PHatType1ER. The percentage of the simulations for which the Phat estimate was significantly different from zero at the 0.05 alpha level.

CliffdType1ER. The percentage of the simulations for which the overall Cliff's d estimate was significantly different from zero at the 0.05 alpha level.

StdMDType1ER. The percentage of the simulations for which the overall StdMD estimate was significantly different from zero at the 0.05 significance level.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

as.data.frame(calculate4GType1Error(mean=0,sd=1,N=40,reps=100,type="n",seed=17+1056,StdAdj = 0.5,
 Blockmean=0.5))
 # Results for reps=300
#    Design GrpSize BEIncluded   ObsPHat   ObsCliffd   ObsStdES PHatType1ER CliffdType1ER StdES..
#1 4G_n_het      40        Yes 0.5034729 0.006945833 0.01316457        0.03    0.02333333 0.046..

#as.data.frame(calculate4GType1Error(mean=0,sd=1,N=40,reps=300,type="lap",seed=17+2056,
#  StdAdj = 0.5,Blockmean=0.5))
#      Design GrpSize BEIncluded   ObsPHat    ObsCliffd  ObsStdES PHatType1ER CliffdType1ER Std..
#1 4G_lap_het      40        Yes 0.4992708 -0.001458333 0.0014446  0.04333333          0.04  0.06

Description

This function calculates the following statistics for a set of data: length, mean, median, variance, standard error of the mean, and confidence interval bounds. The input data must be a vector of 2 or more numerical values.

Usage

calculateBasicStatistics(x, alpha = 0.05)

Arguments

Value

A dataframe comprising the length, mean, variance, standard error and confidence limit bounds of the input data x.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ShortExperimentNames <- c("E1", "E2", "E3", "E4")
FullExperimentNames <- c("EUBAS", "R1UCLM", "R2UCLM", "R3UCLM")
Metrics <- c("Comprehension", "Modification")
Groups <- c("A", "B", "C", "D")
Type <- c(rep("4G", 4))
StudyID <- "S2"
Control <- "SC"
ReshapedData <- ExtractExperimentData(
  KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM,
  ExperimentNames = FullExperimentNames, idvar = "ParticipantID",
  timevar = "Period", ConvertToWide = TRUE
)
NewTable <- ConstructLevel1ExperimentRData(
  ReshapedData, StudyID, ShortExperimentNames, Groups,
  Metrics, Type, Control
)
calculateBasicStatistics(NewTable$r)
#    N    Mean Median Variance      SE LowerBound UpperBound
# 1 32 0.06175 0.1688   0.2482 0.08808    -0.1109     0.2344

Description

This function implements finds Cliff's d and its confidence intervals. The null hypothesis is that for two independent group, P(X<Y)=P(X>Y). The function reports a 1-alpha confidence interval for P(X>Y)-P(X<Y). The algorithm computes a confidence interval for Cliff's d using the method in Cliff, 1996, p. 140, eq 5.12. The function is based on code produce by Rand Wilcox but has been amended. The plotting function has been removed and the dependency on Wilcox's binomci function has been removed. Construction of confidence intervals if values in one group are all larger than values in the other group has been amended to use the smallest non-zero variance method. Upper and lower confidence interval bounds cannot assume invalid values, i.e. values <-1 or >1.

Usage

calculateCliffd(x, y, alpha = 0.05, sigfig = -1)

Arguments

Value

list including the value of Cliffs d its consistent variance and confidence intervals and the equivalent probability of superiority value and its confidence intervals.

Author(s)

Rand Wilcox, amendments Barbara Kitchenham and Lech Madeyski

Examples

x=c(1.2,3,2.2,4,2.5,3)
y=c(3,4.2,4,6,7,5.9)
calculateCliffd(x,y)
#  $n1
# [1] 6
# $n2
# [1] 6
# $d
# [1] -0.8611111
# $sqse.d
# [1] 0.02017931
# $phat
# [1] 0.06944444
z=c(1,2,3,4)
y=c(5,6,7,8)
calculateCliffd(z,y)
# $n1
# [1] 4
# $n2
# [1] 4
# $d
# [1] -1
# $sqse.d
# [1] 0.009765625
# $phat
# [1] 0

Description

This function calculates the following statistics data within groups: length, mean, median, variance, standard error of the mean, and confidence interval bounds.

Usage

calculateGroupSummaryStatistics(x, Group)

Arguments

Value

A dataframe comprising the number, mean, variance, standard error and confidence limit bounds of the data in each category

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ShortExperimentNames <- c("E1", "E2", "E3", "E4")
FullExperimentNames <- c("EUBAS", "R1UCLM", "R2UCLM", "R3UCLM")
Metrics <- c("Comprehension", "Modification")
Groups <- c("A", "B", "C", "D")
Type <- c(rep("4G", 4))
StudyID <- "S2"
Control <- "SC"
ReshapedData <- ExtractExperimentData(
  KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM,
  ExperimentNames = FullExperimentNames, idvar = "ParticipantID", timevar = "Period",
  ConvertToWide = TRUE
)
NewTable <- ConstructLevel1ExperimentRData(
  ReshapedData, StudyID,
  ShortExperimentNames, Groups, Metrics, Type, Control
)
SeqGroupLev <- NULL
N.NT <- length(NewTable$r)
for (i in 1:N.NT) {
  if (NewTable$n[i] <= 8) SeqGroupLev[i] <- as.character(NewTable$n[i])
  if (NewTable$n[i] > 8) SeqGroupLev[i] <- as.character(9)
}
calculateGroupSummaryStatistics(NewTable$r, Group = SeqGroupLev)
#     N    Mean  Median Variance  StDev     SE
#  1  4 -0.0833 -0.1699   0.2314 0.4810 0.2405
#  2 12  0.3658  0.4477   0.2109 0.4592 0.1326
#  3 16 -0.1300 -0.2214   0.1933 0.4397 0.1099

Description

This function calculates Hedges g and Hedges g adjusted given the basic experimental statistics - the mean values for participants, number of observations (participants), and standard deviation in both the control group and the treatment group. . Hence, the function assumes the data is held as summary statistics including the control group mean, standard deviation and sample size and equivalent values for treatment group

Usage

calculateHg(Mc, Mt, Nc, Nt, SDc, SDt)

Arguments

Value

data frame composed of Hedges' g and Hedges' g adjusted effect sizes

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

calculateHg(10, 15, 20, 20, 2, 2)
#    Hg    HgAdjusted
# 1  2.5   2.450276

Description

@title calculateKendalltaupb @description Computes point bi-serial version of Kendall's tau plus a 1-alpha confidence interval using the method recommended by Long and Cliff (1997). The algorithm is based on Wilcox's code but was extended to return the consistent variance and the confidence intervals based on the t-distribution. Also added a Diagnostic parameter to output internal calculations.

Usage

calculateKendalltaupb(x, y = NULL, alpha = 0.05, alternative = "two.sided")

Arguments

Value

list containing the estimate of Kendall's tau, the consistent variance of tau and its confidence intervals based on the t-test (recommended by Long and Cliff)

Author(s)

Rand Wilcox, Barbara Kitchenham and Lech Madeyski

Examples

set.seed(123)
a <- c(1.2, 3, 1.8, 2, 2, 0.5, 0.5, 1, 3, 1)
b <- c(1, 1, 1, 1, 1, 0, 0, 0, 0, 0)
calculateKendalltaupb(a,b,alpha=.05)
#$cor
#[1] 0.3555556
#$cit
#[1] -0.1240567  0.5555556
#$n
#[1] 10
#$df
#[1] 7
#$consistentvar
#[1] 0.04113925
#$sig
#[1] FALSE
set.seed(234)
a2=c(rnorm(10,0,1),rnorm(10,0.5,1))
b2=c(rep(0,10),rep(1,10))
calculateKendalltaupb(a2,b2,alpha=.05,alternative='greater')
#$cor
#[1] 0.2842105
#$cit
#[1] 0.06517342 0.52631579
#$n
#[1] 20
#$df
#[1] 17
#consistentvar
#1] 0.01585379
#$sig
#[1] TRUE
calculateKendalltaupb(a2,b2,alpha=.05,alternative='less')
#$cor
#[1] 0.2842105
#$cit
#[1] -0.5263158  0.5032476
#$n
#[1] 20
#$df
#[1] 17
#$consistentvar
#[1] 0.01585379
#$sig
#[1] FALSE

Description

The function uses a simulates a large experiment to estimate the asymptotic values of the probability of superiority, Cliff's d and the standardized mean difference data for a four group randomized blocks experiment for four different distributions: Normal (i.e. type='n'), log-normal (i.e. type='l'), gama (i.e. type='g') and Laplace (i.e., type='lap').

Usage

calculateLargeSampleRandomizedBlockDesignEffectSizes(
  meanC = 0,
  sdC = 1,
  diff,
  N = 5e+06,
  type = "n",
  Blockmean = 0,
  StdAdj = 0
)

Arguments

Value

A tibble identifying the sample statistics and the values of the probability of superiority, Cliff's d and StdMD (labelled StdES)

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

set.seed=400
calculateLargeSampleRandomizedBlockDesignEffectSizes(
  meanC=0, sdC=1, diff=.5, N=100000, type='n',Blockmean=0.5,StdAdj = 0)
#  MeanC   SdC MeanT   SdT    BE  Phat Cliffd   UES   Var StdES
#  <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>
#1     0     1   0.5     1   0.5 0.638  0.277 0.501 0.998 0.502

Description

The function simulates a large experiment to estimate the asymptotic values of the probability of superiority, Cliff's d and the standardized mean difference data for a two group randomized experiment for four different distributions: Normal (i.e. type="n"), log-normal (i.e. type="l"), gama (i.e. tyep="g") and Laplace (i.e., type="lap").

Usage

calculateLargeSampleRandomizedDesignEffectSizes(
  meanC = 0,
  sdC = 1,
  diff = 0,
  N = 5e+06,
  type = "n",
  StdAdj = 0,
  reporttrans = "No"
)

Arguments

Value

A tibble identifying the sample statistics and the values of the probability of superiority, Cliff's d and StdMD (labelled StdES)

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

set.seed=400
calculateLargeSampleRandomizedDesignEffectSizes(meanC=0, sdC=1, diff=.5,
N=10000, type="n",StdAdj = 0) #N=100000, type="n",StdAdj = 0)
# A tibble: 1 x 9
#     MeanC   SdC MeanT   SdT  Phat Cliffd   UES   Var StdES
#     <dbl> <dbl> <dbl> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>
#1  0.00642  1.00 0.519 0.995 0.642  0.284 0.513 0.996 0.514
#1     0     1   0.5     1 0.637  0.275 0.499  1.01 0.497
as.data.frame(calculateLargeSampleRandomizedDesignEffectSizes(meanC=0, sdC=1, diff=0.707104,
N=100000, type="l",StdAdj = 0,reporttrans="Yes"))
#N=1000000, type="l",StdAdj = 0,reporttrans="Yes"))
#     MeanC     StdC    MeanT     StdT      Phat    Cliffd      UES      Var
#1 1.647446 2.219114  3.33124 4.404537 0.6926779 0.3853558 1.683795 12.1622
#       StdES   MeanCTrans MeanTTrans StdCTrans StdTTrans PhatTrans CliffdTrans
#1  0.4828175 -0.004298487  0.7066049  1.001199 0.9963736 0.6926779   0.3853558
#   UESTrans VarTrans StdESTrans
#1 0.7109034  0.99758  0.7117651

Description

This function analyses data on r values obtained in the format obtained from the ConstructLevel1ExperimentRData function and finds the r-value for each metric for each experiment.

Usage

CalculateLevel2ExperimentRData(
  Level1Data,
  Groups,
  StudyID,
  ExperimentNames,
  Metrics,
  Type
)

Arguments

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ShortExperimentNames <- c("E1", "E2", "E3", "E4")
FullExperimentNames <- c("EUBAS", "R1UCLM", "R2UCLM", "R3UCLM")
Metrics <- c("Comprehension", "Modification")
Groups <- c("A", "B", "C", "D")
Type <- c(rep("4G", 4))
StudyID <- "S2"
Control <- "SC"
# Obtain experimental data from each file and put in wide format
ReshapedData <- ExtractExperimentData(
  KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM,
  ExperimentNames = FullExperimentNames, idvar = "ParticipantID",
  timevar = "Period", ConvertToWide = TRUE
)
Lev1Data <- ConstructLevel1ExperimentRData(
  ReshapedData, StudyID, ShortExperimentNames, Groups,
  Metrics, Type, Control
)
CalculateLevel2ExperimentRData(Lev1Data,
  Groups = Groups, StudyID = StudyID,
  ExperimentNames = ShortExperimentNames, Metrics = Metrics, Type = Type
)
# A tibble: 8 x 10
#  StudyID ExpID     N Metric        PooledVar1 PooledVar2 VarProp PooledVar PooledDiffVar    r.Exp
#  <chr>   <chr> <int> <chr>              <dbl>      <dbl>   <dbl>     <dbl>         <dbl>    <dbl>
# 1 S2      S2E1     24 Comprehension     0.0148     0.0212   0.412    0.0180        0.0248  0.311
# 3 S2      S2E2     22 Comprehension     0.0487     0.0224   0.684    0.0356        0.0534  0.250
# 4 S2      S2E2     22 Modification      0.0445     0.0266   0.626    0.0356        0.0628  0.117
# 5 S2      S2E3     22 Comprehension     0.0353     0.0402   0.467    0.0377        0.105  -0.391
# 6 S2      S2E3     22 Modification      0.0433     0.0414   0.511    0.0424        0.0997 -0.176
# 7 S2      S2E4     18 Comprehension     0.0439     0.0237   0.649    0.0338        0.0355  0.475
# 8 S2      S2E4     18 Modification      0.0322     0.0592   0.353    0.0457        0.0894  0.0222

Description

The function simulates multiple five group families of either two-group or four-group experiments and estimates the power, individual estimate error, and the small sample bias obtained after synthesizing the analysis results obtained from the experiments in each family. The power is estimated as the percentage of families for which the overall mean of the five experiments was significantly different from zero. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The simulations can be repeated for different mean differences between the control mean and treatment mean depending on the parameter diff. The output is a table of values identifying the observed values of three effect sizes: Cliff's d, PHat and StdMD, estimate error and their related small sample bias and power for each set of simulated families. The synthesis method for all the effect sizes is based on calculating the overall mean and variance for experiments in each family and then using those values to calculate the overall effect size and its variance. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculateMABias(
  mean = 0,
  sd = 1,
  N,
  reps,
  diff = c(0.2, 0.5, 0.8),
  Experiments = 5,
  Expected.StdMD = c(0.2, 0.5, 0.8),
  Expected.PHat = c(0.556, 0.638, 0.714),
  type = "n",
  FourG = FALSE,
  seed = 223,
  StdAdj = 0,
  Blockmean = 0,
  StdExp = 0,
  MAMethod = "PM",
  alpha = 0.05
)

Arguments

Value

Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

BEIncluded. Specifies whether or not a block effect was introduced. Always set to "No" for two-group experiments.

GrpSize. Specifies the size of each group in the individual experiments.

Diff. The size of the difference between the control and treatment converted to an ordinal scale (Small, Medium, Large)

NPBias The relative difference between the average of the observed values of either Cliff's d or centralised PHat and the population value

StdMDBias. The relative difference between the average of the observed values of StdMDBias and the theoretical value

NPMdMRE The median of the absolute relative difference between the observed values of either Cliff's d or centralised PHat and the theoretical value for each experiment.

StdMDMdMRE The median of the absolute relative difference between the observed values of StdMD and the population value for each experiment.

ObsPHat. The average of the average Phat value found for each family in the set of simulations.

ObsCliffd. The average of the average Cliffd value found for each family in the set of simulations.

ObsStdES. The average of StdMD calculated for each family in the set of simulations.

PHatPower. The percentage of the simulations, for a specific mean difference, for which the overall Phat estimate was significantly different from zero at the nominated alpha level using one-sided tests.

CliffdPower. The percentage of the simulations, for a specific mean difference, for which the overall Cliff's d estimate was significantly different from zero at the nominated alpha level using one-sided tests.

StdMDPower. The percentage of the simulations, for a specific mean difference, for which the overall StdMD estimate was significantly different from zero at the nominated alpha level using one-sided tests.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

# as.data.frame(calculateMABias(mean=0,sd=1,N=10,diff=c(0.2,0.5,0.8), Experiments=5,reps=10,
# Expected.StdMD=c(0.2,0.5,0.8), Expected.PHat=c(0.556,0.638,0.714), type="n",FourG=FALSE,
# seed= 123, StdAdj = 0, Blockmean=0, StdExp=0))
#  Design Blockmean GrpSize   Diff     NPBias  StdMDBias   NPMdMRE StdMDMdMRE ObsPHat ObsCliffd..
#1   2G_n        No      10  Small 0.09285714 0.02606704 0.8928571  1.0741432  0.5612    0.1224..
#2   2G_n        No      10 Medium 0.03768116 0.01740262 0.2391304  0.4171896  0.6432    0.2864..
#3   2G_n        No      10  Large 0.03738318 0.01523651 0.2009346  0.2490287  0.7220    0.4440..
#  PHatPower CliffdPower StdESPower
#1       0.2         0.2        0.3
#2       0.7         0.7        0.7
#3       1.0         1.0        1.0
as.data.frame(calculateMABias(mean=0,sd=1,N=10,diff=c(0.2,0.5,0.8), Experiments=5,reps=4,
 Expected.StdMD=c(0.2,0.5,0.8), Expected.PHat=c(0.556,0.638,0.714), type="n",FourG=TRUE,
 seed= 123,StdAdj = 0.5,Blockmean=0.5,StdExp=0))
 #Results for reps=10
#    Design Blockmean GrpSize   Diff     NPBias  StdMDBias   NPMdMRE StdMDMdMRE ObsPHat ObsClif..
#1 4G_n_het       Yes      10  Small -0.1321429 -0.1372277 0.6696429  0.4698935  0.5486  0.0972..
#2 4G_n_het       Yes      10 Medium -0.1869565 -0.1882479 0.2318841  0.1472392  0.6122  0.2244..
#3 4G_n_het       Yes      10  Large -0.1864486 -0.2010029 0.1612150  0.1531253  0.6741  0.3482..
#  PHatPower CliffdPower StdESPower
#1       0.4         0.4        0.4
#2       0.9         0.9        0.8
#3       1.0         1.0        1.0

Description

The function simulates multiple five group families of either two-group or four-group experiments and estimates the Type1 Error rate obtained after synthesizing the analysis results obtained from the experiments in each family. The Type1 Error is estimated as the percentage of families for which the overall mean of the five experiments was significantly different from zero. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The simulations can be repeated for different sample sizes depending on the parameter N. The output is a table of values identifying three observed effect size estimates (Cliff's d, PHat and StdMD) and their related type 1 error rates for each set of simulated families. The synthesis method for all three effect sizes is based on calculating the overall mean and variance for the family of experiments, and then using those values to calculate the effect size variance and its variance. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculateMAType1Error(
  mean = 0,
  sd = 1,
  N = c(5, 10, 15, 20, 30, 40),
  reps,
  type = "n",
  seed = 123,
  Experiments = 5,
  FourG = FALSE,
  StdAdj = 0,
  Blockmean = 0,
  BlockStdAdj = 0,
  StdExp = 0,
  MAMethod = "PM",
  alpha = 0.05
)

Arguments

Value

Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

BEIncluded. Specifies whether or not a block effect was introduced. Always set to "No" for two-group experiments.

GrpSize. Specifies the size of each group in the individual experiments.

ObsPHat. The average of the average Phat value found for each family in the set of simulations.

ObsCliffd. The average of the average Cliffd value found for each family in the set of simulations.

ObsStdES. The average of StdMD calculated for each family in the set of simulations.

PHatType1ER. The percentage of the simulations, for a specific group size, for which the overall Phat estimate was significantly different from zero at the nominated alpha level.

CliffdType1ER. The percentage of the simulations, for a specific group size, for which the overall Cliff's d estimate was significantly different from zero at the nominated alpha level.

StdMDType1ER. The percentage of the simulations, for a specific group size, for which the overall StdMD estimate was significantly different from zero at the nominated alpha level.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

# as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="n",Experiments=5,
#  FourG=FALSE,StdAdj=0,Blockmean=0,seed=123))
#  Design BEIncluded GrpSize ObsPHat ObsCliffd     ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1   2G_n         No       5  0.4848   -0.0304 -0.054156883           0             0          0.0
#2   2G_n         No      10  0.5036    0.0072  0.002888142           0             0          0.1

#as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="l",Experiments=5,
#  FourG=FALSE,StdAdj=0,Blockmean=0,seed=123))
#   Design BEIncluded GrpSize ObsPHat ObsCliffd    ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1   2G_l         No       5  0.4848   -0.0304 -0.02789656           0             0          0.0
#2   2G_l         No      10  0.5036    0.0072  0.06473696           0             0          0.2

#as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="n",Experiments=5,
#  FourG=TRUE,StdAdj=0.5,Blockmean=0.5,seed=123))
#   Design BEIncluded GrpSize ObsPHat ObsCliffd   ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#  1 4G_n_het        Yes       5  0.5108    0.0216 0.01361820           0             0          0.1
#  2 4G_n_het        Yes      10  0.5069    0.0138 0.01700672           0             0          0.0

as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=5,type="l",Experiments=5,
 FourG=TRUE,StdAdj=0,Blockmean=0.5,seed=123))
 #Results for reps=10
#   Design BEIncluded GrpSize ObsPHat ObsCliffd   ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1   4G_l        Yes       5  0.5108    0.0216 0.07578257           0             0          0.2
#2   4G_l        Yes      10  0.5072    0.0144 0.04839936           0             0          0.0

Description

The function uses simulation to assess the accuracy when the mean difference is zero, and the type 1 error rates of parametric and non-parametric effect sizes for both two group randomized designs and four group randomized block designs, for each of four different distributions.

Usage

calculateNullESAccuracy(
  mean = 0,
  sd = 1,
  N = 10,
  reps = 10,
  type = "n",
  seed = 123,
  StdAdj = 0,
  Blockmean = 0.5
)

Arguments

Value

A tibble identifying the median absolute error for the effect sizes Cliff's d, phat and StdMD and the Type 1 error rate, estimated from the proportion of significant effect sizes in the simulated experiments.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

as.data.frame(
  calculateNullESAccuracy(
    mean=0,sd=1,N=10,reps=30,type='n',seed=123,StdAdj = 0,Blockmean = 0.5))
#   Design Obs CliffdAbsError PHatAbsError StdESdAbsError  varCliffd    varPHat
# 1   2G_n  20           0.20         0.10      0.2624447 0.05530851 0.01382713
# 2   4G_n  20           0.16         0.08      0.1848894 0.05447540 0.01361885
#    varStdES    ObsCliffd   ObsPHat     ObsStdES CliffdType1ER PHatType1ER
# 1 0.1425374  0.021333333 0.5106667 0.0001190251             0           0
# 2 0.1484728 -0.009333333 0.4953333 0.0295002335             0           0
#   StdESType1ER
# 1   0.03333333
# 2   0.03333333
#as.data.frame(
 # calculateNullESAccuracy(
 #   mean=0,sd=1,N=10,reps=100,type='n',seed=123,StdAdj = 0,Blockmean = 0.5))
#  Design Obs CliffdAbsError PHatAbsError StdESdAbsError  varCliffd    varPHat  varStdES ObsCliffd
#1   2G_n  20           0.21        0.105      0.3303331 0.08064949 0.02016237 0.2488365   -0.0010
#2   4G_n  20           0.16        0.080      0.2565372 0.05933430 0.01483358 0.1769521    0.0052
#  ObsPHat    ObsStdES CliffdType1ER PHatType1ER StdESType1ER
#1  0.4995 -0.02395895          0.07        0.08         0.08
#2  0.5026  0.03769940          0.01        0.01         0.02

Description

This function calculates the probability of superiority (i.e., Phat) and its confidence interval based on Brunner and Munzel (2000) heteroscedastic analog of WMW test. It is based on Wilcox'x bmp function with some amendments. It does not include a plotit facility. It uses the smallest non-zero variance to identify confidence intervals and statistical significance for values of Phat=0 and Phat=1. It ensure that confidence intervals do not take on invalid values such as values <0 or >1.

Usage

calculatePhat(x, y, alpha = 0.05, sigfig = -1)

Arguments

Value

list including the value of the t-test for PHat, the estimate of PHat and Cliff's d, and the confidence intervals for PHat.

Author(s)

Rand Wilcox amendments by Barbara Kitchenham and Lech Madeyski

Examples

x <- c(1.2, 3.0, 2.2, 4.0, 2.5, 3.0)
y <- c(3, 4.2, 4, 6, 7, 5.9)
reproducer:::calculatePhat(x, y)
# $test.stat
# [1] 6.381249
# $phat
# [1] 0.9305556
# $dhat
# [1] 0.8611111
# $sig.level
# [1] 0.0001191725
# $s.e.
# [1] 0.06747199
# $ci.p
# [1] 0.7783001 1.0000000
# $df
# [1] 9.148489
# Another example:
z <- c(1, 2, 3, 4)
y <- c(5, 6, 7, 8)
reproducer:::calculatePhat(z, y)
# $test.stat
# [1] 10.6066
# $phat
# [1] 1
# $dhat
# [1] 1
# $sig.level
# [1] 4.135921e-05
# $s.e.
# [1] 0.04419417
# $ci.p
# [1] 0.8918608 1.0000000
# $df
# [1] 6

Description

This helper function constructs the theoretical effect sizes and distribution statistics four (normal, lognormal, Laplace & gamma) given specific parameter values for the distributions and is used to support the calculation of population statistics for two and four group experiments.

Usage

calculatePopulationStatistics(mean, std, type = "n")

Arguments

Value

dataframe containing the expected standardized effect size, mean, variance,skewness and kurtosis statistics for samples from the specific distribution

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

reproducer:::calculatePopulationStatistics(mean=0, std=1, type='l')
#   RawMean RawVariance RawEffectSize RawSkewness RawKurtosis
#1 1.648721    4.670774      0.762874    6.184877    88.54343
reproducer:::calculatePopulationStatistics(mean=0, std=1, type='n')
#   RawMean RawVariance RawEffectSize RawSkewness RawKurtosis
# 1       0           1             0           0           3

Description

This function calculates the r value for a 2-group (2G) or 4-Group (4G) Crossover experiment for each sequence group and each outcome metric. The function returns both the exact r value and the r value based on pooled variances for each sequence group and outcome metric

Usage

CalculateRLevel1(
  Dataset,
  StudyID,
  Groups = c("A", "B", "C", "D"),
  ExperimentName,
  Metrics,
  Type,
  Control
)

Arguments

Details

script to obtain correlation coefficients

Value

table this is a tibble holding information identifying for each metric and sequence group the first time period and second time period variance, the pooled variance, the variance of the difference values and the exact r and pooled r. # importFrom stats # importFrom var # importFrom tibble

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ExperimentNames <- c("EUBAS", "R1UCLM", "R2UCLM", "R3UCLM")
ShortExperimentNames <- c("E1", "E2", "E3", "E4")
Metrics <- c("Comprehension", "Modification")
Type <- c("4G", "4G", "4G", "4G")
Groups <- c("A", "B", "C", "D")
StudyID <- "S2"
Control <- "SC"
# Obtain experimental data from a file and put in wide format
dataset2 <- KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM
ReshapedData <- ExtractExperimentData(dataset2,
  ExperimentNames = ExperimentNames,
  idvar = "ParticipantID", timevar = "Period", ConvertToWide = TRUE
)
# Calculate the correlations for each sequence group and each metric.
CalculateRLevel1(
  Dataset = ReshapedData[[1]], StudyID, Groups = c("A", "B", "C", "D"),
  ExperimentName = ShortExperimentNames[1], Metrics, Type = Type[1], Control
)
# A tibble: 8 x 15
# # A tibble: 8 x 15
# Study Exp   Group Metric Id        n ControlFirst    var1   var2
# <chr> <chr> <chr> <chr>  <chr> <int> <lgl>          <dbl>  <dbl>
#   1 S2    E1    A     Compr… S2E1A     6 FALSE        0.0183  0.0163
# 2 S2    E1    B     Compr… S2E1B     6 TRUE         0.0201  0.0326
# 3 S2    E1    C     Compr… S2E1C     6 FALSE        0.00370 0.0155
# 4 S2    E1    D     Compr… S2E1D     6 TRUE         0.0173  0.0201
# 5 S2    E1    A     Modif… S2E1A     6 FALSE        0.0527  0.0383
# 6 S2    E1    B     Modif… S2E1B     6 TRUE         0.0185  0.0482
# 7 S2    E1    C     Modif… S2E1C     6 FALSE        0.00655 0.0244
# 8 S2    E1    D     Modif… S2E1D     6 TRUE         0.0222  0.0266
# # … with 6 more variables: varp <dbl>, ControlVarProp <dbl>,
# #   VarProp <dbl>, vardiff <dbl>, r <dbl>, r.p <dbl>

Description

Function calculates the Hedges small sample size adjustment for standardized mean effect sizes. It calculates the exact value unless the caller sets the parameter exact to FALSE, or the degrees of freedom is too large.

Function calculates the small sample size adjustment for standardized mean effect sizes

Usage

calculateSmallSampleSizeAdjustment(df, exact = TRUE)

calculateSmallSampleSizeAdjustment(df, exact = TRUE)

Arguments

Value

small sample size adjustment value

small sample size adjustment value

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

df <- 2
c <- calculateSmallSampleSizeAdjustment(df)

df <- c(5, 10, 17)
adjexact <- calculateSmallSampleSizeAdjustment(df)
# adjexact=0.8407487 0.9227456 0.9551115
# Hedges and Olkin values 0.8408, 0.9228,0.9551
adjapprox <- calculateSmallSampleSizeAdjustment(df, FALSE)
# adjapprox=0.8421053 0.9230769 0.9552239
df <- 2
a <- calculateSmallSampleSizeAdjustment(df)
# > a
# [1] 0.5641896

df <- c(5, 10, 17)
adjexact <- calculateSmallSampleSizeAdjustment(df)
# > adjexact
# [1] 0.8407487 0.9227456 0.9551115
# Hedges and Olkin values 0.8408, 0.9228,0.9551
adjapprox <- calculateSmallSampleSizeAdjustment(df, FALSE)
# > adjapprox
# [1] 0.8421053 0.9230769 0.9552239
# Another example:
df <- c(10, 25, 50)
calculateSmallSampleSizeAdjustment(df, exact = TRUE)
# [1] 0.9227456 0.9696456 0.9849119
calculateSmallSampleSizeAdjustment(df, exact = FALSE)
# [1] 0.9230769 0.9696970 0.9849246

Description

This is a helper function to stop simulations failing if the metafor function rma fails for example cannot converge properly for a specific dataset.

Usage

CatchError(expr)

Arguments

Value

A message confirming whether the expression has performed successfully

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ES=c(0.2,0.3)
ESvar=c(0.04,0.03)
outcome=reproducer:::CatchError(metafor::rma(ES,ESvar,method='Meth'))
outcome
# [1] 'Failure'

Description

This helper function checks whether a vector variable comprises only zeros and 1's.

Usage

checkIfValidDummyVariable(vector)

Arguments

Value

The logical value: TRUE - if a vector variable passed as the function's parameter represents a valid dummy variable, i.e., comprises only zeros and 1's. FALSE - if a vector variable passed as the function's parameter does not represent a valid dummy variable.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

print(reproducer:::checkIfValidDummyVariable(c(0, 1, 0, 0)))
# [1] TRUE
print(reproducer:::checkIfValidDummyVariable(c(0, 1, 2, 0)))
# [1] FALSE

Description

Data form a set of primary studies on reading methods for software inspections. They were reported and analysed by M. Ciolkowski ('What do we know about perspective-based reading? an approach for quantitative aggregation in software engineering', in Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement, ESEM'09, pp. 133-144, IEEE Computer Society, 2009), corrected and re-analysed by Madeyski and Kitchenham ('How variations in experimental designs impact the construction of comparable effect sizes for meta-analysis' (to be submitted)).

Usage

Ciolkowski09ESEM.MetaAnalysis.PBRvsCBRorAR

Format

A data frame with 21 rows and 7 variables:

Study

Name of empirical study

Ref.

Reference to the paper reporting primary study or experimental run where data were originally reported

Control

Control treatment: Check-Based Reading (CBR) or Ad-hoc Reading (AR)

Within-subjects

Yes - if the primary study used the within-subjects experimental design, No - if the primary study did not use the within-subjects experimental design

Cross-over

Yes - if the primary study used the cross-over experimental design, No - if the primary study did not use the cross-over experimental design

d_ByCiolkowski

d effect size calculated by Ciolkowski

d_ByOriginalAuthors

d effect size as reported by the original authors

Details

If you use this data set please cite: Lech Madeyski and Barbara Kitchenham, 'How variations in experimental designs impact the construction of comparable effect sizes for meta-analysis', 2015.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

Ciolkowski09ESEM.MetaAnalysis.PBRvsCBRorAR

Description

This function provides single-sided and two-sided tests of Cliff's d

Usage

Cliffd.test(x, y, alpha = 0.05, alternative = "two.sided", sigfig = -1)

Arguments

Value

The values of Cliff's d and its standard error, the t-value, its pvalue and the upper and lower confidence interval.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

a=c(1.2,3,2.2,4,2.5,3)
b=c(3,4.2,4,6,7,5.9)
Cliffd.test(a,b,alpha = .05,alternative='two.sided',sigfig = -1)
# A tibble: 1 x 7
#       d sqse.d d.tvalue d.pvalue d.ci.lower d.ci.upper d.sig
#   <dbl>  <dbl>    <dbl>    <dbl>      <dbl>      <dbl> <lgl>
#1 -0.861 0.0202    -42.7 1.20e-12     -0.896     -0.816 TRUE

Cliffd.test(b,a,alpha = .05,alternative='greater',sigfig = -1)
# A tibble: 1 x 7
#      d sqse.d d.tvalue d.pvalue d.ci.lower d.ci.upper d.sig
#  <dbl>  <dbl>    <dbl>    <dbl>      <dbl>      <dbl> <lgl>
#1 0.861 0.0202     42.7 5.99e-13      0.824          1 TRUE

Description

The function constructs various different d-style effect sizes for a set of different experiments given basic statistics from each experiment ( the mean value of the control group Mc, the mean value of the treatment group Mt, the standard deviation of the control group SDc, standard deviation of the the treatment group SDt, the number of observations (participants) in the control group Nc, and the number of observations (participants) in the treatment group Nt). The input variables can be vectors or individual numbers but all input vectors must be of the same length. The function returns Glass's Delta, Cohen's D, point bi-serial r (based on Hedges'g unadjusted), Hedges'g and Hegdes' g adjusted for small sample size.

Usage

constructEffectSizes(Mc, Mt, SDc, SDt, Nc, Nt)

Arguments

Value

data frame composed of five effect sizes (Glass delta, Cohen's d, Hedges' g, r, Hedges' g adjusted)

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

constructEffectSizes(10, 15, 0.3, 0.2, 15, 15)

Mt <- c(0.633, 0.673, 0.423, 0.727, 0.631)
Mc <- c(0.612, 0.526, 0.356, 0.618, 0.534)
SDt <- c(0.198, 0.115, 0.172, 0.088, 0.122)
SDc <- c(0.159, 0.089, 0.111, 0.166, 0.119)
Nt <- c(12, 12, 14, 10, 8)
Nc <- c(12, 12, 14, 10, 8)
EffectSizes <- constructEffectSizes(Mc, Mt, SDc, SDt, Nt, Nc)
EffectSizes
# GlassDelta    Cohend   Hedgesg          r HedgesgAdjusted
# 1  0.1320755 0.1221516 0.1169513 0.05837591       0.1129107
# 2  1.6516854 1.4931812 1.4296121 0.58151846       1.3802200
# 3  0.6036036 0.4803405 0.4628677 0.22547423       0.4493641
# 4  0.6566265 0.8648343 0.8204538 0.37953300       0.7857047
# 5  0.8151261 0.8604924 0.8049169 0.37335594       0.7608781

Description

This function returns the r value for a 2-group (2G) or 4-Group (4G) Crossover experiment for a group of 1 or more experiments for each sequence group and each outcome metric. For sets of 2 or more experiments, the experiments are assumed to be replicates and to report the same sets of Metrics and have the same Control treatment and use the same sequence Group identifiers, but are not necessarily the same Type. We return both the exact r value and the r value based on pooled variances for each sequence group and outcome metric.

Usage

ConstructLevel1ExperimentRData(
  Data,
  StudyID,
  ExperimentNames,
  Groups,
  Metrics,
  Type,
  Control
)

Arguments

Value

R.Data.Table this is a tibble holding information identifying for each metric and sequence group the first time period and second time period variance, the pooled variance, the variance of the difference values and the exact r and pooled r.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

#
ShortExperimentNames <- c("E1", "E2", "E3", "E4")
FullExperimentNames <- c("EUBAS", "R1UCLM", "R2UCLM", "R3UCLM")
Metrics <- c("Comprehension", "Modification")
Groups <- Groups <- c("A", "B", "C", "D")
Type <- c(rep("4G", 4))
StudyID <- "S2"
Control <- "SC"
# Obtain experimental data from each file and put in wide format
dataset2 <- KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM
ReshapedData <- ExtractExperimentData(dataset2,
  ExperimentNames = FullExperimentNames,
  idvar = "ParticipantID", timevar = "Period", ConvertToWide = TRUE
)
# Calculate the correlations for each sequence group and each metric in each experiment
ConstructLevel1ExperimentRData(
  Data = ReshapedData, StudyID = StudyID,
  ExperimentNames = ShortExperimentNames, Groups = Groups, Metrics = Metrics, Type = Type,
  Control = Control
)
# # A tibble: 32 x 15
# Study Exp   Group Metric Id        n ControlFirst    var1   var2    varp
# <chr> <chr> <chr> <chr>  <chr> <int> <lgl>          <dbl>  <dbl>   <dbl>
#   1 S2    E1    A     Compr… S2E1A     6 FALSE        0.0183  0.0163 0.0173
# 2 S2    E1    B     Compr… S2E1B     6 TRUE         0.0201  0.0326 0.0263
# 3 S2    E1    C     Compr… S2E1C     6 FALSE        0.00370 0.0155 0.00962
# 4 S2    E1    D     Compr… S2E1D     6 TRUE         0.0173  0.0201 0.0187
# 5 S2    E1    A     Modif… S2E1A     6 FALSE        0.0527  0.0383 0.0455
# 6 S2    E1    B     Modif… S2E1B     6 TRUE         0.0185  0.0482 0.0333
# 7 S2    E1    C     Modif… S2E1C     6 FALSE        0.00655 0.0244 0.0155
# 8 S2    E1    D     Modif… S2E1D     6 TRUE         0.0222  0.0266 0.0244
# 9 S2    E2    A     Compr… S2E2A     6 FALSE        0.0194  0.0425 0.0309
# 10 S2    E2    B     Compr… S2E2B     6 TRUE         0.0198  0.0192 0.0195
# # … with 22 more rows, and 5 more variables: ControlVarProp <dbl>,
# #   VarProp <dbl>, vardiff <dbl>, r <dbl>, r.p <dbl>

Description

This function analyses one or more crossover experiments where each experiment can be either a two group or four group experiment as specified by Type parameter. The file parameter includes a dataset comprising one or more experiments as defined by the ExperimentNames parameter and each experiment includes values for every output variable defined in the Metrics parameter. After being analysed using the linear modeling lmer function of the lme4 package, the residuals are assessed for normality based on the Anderson-Darling test. Warning 1. This function should only be used with data sets that include one or more individual crossover experiments in the format used in the datasets reported in the reproducer package that were used in the paper B. Kitchenham, L. Madeyski, G. Scanniello, and C. Gravino, “The importance of the correlation between results from individual participants in crossover studies,” IEEE Transactions in SoftwareEngineering, 2021 (Accepted for Publication). [Online]. Available: https://doi.org/10.1109/TSE.2021.30. Warning 2. The lmer function assumes that when experiments include multiple data from the same participant, the correlation between measures for the same participant will be positive. If there is no positive correlation, the function will deliver the warning: boundary (singular) fit: see ?isSingular. This does not mean the analysis has failed. It means that the within participant and between participant variance are set to the same value.

Usage

crossoverResidualAnalysis(file, StudyID, ExperimentNames, Type, Metrics)

Arguments

Value

The results of analysing the residuals for each experiment and metric using the Anderson-Darling test (ADpval) and the number of outliers (NUmOut).

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

File=reproducer::KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello15EMSE
crossoverResidualAnalysis(
  File,StudyID='S1',ExperimentNames=c('USB2'),Type=c('4G'),
  Metrics=c('Correctness','Time','Efficiency'))
#  Study  Exp     Metrics  N ADpval NumOut
#1    S1 USB2 Correctness 24 0.0846      2
#2    S1 USB2        Time 24 0.0448      1
#3    S1 USB2  Efficiency 24  0.365      4

Description

Density curve overlaid on histogram

Usage

densityCurveOnHistogram(df, colName, limLow, limHigh)

Arguments

Value

A figure being a density curve overlaid on histogram

Author(s)

Lech Madeyski

Examples

densityCurveOnHistogram(Madeyski15EISEJ.PropProjects, "STUD", 0, 100)
# densityCurveOnHistogram(data.frame(x<-rnorm(50, mean=50, sd=5)), 'x', 0, 100)

Description

This helper function is called by the function 'crossoverResidualAnalysis' to perform either an AB/BA crossover analysis or a four-group crossover on the data set defined by the parameter DataSet depending on the value of the Type parameter.

Usage

doLM(DataSet, Metric, Type)

Arguments

Value

The data analysis results provided by the lmer function of the lme4 package

Author(s)

Barbara Kitchenham and Lech Madeyski

Description

95 The procedure is based on finding the upper and lower 0.025 bounds for the related t-variable. The t-variable needs to be adjusted for bias by multiplying by c The upper and lower bounds on the t-variable are then used to calculate to upper and lower bounds on the repeated measures effect size (d_RM) by multiplying the upper and lower bound of the t-variable by sqrt((n1+n2)/(2*(n1*n2))). Upper and lower bounds on the equivalent independent groups effect size (d_IG) are found by multiplying the upper and lower bounds on d_RM by sqrt(1-r).

Usage

effectSizeCI(
  expDesign,
  t,
  n1,
  n2,
  r = 0,
  epsilon = 1e-10,
  maxsteps = 1000,
  stepsize = 3
)

Arguments

Value

A list of Confidence Intervals for: t-statistic (t_LB and t_UB), repeated-measures effect size d_RM (d_RM_LB, d_RM_UB), independent groups effect size (d_IG_LB, d_IG_UB)

Author(s)

Lech Madeyski and Barbara Kitchenham

Examples

effectSizeCI(expDesign = "CrossOverRM", t = 14.4, n1 = 15, n2 = 15, r = 0.6401)
effectSizeCI(expDesign = "BeforeAfterRM", t = 14.16536, n1 = 15, n2 = 0, r = 0.6146771)
effectSizeCI(expDesign = "IG", t = -6.344175, n1 = 15, n2 = 15)
effectSizeCI(expDesign = "CrossOverRM", t = 0.5581, n1 = 6, n2 = 6, r = 0.36135)
effectSizeCI(expDesign = "CrossOverRM", r = 0.855, t = 4.33, n1 = 7, n2 = 6)

Description

This function reads datasets from a defined directory in the reproducer package that hold the results of a family crossover experiments in the long format. It converts the data to the wide format if required.

Usage

ExtractExperimentData(
  DataSet,
  ExperimentNames,
  idvar = "ParticipantID",
  timevar = "Period",
  ConvertToWide = TRUE
)

Arguments

Value

A list with an entry for the data for each experiment. If ConvertToWide is TRUE, it returns the data in the wide format otherwise it returns the data as it was read. Within each list item the data is returned as a tibble #importFrom stats # importFrom tibble # importFrom base

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ExperimentNames <- c("EUBAS", "R1UCLM", "R2UCLM", "R3UCLM")
Metrics <- c("Comprehension", "Modification")
Groups <- c("A", "B", "C", "D")
Type <- c(rep("4G", 4))
StudyID <- "S2"
Control <- "SC"
# Obtain experimental data from each file and put in wide format
dataset2 <- KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM
ReshapedData <- ExtractExperimentData(dataset2,
  ExperimentNames = ExperimentNames,
  idvar = "ParticipantID", timevar = "Period", ConvertToWide = TRUE
)
ReshapedData[[1]]

# A tibble: 24 x 15
# ParticipantID ExperimentID.1 SequenceGroup.1 System.1 Treatment.1 Comprehension.1
# <fct>         <fct>          <fct>           <fct>    <fct>                 <dbl>
#   1 1             EUBAS          A               S1       AM                     0.77
# 2 5             EUBAS          A               S1       AM                     0.61
# 3 9             EUBAS          A               S1       AM                     0.61
# 4 13            EUBAS          A               S1       AM                     0.52
# 5 17            EUBAS          A               S1       AM                     0.43
# 6 21            EUBAS          A               S1       AM                     0.77
# 7 2             EUBAS          B               S1       SC                     0.92
# 8 6             EUBAS          B               S1       SC                     0.63
# 9 10            EUBAS          B               S1       SC                     0.51
# 10 14            EUBAS          B               S1       SC                     0.64
# … with 14 more rows, and 9 more variables: Modification.1 <dbl>, CrossOverID.1 <fct>,
#   ExperimentID.2 <fct>, SequenceGroup.2 <fct>, System.2 <fct>, Treatment.2 <fct>,
#   Comprehension.2 <dbl>, Modification.2 <dbl>, CrossOverID.2 <fct>

Description

This function constructs a table identifying the number of participants in each sequence group for a set of experiments each of which used a crossover design.

Usage

ExtractGroupSizeData(
  ExpDataWide,
  StudyID,
  ShortExperimentNames,
  Type,
  Groups = c("A", "B", "C", "D")
)

Arguments

Value

A tibble containing the number of participants in each sequence group in each experiment.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ExperimentNames <- c("EUBAS", "R1UCLM", "R2UCLM", "R3UCLM")
ShortExperimentNames <- c("E1", "E2", "E3", "E4")
Metrics <- c("Comprehension", "Modification")
Type <- c("4G", "4G", "4G", "4G")
Groups <- c("A", "B", "C", "D")
StudyID <- "S2"
Control <- "SC"
# Obtain experimental data from a file and put in wide format
dataset2 <- KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM
ReshapedData <- ExtractExperimentData(dataset2,
  ExperimentNames = ExperimentNames,
  idvar = "ParticipantID", timevar = "Period", ConvertToWide = TRUE
)
ExtractGroupSizeData(ReshapedData, StudyID, ShortExperimentNames, Type, Groups = Groups)
# A tibble: 16 x 4
#  Study Exp   Group     n
#  <chr> <chr> <chr> <int>
# 1 S2    Exp1  A         6
# 2 S2    Exp1  B         6
# 3 S2    Exp1  C         6
# 4 S2    Exp1  D         6
# 5 S2    Exp2  A         6
# 6 S2    Exp2  B         6
# 7 S2    Exp2  C         5
# 8 S2    Exp2  D         5
# 9 S2    Exp3  A         5
# 10 S2    Exp3  B         5
# 11 S2    Exp3  C         6
# 12 S2    Exp3  D         6
# 13 S2    Exp4  A         5
# 14 S2    Exp4  B         5
# 15 S2    Exp4  C         4
# 16 S2    Exp4  D         4

Description

This function extracts summary statistics from meta-analysis results obtained from the rma function of the metafor R package. If required the function transform back to standardized mean difference (effect size type 'd' i.e. Hg) or point biserial correlations (effect size type 'r'). Warning: the ‘ExtractMAStatistics' function works with 'metafor' version 2.0-0, but changes to metafor’s method of providing access to its individual results may introduce errors into the function.

This function extracts summary statistics from meta-analysis results obtained from the rma function of the metafor R package. If required the function transform back to standardized mean difference (effect size type 'd' i.e. Hg) or point biserial correlations (effect size type 'r'). Warning: the ‘ExtractMAStatistics' function works with 'metafor' version 2.0-0, but changes to metafor’s method of providing access to its individual results may introduce errors into the function.

Usage

ExtractMAStatistics(
  maresults,
  Nc,
  Nt,
  Transform = TRUE,
  type = "d",
  sig = 4,
  returnse = FALSE
)

ExtractMAStatistics(
  maresults,
  Nc,
  Nt,
  Transform = TRUE,
  type = "d",
  sig = 4,
  returnse = FALSE
)

Arguments

Value

data frame incl. summary statistics from meta-analysis results: overall mean value for the effect sizes, the p-value of the mean, the upper and lower confidence interval bounds (UB and LB), QE which is the heterogeneity test statistic and QEp which the the p-value of the heterogeneity statistic

data frame incl. summary statistics from meta-analysis results: overall mean value for the effect sizes, the p-value of the mean, the upper and lower confidence interval bounds (UB and LB), QE which is the heterogeneity test statistic and QEp which the the p-value of the heterogeneity statistic

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ExpData <- reproducer::KitchenhamMadeyskiBrereton.ExpData
# Extract the experiment basic statics
S1data <- subset(ExpData, ExpData == "S1")
# Use the descriptive data to construct effect size
S1EffectSizes <- reproducer::PrepareForMetaAnalysisGtoR(
  S1data$Mc, S1data$Mt, S1data$SDc, S1data$SDt, S1data$Nc, S1data$Nt
)
# Do a random effect meta-analysis of the transformed r_pbs effect size
S1MA <- metafor::rma(S1EffectSizes$zr, S1EffectSizes$vi)
# Extract summary statistics from meta-analysis results and transform back to Hg scale
S1MAStats <- reproducer::ExtractMAStatistics(S1MA, sum(S1data$Nc), sum(S1data$Nt), TRUE, "d", 4)
#    mean   pvalue    UB     LB QE  QEp
# 1 0.6658 0.002069 1.122 0.2384  4 0.41
ExpData <- reproducer::KitchenhamMadeyskiBrereton.ExpData
# Extract the experiment basic statics
S1data <- subset(ExpData, ExpData == "S1")
# Use the descriptive data to construct effect size
S1EffectSizes <- reproducer::PrepareForMetaAnalysisGtoR(
  S1data$Mc, S1data$Mt, S1data$SDc, S1data$SDt, S1data$Nc, S1data$Nt
)
# Do a random effect meta-analysis of the transformed r_pbs effect size
S1MA <- metafor::rma(S1EffectSizes$zr, S1EffectSizes$vi)
# Extract summary statistics from meta-analysis results and transform back to Hg scale
ExtractMAStatistics(S1MA, sum(S1data$Nc), sum(S1data$Nt), TRUE, "d", 4)
#     mean   pvalue    UB     LB QE  QEp
# 1 0.6658 0.002069 1.122 0.2384  4 0.41
ExtractMAStatistics(S1MA, sum(S1data$Nc), sum(S1data$Nt), FALSE, "d", 4)
# A tibble: 1 x 6
#   mean  pvalue    UB    LB    QE   QEp
#  <dbl>   <dbl> <dbl> <dbl> <dbl> <dbl>
# 1 0.327 0.00207 0.535 0.119     4  0.41

Description

This function extracts data obtained from the lme4 package lmer function. It assumes a simple randomized experiment with each element having one or more repeated measures. It outputs the mean together with its standard error and confidence interval bounds.

Usage

ExtractSummaryStatisticsRandomizedExp(lmeRA, N, alpha = 0.05)

Arguments

Value

REA.Summary A dataframe holding the number of observations N, the overall mean value as its standard error reported as by the lmer function, and its confidence interval bounds.

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

ShortExperimentNames <- c("E1", "E2", "E3", "E4")
FullExperimentNames <- c("EUBAS", "R1UCLM", "R2UCLM", "R3UCLM")
Metrics <- c("Comprehension", "Modification")
Groups <- c("A", "B", "C", "D")
Type <- c(rep("4G", 4))
StudyID <- "S2"
Control <- "SC"
ReshapedData <- ExtractExperimentData(
  KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM,
  ExperimentNames = FullExperimentNames, idvar = "ParticipantID", timevar = "Period",
  ConvertToWide = TRUE
)
NewTable <- ConstructLevel1ExperimentRData(
  ReshapedData, StudyID, ShortExperimentNames, Groups,
  Metrics, Type, Control
)
resRe <- lme4::lmer(r ~ (1 | Id), data = NewTable)
summary(resRe)
# Linear mixed model fit by REML ['lmerMod']
# Formula: r ~ (1 | Id)
# REML criterion at convergence: 47.8
# Scaled residuals:
#    Min      1Q  Median      3Q     Max
# -1.4382 -0.9691  0.2190  0.8649  1.4761
#
# Random effects:
#  Groups   Name        Variance Std.Dev.
#   Id       (Intercept) 0.03978  0.1994
#   Residual             0.20974  0.4580
#  Number of obs: 32, groups:  Id, 16
#
#  Fixed effects:
#             Estimate Std. Error t value
#  (Intercept)  0.06175    0.09508   0.649
#  N=length(NewTable$r)
ExtractSummaryStatisticsRandomizedExp(lmeRA = resRe, N = 32, alpha = 0.05)
#      N    Mean      SE LowerBound UpperBound
#   1 32 0.06175 0.09508    -0.1319     0.2554

Description

Formatting function to set decimal precision in labels

Usage

fmt()

Author(s)

Lech Madeyski

Description

Function to calculate both effect sizes (dIG, dRM), i.e., independent groups and repeated measures standardized effect sizes and variances, for AB/BA crossover design studies. Function is used in a paper 'Effect Sizes and their Variance for AB/BA Crossover Design Studies' by Lech Madeyski and Barbara Kitchenham.

Usage

getEffectSizesABBA(simulationData)

Arguments

Value

data frame incl. calculated effect sizes and variances: # dIG - independent groups standardized effect size # var.dIG - variance of independent groups standardized effect size # dRM - repeated measures (within-subjects) standardized effect size # var.dRM - variance of repeated measures (within-subjects) standardized effect size # dIG.Fromt - independent groups standardized effect size calculated from t: dIG.Fromt=t*sqrt(1-r)*sqrt((N1+N2)/(2*N1*N2)) # var.dIG.Fromt - variance of independent groups standardized effect size calculated from t: var.dIG.Fromt=var.t*(1-r)*((N1+N2)/(2*N1*N2)) # dRM.Fromt - dRM calculated from t: dRM.Fromt=t*sqrt((N1+N2)/(2*N1*N2)) # var.dRM.Fromt - var.dRM calculated from t: var.dRM.Fromt = var.t*((N1+N2)/(2*N1*N2)) # var.dRM.Fromt2 - var.dRM calculated from t or rather dRM.Fromt: var.dRM.Fromt2=(df/(df-2))*((N1+N2)/(2*N1*N2)+dRM.Fromt^2)- dRM.Fromt^2/c^2 # var.dRM.Approx - var.dRM calculated on a basis of Johnson and Welch (1940) report an approximate formulate for the variance of a t variable: var.dRM.Approx=((N1+N2)/(2*N1*N2)) + (dRM^2)/(2*(N1+N2-2)) #see paper and Equation 49 # var.dIG.Approx - var.dIG calculated on a basis of Johnson and Welch (1940) report an approximate formulate for the variance of a t variable: var.dIG.Approx=(((N1+N2)*(1-r))/(2*N1*N2)) + (dIG^2)/(2*(N1+N2-2)) #see paper and Equation 50 # unstandardizedES - estimated unstandardized technique effect size # periodES - estimated period effect # var.sig - sum of within-subjects variance and between-subjects variance # var.within - within-subjects variance # var.between - between-subjects variance # t - t-value # var.t - variance of t-variable # gRM - Hedges and Olkin (1985) unbiased estimator of the repeated measures effect size gRM=dRM*c # var.gRM - variance of gRM calculated as follows: var.gRM=(df/(df-2))*(((N1+N2)/(2*N1*N2))*c^2+gRM^2)- gRM^2/c^2 #Equation 56 # var.gRM2 - variance of gRM calculated as follows: var.gRM2=var.dRM*c^2 # gIG - Hedges and Olkin (1985) unbiased estimator of the independent groups effect size gIG=dIG*c # var.gIG - variance of gIG calculated as follows: var.gIG=(df/(df-2))*(((N1+N2)/(2*N1*N2))*c^2+gIG^2)- gIG^2/c^2 #Equation 57 # var.gIG2 - variance of gRM calculated as follows: var.gIG2=var.dIG*c^2 # r - the correlation between the values observed for the same subject

Author(s)

Lech Madeyski and Barbara Kitchenham

Examples

simulationData <- getSimulationData(25, 18.75, 50, 10, 5, 500) # generate simulated data set
es <- getEffectSizesABBA(simulationData) # return effect sizes and variances
# OR
simulationData <- getSimulationData(25, 18.75, 50, 10, 5, 15)
es <- getEffectSizesABBA(simulationData) # return effect sizes and variances

Description

Function to calculate both effect sizes (dIG.ipe, dRM.ipe), i.e., independent groups and repeated measures standardized effect sizes and variances, for AB/BA crossover design studies ignoring period effect (thus wrong). Function was removed in the revision of the paper 'Effect Sizes and their Variance for AB/BA Crossover Design Studies' by Lech Madeyski and Barbara Kitchenham.

Usage

getEffectSizesABBAIgnoringPeriodEffect(simulationData)

Arguments

Value

data frame incl. calculated effect sizes and variances: # dIG.ipe - independent groups standardized effect size # var.dIG.ipe - variance of independent groups standardized effect size # dRM.ipe - repeated measures (within-subjects) standardized effect size # var.dRM.ipe - variance of repeated measures (within-subjects) standardized effect size # dIG.Fromt.ipe - independent groups standardized effect size calculated from t: dIG.Fromt=t*sqrt(1-r)*sqrt((N1+N2)/(2*N1*N2)) # var.dIG.Fromt.ipe - variance of independent groups standardized effect size calculated from t: var.dIG.Fromt=var.t*(1-r)*((N1+N2)/(2*N1*N2)) # dRM.Fromt.ipe - dRM calculated from t: dRM.Fromt=t*sqrt((N1+N2)/(2*N1*N2)) # var.dRM.Fromt.ipe - var.dRM calculated from t: var.dRM.Fromt = var.t*((N1+N2)/(2*N1*N2)) # var.dRM.Fromt2.ipe - var.dRM calculated from t or rather dRM.Fromt: var.dRM.Fromt2=(df/(df-2))*((N1+N2)/(2*N1*N2)+dRM.Fromt^2)- dRM.Fromt^2/c^2 # unstandardizedES.ipe - estimated unstandardized technique effect size # var.sig.ipe - sum of within-subjects variance and between-subjects variance # var.within.ipe - within-subjects variance # var.between.ipe - between-subjects variance # t.ipe - t-value # var.t.ipe - variance of t-variable

Author(s)

Lech Madeyski and Barbara Kitchenham

Examples

simulationData <- getSimulationData(25, 18.75, 50, 10, 5, 500) # generate simulated data set
es.ipe <- getEffectSizesABBAIgnoringPeriodEffect(simulationData) # return effect sizes and variances

Description

Function to generate the simulated data set used in a paper 'Effect Sizes and their Variance for AB/BA Crossover Design Studies' by Lech Madeyski and Barbara Kitchenham

Usage

getSimulationData(
  var,
  covar,
  meanA1,
  treatmentDiff,
  periodEffect,
  numOfSamples
)

Arguments

Details

——————————————————————————————————- Functions related to a paper 'Effect sizes and their variance for AB/BA crossover design studies' by Lech Madeyski and Barbara Kitchenham ——————————————————————————————————-

Value

Data frame: 'data.frame': 4*numOfSamples obs. of 5 variables: $ pid : int 1 2 3 4 5 6 7 8 9 10 ... $ technique: Factor w/ 2 levels 'T1','T2': ... $ period : Factor w/ 2 levels 'P1','P2': ... $ sequence : Factor w/ 2 levels 'S1','S2': ... $ result : num ...

Author(s)

Lech Madeyski and Barbara Kitchenham

Examples

# generate the simulated data set from the paper
data <- getSimulationData(25, 18.75, 50, 10, 5, 500)
data <- getSimulationData(25, 18.75, 50, 10, 5, 15)

Description

Function provides the theoretical value of the t-statistic, variance of t, and variance of the effect sizes based on the parameters built into crossover model data simulated by the getSimilationData() function. Function is used in a paper 'Effect Sizes and their Variance for AB/BA Crossover Design Studies' by Lech Madeyski and Barbara Kitchenham.

Usage

getTheoreticalEffectSizeVariancesABBA(
  theoreticalvarW,
  theoreticalTechniqueEffect,
  theoreticalrho,
  N1,
  N2
)

Arguments

Value

data frame incl. calculated: theoreticalt - the theoretical value of the t-statistic theoreticalvart - variance of t theoreticalvardIG - variance of the effect size dIG based on the parameters built into crossover model data simulated by the getSimilationData function theoreticalvardRM - variance of the effect size dRM based on the parameters built into crossover model data simulated by the getSimilationData function

Author(s)

Lech Madeyski and Barbara Kitchenham

Examples

# Generates data used in Table 15 of the paper
theoreticalEffectSizeVariances <- getTheoreticalEffectSizeVariancesABBA(6.25, -10, 0.75, 15, 15)

Description

Data illustrate correlations between results from individual participants in a family of five cross-over experiments conducted by Abrahao et al: [1] S. Abrahao, C. Gravino, E. Insfran Pelozo, G. Scanniello, and G. Tortora, 'Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: Results from a family of five experiments,' IEEE Transactions on Software Engineering, vol. 39, no. 3, pp. 327–342, March 2013 The five experiments assess whether the comprehensibility of function requirements improve when software models include UML sequence diagrams. If you use this data set please cite: [1] S. Abrahao, C. Gravino, E. Insfran Pelozo, G. Scanniello, and G. Tortora, 'Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: Results from a family of five experiments,' IEEE Transactions on Software Engineering, vol. 39, no. 3, pp. 327–342, March 2013 [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Abrahao13TSE

Format

A data frame with 224 rows and 8 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each of the five experiments in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A (DM-NODM,ECP-EPlat or MShop-Theatre ), B (NODM-DM,ECP-EPlat or MShop-Theatre ), C(DM-NODM,EPlat-ECP or Theatre-MShop), D(NODM-DM,EPlat-ECP or Theatre-MShop)

System

<fct>|Software systems used in the experiment: ECP an e-commerce platform from which CDs and books can be bought, EPlat a system for the management of courses, lectures and students of a university, M-Shop a system for managing sales at a music shop, Theatre a system for managing bookings for a theatre.

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment: A Dynamic Model (DM) vs No Dynamic Model (NODM)

Comprehension

<dbl>|Dependent variable: The comprehension level the software engineer achieved based on the F-measure

CrossOverID

<fct>|CrossOver category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For a 2 group crossover, the category is set to CO1 only

Ability

<fct>|Ability: An assessment of the ability of participants: Low, High, NA (not available)

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Abrahao13TSE

Description

Data illustrate correlations between results from individual participants in a family of two cross-over experiments conducted by Gravino et al.: [1] C. Gravino, G. Scanniello, and G. Tortora, 'Source-code comprehension tasks supported by UML design models: Results from a controlled experiment and a differentiated replication,' Journal of Visual Languages and Computing, vol. 28, pp. 23–38, 2015. The experiments assess whether the comprehension of object oriented source-code increases used with UML class and sequence diagrams produced in the software design phase. If you use this data set please cite: [1] C. Gravino, G. Scanniello, and G. Tortora, 'Source-code comprehension tasks supported by UML design models: Results from a controlled experiment and a differentiated replication,' Journal of Visual Languages and Computing, vol. 28, pp. 23–38, 2015. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Gravino15JVLC

Format

A data frame with 64 rows and 9 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each of the three experiments in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A , B , C, D

System

<fct>|Software systems used in the experiment: Music shop, a system for handling the sales of a music shop. Theater ticket, a system for managing theatre reservations.

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment: Mo, design models were available, NOMo design models were not available

Comprehension

<dbl>|Dependent variable: The level of comprehension achieved by the software engineer.

Time

<dbl>|Dependent variable: The time [min] taken to complete the comprehension task.

CrossOverID

<fct>|CrossOver category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For 2 group crossover, the category is set to CO1 only

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Gravino15JVLC

Description

Data illustrate correlations between results from individual participants in cross-over experiment P2007 (Smell and Library) conducted by Madeyski, see: [1] Lech Madeyski, Test-Driven Development: An Empirical Evaluation of Agile Practice. (Heidelberg, London, New York): Springer, 2010. Foreword by Prof. Claes Wohlin. If you use this data set please cite: [1] Lech Madeyski, Test-Driven Development: An Empirical Evaluation of Agile Practice. (Heidelberg, London, New York): Springer, 2010. Foreword by Prof. Claes Wohlin. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Madeyski10

Format

'KitchenhamEtAl.CorrelationsAmongParticipants.Madeyski10': a data frame with 45 rows and 10 variables:

ExperimentID

<fct>| ExperimentID: This experiment is the only cross-over experiment in the family of TDD and Pair-Programming experiments conducted by Madeyski, so all values in this column are set to 'P2007'.

ParticipantID

<fct> | Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct> | Experimental Sequence Group: A (TLSP-TFSP), B (TFSP-TLSP)

System

<fct> | Software system to develop: Smell (a tool for identifying bad code smells in Java source code through the use of a set of software metrics) or Library (a library application)

Period

<fct> | Time period of the cross-over experiment: 1 or 2

Treatment

<fct> | Experimental Treatment: Test-First Solo Programming (TFSP) vs Test-Last Solo Programming (TLSP)

PATP

<dbl> | Dependent variable: Percentage of Acceptance Tests Passed

NATPPH

<dbl> | Dependent variable: Number of Acceptance Tests Passed Per Hour

CBO

<dbl> | Dependent variable: Mean value of Coupling Between Objects (CBO), see CK set of metrics

WMC

<dbl> | Dependent variable: Mean value of Weighted Number of Methods in Class (WMC), see CK set of metrics

RFC

<dbl> | Dependent variable: Mean value of Response For a Class (RFC), see CK set of metrics

CrossOverID

<fct> | Cross-Over Code. This experiment is a simple two-group cross-over experiment with one cross-over code, so all values in this column are set to 'CO1'. However, four-group experiments require a code to identify the linked sequence groups (although that can be deduced from the system used in the first time period). A crossover code is also essential for non-parametric analysis.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Madeyski10

Description

Data illustrate correlations between results from individual participants in a family of two cross-over experiments conducted by Reggio et al: [1] G. Reggio, F. Ricca, G. Scanniello, F. D. Cerbo, and G. Dodero,'On the comprehension of workflows modeled with a precise style: results from a family of controlled experiments'. Software and Systems Modeling, vol. 14, pp. 1481–1504, 2015. The experiments assess whether the level of formality/precision in workflow model influences comprehension. If you use this data set please cite: [1] G. Reggio, F. Ricca, G. Scanniello, F. D. Cerbo, and G. Dodero, 'On the comprehension of workflows modeled with a precise style: results from a family of controlled experiments'. Software and Systems Modeling, vol. 14, pp. 1481–1504, 2015. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The Importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Reggio15SSM

Format

A data frame with 78 rows and 9 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each of the three experiments in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A , B , C, D

System

<fct>|Software systems used in the experiment: PO, a system to process orders for an online shop. DM, a system to manage an online document review process.

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment:

Comprehension

<dbl>|Dependent variable: The comprehension level obtained by each participant.

Time

<dbl>|Dependent variable: The time [min] taken by each participant to complete the comprehension task.

CrossOverID

<fct>|CrossOver category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For a 2 group crossover, the category is set to CO1 only

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Reggio15SSM

Description

Data illustrate correlations between results from individual participants in a family of four cross-over experiments conducted by Ricca et al.: [1] F. Ricca, M. D. Penta, M. Torchiano, P. Tonella, and M. Ceccato 'How developers’ experience and ability influence web application comprehension tasks supported by uml stereotypes: A series of four experiments', IEEE Transactions on Software Engineering, vol. 36, no. 1, pp. 96-118, 2010. Although we present the full data set, only the first two experiments were used in the correlation study, because many of the observations in the final two studies were unpaired. The experiments assess whether participants performance comprehension tasks better when using source code complemented by standard UML diagrams (UML) or by diagrams stereotyped using the Conallen notation (Conallen). If you use this data set please cite: [1] F. Ricca, M. D. Penta, M. Torchiano, P. Tonella, and M. Ceccato 'How developers’ experience and ability influence web application comprehension tasks supported by uml stereotypes: A series of four experiments', IEEE Transactions on Software Engineering, vol. 36, no. 1, pp. 96—118, 2010. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The Importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Ricca10TSE

Format

A data frame with 176 rows and 10 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each of the four experiments in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A , B , C, D

System

<fct>|Software systems used in the experiment: Two Java-based Web applications, Claros and WfMS

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment: UML or Conallon

FMeasure

<dbl>|Dependent variable: The comprehension level achieved by the participant.

Time

<dbl>|Dependent variable: The time [min] to complete the experimental task

CrossOverID

<fct>|CrossOver category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For 2 group crossover, the category is set to CO1 only

Ability

<fct>| h: High l: Low, NA: Not available

Experience

<fct>| G: Master students, U: undergraduates, P: researchers

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Ricca10TSE

Description

Data illustrate correlations between results from individual participants in a family of three of four cross-over experiments conducted by Ricca et al: [1] F. Ricca, G. Scanniello, M. Torchiano, G. Reggio, and E. Astesiano, 'Assessing the effect of screen mockups on the comprehension of functional requirements,' ACM Transactions on Software Engineering and Methodology, vol. 24, no. 1, pp. 1:1–1:38, Oct. 2014. The goal of the study was to assess whether stakeholders benefit from the presence of screen mock-ups in the comprehension of functional requirements represented with use cases. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Ricca14TOSEM

Format

A data frame with 176 rows and 10 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each of the three experiments in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A , B , C, D

System

<fct>|Software systems used in the experiment: AMICO, a system for management of condominiums. EasyCoin, a system for cataloguing collections of coins.

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment: Screen mockup available (S) vs Text only (T)

Time

<dbl>|Dependent variable: The time [min] taken to perform the software engineering task.

Comprehension

<dbl>|Dependent variable: The comprehension level the software engineers.

Efficiency

<dbl>|Dependent variable: The ratio of comprehension to time.

CrossOverID

<fct>|CrossOver category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For a 2 group crossover, the category is set to CO1 only.

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Ricca14TOSEM

Description

Data illustrate correlations between results from individual participants in a cross-over experiment conducted by Romano et al.: [1] S. Romano, G. Scanniello, D. Fucci, N. Juristo, and B. Turhan, 'The effect of noise on software engineers’ performance', in Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ESEM'18, 2018. The experiments assess whether noise has an impact on the performance of software engineers. If you use this data set please cite: [1] S. Romano, G. Scanniello, D. Fucci, N. Juristo, and B. Turhan, 'The effect of noise on software engineers’ performance', in Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ESEM'18, 2018. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The Importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020). The experiment had two parts but Kitchenham et al. only use the data from the first part of the experiment.

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Romano18ESEM

Format

A data frame with 194 and 10 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each part of the experiment. Exp.1 identifies data from the first part of the experiment, Exp.2 identifies data from the second part of the experiment.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for both parts of the experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A , B

System

<fct>|Software systems used in the experiment: For the first part of the experiment, M-Shop (a system for managing a music shop) and Theater (a system for managing theatre reservations). For the second part of the experiment: AveCalc (a system that manages as electronic register and LaTazza (a system for a drinks vending machine)

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment: NOISE, participants were asked to perform a comprehension task in a noisy environment. NORMAL, participants were asked to perform a comprehension task under normal working conditions.

Fc

<dbl>|Dependent variable: the balanced F-measure which represents the trade-off between precision and recall, measured in the first part of the experiment.

Avg

<dbl>|Dependent variable: The average number of fully correct answers, measured in the first part of the experiment.

Ff

<dbl>|Dependent variable: Effectiveness of fault correction. Measured in the second part of the experiment.

CrossOverID

<fct>|CrossOver category: For 4 group crossovers, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For 2 group crossover, the category is set to CO1 only.

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Romano18ESEM

Description

Data illustrate correlations between results from individual participants in a family of two cross-over experiments conducted by Scanniello et al: [1] G. Scanniello, M. Staron, H. Burden, and R. Heldal, 'On the effect of using SysML requirement diagrams to comprehend requirements: results from two controlled experiments,' in Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE. ACM, 2014. The two experiments investigate whether requirements specified as SysML requirement diagrams improve the comprehensibility of requirements. If you use this data set please cite: [1] G. Scanniello, M. Staron, H. Burden, and R. Heldal, 'On the effect of using SysML requirement diagrams to comprehend requirements: results from two controlled experiments', in Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE. ACM, 2014. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14EASE

Format

A data frame with 174 rows and 9 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each experiment in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A (RD-NORD,Automobile-ESS), B (NORD-RD,ESS-Automobile), C(NORD-RD,Automobile-ESS), D(RD-NORD,ESS-Automobile).

System

<fct>|Software systems used in the experiment: Automobile: A system for controlling car behavior with use cases about entering the car, anti-lock breaking or operating the climate control of a car. ESS (Enhanced Security System) a system designed to detect potential home intruders.

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment: RD availability of a SysML requirements diagram vs No requirements diagram (NORD)

Time

<dbl>|Dependent variable: The time [min] required for the comprehension task.

Comprehension

<dbl>|Dependent variable: The comprehension level the software engineer achieved.

CrossOverID

<fct>|CrossOver category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For 2 group crossover, the category is set to CO1 only

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14EASE

Description

Data illustrate correlations between results from individual participants in a cross-over experiment conducted by Scanniello and Erra: [1] G. Scanniello and U. Erra, 'Distributed modeling of use case diagrams with a method based on think-pair-square: Results from two controlled experiments', Journal of Visual Languages and Computing, vol. 25, no. 4, pp. 494–517, 2014. The experiment investigated whether a new method based on think-pair-square and its implementation in a integrated communication/modeling environment (TPS approach) is as effective as traditional face-to-face (F2F approach) for requirements elicitation. The experiment was performed in two stages using different software systems. If you use this data set please cite: [1] G. Scanniello and U. Erra, 'Distributed modeling of use case diagrams with a method based on think-pair-square: Results from two controlled experiments,” Journal of Visual Languages and Computing, vol. 25, no. 4, pp. 494–517, 2014. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The Importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14JVLC

Format

A data frame with 36 rows and 12 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each experiment in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each team of four participants, unique for the specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A , B

System

<fct>|Software systems used in the experiment: Library (a software system to manage books and users of a library) and FilmCollection (a software system for the selling and the rental of films in a shop) in ExperimentStage1 and Rent (a car rental software to manage cars, customers, and reservations) and ECP (an E-Commerce Platform to order CDs and books via the Internet from an on line catalogue), in ExperimentStage2.

Treatment

<fct>|Experimental Treatment: TPS vs F2F.

Period

<fct>|Time period of the cross-over experiment: 1 or 2 within each stage of the experiment

Time

<dbl>|Dependent variable: The total time [min] to accomplish the requirement engineering task.

Quality

<dbl>|Dependent variable: The quality of the requirements engineering task.

CrossOverID

<fct>|Crossover category: For a single 2 group crossover experiment, the value is set to CO1 for each experiment stage.

ExperimentPeriod

<fct>|ExperimentPeriod: The time period across both stages of the experiment.

ExperimentStage

<fct>|ExperimentStage: 1 first stage, 2 second stage.

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14JVLC

Description

Data illustrate correlations between results from individual participants in a family of four cross-over experiments conducted by Scanniello et al: [1] G. Scanniello, C. Gravino, M. Genero, J.A. Cruz-Lemus, and G. Tortora, 'On the Impact of UML Analysis Models on Source-Code Comprehensibility and Modifiability', ACM Transactions on Software Engineering and Methodlogy, vol. 23, no. 2, pp. 13:1-13:26, 2014 The family of experiments investigated whether the availability of analysis models in addition to the source code made the code easier to understand and modify. If you use this data set please cite: [1] G. G. Scanniello, C. Gravino, M. Genero, J.A. Cruz-Lemus, and G. Tortora, 'On the Impact of UML Analysis Models on Source-Code Comprehensibility and Modifiability', ACM Transactions on Software Engineering and Methodology, vol. 23, no. 2, pp. 13:1-13:26, 2014 [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM

Format

'KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM': a data frame with 172 rows and 9 variables:

ExperimentID

<fct> | ExperimentID: A unique identifier for each experiment in the data set.

ParticipantID

<fct> | Participant ID: An identifier for each participant, unique for a specific experiment.

Treatment

<fct> | Experimental Treatment: AM an Analysis Model with source code (AM) vs Source Code only (SC)

SequenceGroup

<fct> | Experimental Sequence Group: A (AM-SC,S1-S2), B (SC-AM,S1-S2), C(AM-SC,S2-S1), D(SC-AM,S2-S1)

System

<fct> | Software systems used in the experiment: S1 A system to sell and manage CDs/DVDs in a music shop, S2 A system to book and by theater tickets.

Comprehension

<dbl> | Dependent variable: The comprehension level the software engineer achieved based on the F-measure

Modification

<dbl> | Dependent variable: The modifiability level the software engineer achieved based on the F-measure

Period

<fct> | Time period of the cross-over experiment: 1 or 2

CrossOverID

<fct> | CrossOver category: For 4 group the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For 2 group crossover, the category is set to CO1 only

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello14TOSEM

Description

Data illustrate correlations between results from individual participants in cross-over experiment usb2 conducted by Scanniello et al: [1] G. Scanniello, A. Marcus, and D. Pascale, 'Link analysis algorithms for static concept location: an empirical assessment', Empirical Software Engineering, vol. 20, no. 6, pp. 1666–1720, 2015. The goal of the experiment is to assess whether a new technique (implemented as an Eclipse plug-in) for static concept location (proposed by the authors) supports users in identifying the places in the code where changes are to be made.

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello15EMSE

Format

'KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello15EMSE': a data frame with 48 rows and 10 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each experiment in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A (CL-NOCL,Jedit-Atunes), B (NOCL-CL,Atunes-Jedit), C(NOCL-CL,Jedit-Atunes), D(CL-NOCL,Atunes-Jedit)

System

<fct>|Software systems used in the experiment: Jedit and Atunes

Treatment

<fct>|Experimental Treatment: Use of Concept Location plug-in (CL) vs no Concept Location plug-in (NOCL)

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Correctness

<int>|Dependent variable: 0, 1, 2, 3, 4. The participants are asked to indicate a single change method for each of 4 bug reports. A change method is correctly identified if that method is in the change set of the bug report.

Time

<dbl>|Dependent variable: The total time [min] to accomplish concept location tasks, i.e.,to identify (four) bugs given their reports

Efficiency

<dbl>|Dependent variable: The participants’ efficiency in the execution of concept location tasks. It is computed dividing correctness by time.

CrossOverID

<fct>|Crossover category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2.

Details

If you use this data set please cite: [1] G. Scanniello, A. Marcus, and D. Pascale, 'Link analysis algorithms for static concept location: an empirical assessment', Empirical Software Engineering, vol. 20, no. 6, pp. 1666–1720, 2015. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello15EMSE

Description

Data illustrate correlations between results from individual participants in a family of four cross-over experiments conducted by Scanniello et al.: [1] G. Scanniello, M. Risi, P. Tramontana, and S. Romano, 'Fixing faults in C and Java source code: Abbreviated vs. full-word identifier names', ACM Transactions on Software Engineering Methodology, vol. 26, no. 2, 2017. The experiments assess whether whether the use of abbreviated identifier names (ABBR), impacts the effectiveness of fault fixing in C and Java source code in comparison with full-word identifier names (FULL). If you use this data set please cite: [1] G. Scanniello, M. Risi, P. Tramontana, and S. Romano, “Fixing faults in C and Java source code: Abbreviated vs. full-word identifier names', ACM Transactions on Software Engineering Methodology, vol. 26, no. 2, 2017. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'On the Importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello17TOSEM

Format

A data frame with 200 rows and 17 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each of the experiments in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A , B , C, D

System

<fct>|Software systems used in the experiments: The Unibas experiment used Agenda (a system for tracking personal contacts) and Gas-Station (a system for managing a petrol station). The UniNa experiment used Financial (a system which is a command line option price calculator) and Hotel-Reservation. The POLINA and PROF experiments used AveCalc (a system that manages as electronic register and LaTazza (a system for a drinks vending machine).

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment:ABBR, abbreviated names. FULL, full names

Time

<dbl>|Dependent variable: The time each participant spent performing the SE task.

FMeasure

<dbl>|Dependent variable: The effectiveness of the participants taking into account correctness and completeness of the fault fixing tasks

Efficiency

<dbl>|Dependent variable: The ratio of effectiveness to time.

CrossOverID

<fct>|CrossOver category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For 2 group crossover, the category is set to CO1 only.

Language

<fct>|Java or C. The language was the same for all participants in a specific experiment. POLINA and PROF used Java, UNIBAS and UNINA used C.

Ident

<dbl>|Dependent variable: The number of faults identified.

Fixed

<dbl>|Dependent variable: The number of faults identified.

WrongIdent

<dbl>|Dependent variable: The number of faults incorrectly identified

WronglyFixed

<dbl>|Dependent variable: The number of faults incorrectly fixed.

precision

<dbl>|Dependent variable: The ratio of number of faults correctly fixed to the number of faults correctly identified.

recall

<dbl>|Dependent variable: The ratio of number of faults correctly fixed to the total number of fault.

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Scanniello17TOSEM

Description

Data illustrate correlations between results from individual participants in a family of three cross-over experiments conducted by Torchiano et al: [1] M. Torchiano, G. Scanniello, F. Ricca, G. Reggio, and M. Leotta, 'Do UML object diagrams affect design comprehensibility? Results from a family of four controlled experiments.' Journal of Visual Languages and Computing, vol. 41, pp. 10–21, 2017. Although the paper reports four experiment, we only have data from three of those experiments. The experiments assess whether the comprehensibility of UML specifications improve when the software documents include UML object diagrams as well as the standard UML class diagrams. If you use this data set please cite: [1] M. Torchiano, G. Scanniello, F. Ricca, G. Reggio, and M. Leotta, 'Do UML object diagrams affect design comprehensibility? Results from a family of four controlled experiments.' Journal of Visual Languages and Computing, vol. 41, pp. 10–21, 2017. [2] Barbara Kitchenham, Lech Madeyski, Giuseppe Scanniello and Carmine Gravino, 'The importance of the Correlation between Results from Individual Participants in Crossover Experiments' (to be submitted as of 2020).

Usage

KitchenhamEtAl.CorrelationsAmongParticipants.Torchiano17JVLC

Format

A data frame with 214 rows and 8 variables:

ExperimentID

<fct>|ExperimentID: A unique identifier for each of the three experiments in the data set.

ParticipantID

<fct>|Participant ID: An identifier for each participant, unique for a specific experiment.

SequenceGroup

<fct>|Experimental Sequence Group: A , B , C, D

System

<fct>|Software systems used in the experiment: File System manager (FS) for folders, files, links. Roads system (R) handles maps made up of cities connected by means of roads. Train (T) a system to manage timetables, trains, and paths. Catalogue system (C). It collects category of items (e.g., cars) and items (e.g., car models) based on a set of features (e.g., number of doors). In PoliTo2, only FS and T were administered to the participants, while in UniBas1 and UniGe1 all the four experimental objects were used.

Period

<fct>|Time period of the cross-over experiment: 1 or 2

Treatment

<fct>|Experimental Treatment: Object Diagram (OD) vs No Object Diagram (NoOD)

Comprehension

<dbl>|Dependent variable: The comprehension level the software engineers. For PoliTo2 Comprehension was based on answering a set of 4 questions, for UniBas and UniGe comprehension was measured using the F metric.

CrossOverID

<fct>|CrossOver category: For 4 group crossover designs, the crossover category specifies the matching pairs of sequence groups, CO1 and CO2. For 2 group crossover, the category is set to CO1 only

Examples

KitchenhamEtAl.CorrelationsAmongParticipants.Torchiano17JVLC

Description

If you use this data set please cite this R package and the following paper: Lech Madeyski and Barbara Kitchenham, 'Effect Sizes and their Variance for AB/BA Crossover Design Studies', Empirical Software Engineering, vol. 24, no.4, p. 1982-2017, 2018. DOI: 10.1007/s10664-017-9574-5

Usage

KitchenhamMadeyski.SimulatedCrossoverDataSets

Format

A data frame with variables:

actualSampleSize

Sample size

SSFull

Sample Size

CFull

Correlation

ESFull

Effect Size

Accuracy

Accuracy

PropSig

...

WrongTSig

...

Details

This is simulated normally distributed data from 30 subjects, with technique A being 10 units more effective than technique B, and there is a period effect equaling 5 units. Subject 1 to 15 used technique B first while subjects 16 to 30 used technique A first.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyski.SimulatedCrossoverDataSets

Description

This data is used in the paper: Barbara Kitchenham, Lech Madeyski and Pearl Brereton. Meta-analysis for Families of Experiments: A Systematic Review and Reproducibility Assessment, Empirical Software Engineering (2019) doi:10.1007/s10664-019-09747-0. This data set reports the meta-analysis results reported by the authors of the primary studies included in the systematic review that reported results on a per document basis which for S7 and S11 was equivalent to reporting the results for each time period.

Usage

KitchenhamMadeyskiBrereton.ABBAMetaAnalysisReportedResults

Format

A text file with variables:

Study

This field includes the study identifier of each of the the 3 primary studies which reported results per document.

Type

This identifies the type of effect size used by the study authors. d or g refer to d_IG and g_IG, P is the aggregated p values, if the repeated measures (RM) estimate was obtained it is appropriately specified.

Source

Always set to Rep. This identifies that the data was as reported by the primary study authors.

mean

The overall mean effect size reported by the study authors

pvalue

The one-sided p-value associated with the overall mean reported by the study authors. NA means the authors did not report this statistic.

UB

The upper bound of the confidence interval of the overall mean as reported by the primary study authors. NA means the authors did not report this statistic.

LB

The lower bound of the confidence interval of the overall mean as reported by the primary study authors. NA means the authors did not report this statistic.

QE

The heterogeneity statistic associated with the meta-analysis as reported by the study authors. NA means the authors did not report this statistic.

Qep

The p-value of the heterogeneity statistic associated with the meta-analysis as reported by the study authors. NA means the authors did not report this statistic.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBrereton.ABBAMetaAnalysisReportedResults

Description

This data is used in the paper: Barbara Kitchenham, Lech Madeyski and Pearl Brereton. Meta-analysis for Families of Experiments: A Systematic Review and Reproducibility Assessment, Empirical Software Engineering (2019) doi:10.1007/s10664-019-09747-0. This file holds the individual effect sizes for the first time period (or equivalently the first document), as reported by the 3 primary studies in the systematic review that reported results for each document/time period separately.

Usage

KitchenhamMadeyskiBrereton.ABBAReportedEffectSizes

Format

A text file with variables:

Study

This field includes the study identifier of each of the 3 primary studies which were included in the systematic review. The studies are S3, S7 and S11.

Type

This identifies the type of effect size used by the study authors. d or g refer to dIG and gIG.

Source

Always set to Rep. This identifies that the data was as reported by the primary study authors.

Design

Mixed means different experiments in a particular family used different methods (onlyS3 used mixed methods and 4 experiments used the 4 group crossover and one used an independent groups design). ABBACO is the standard 2-group crossover design.

Exp1

This is the reported standardised effect size for the first time period and the first experiment in the family.

Exp2

This is the reported standardised effect size for the first time period and second experiment in the family.

Exp3

This is the reported standardised effect size for the first time period and the third experiment in the family.

Exp4

This is the reported standardised effect size for the first time period and the fourth experiment in the family. NA means there was no fourth experiment in the family.

Exp5

This is the reported standardised effect size for the first time period and the fifth experiment in the family. NA means there was no fifth experiment in the family.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBrereton.ABBAReportedEffectSizes

Description

This data is used in the paper: Barbara Kitchenham, Lech Madeyski and Pearl Brereton. Meta-analysis for Families of Experiments: A Systematic Review and Reproducibility Assessment, Empirical Software Engineering (2019) doi:10.1007/s10664-019-09747-0. This file holds the descriptive data for each document and each experiment for studies 3, 7 and 11 which include the mean, standard deviation and sample size for the control and treatment techniques. These studies performed ABBA crossover experiments and reported data for each document separately. Note Study 3 also undertook an independent groups study but data from that experiment is held in the ExpData file.

Usage

KitchenhamMadeyskiBrereton.DocData

Format

A text file with variables:

Study

This field includes the study identifier of each of the 3 primary studies which reported their basic statistics on a time period & document basis.

Exp

This identifies the experiment to which the descriptive data belongs.

Doc

This identifies whether the data arose from the document used in the first or second time period. The value 'Doc1' identifies the data as coming from the first document or first time period. The value 'Doc2' identifies the data as coming from the second time period or document. Note for Study 3 we used the analysis of a specific document that was used in all 4 ABBA experiments. For studies 7 and 11, the authors identified which we used in r=each time period and Doc1 refers to data from the first time period.

Mc

The mean value of the observations obtained using the control technique for the identified document.

SDc

The standard deviation of the observations obtained using the control technique for the identified document.

Nc

The number of participants using the control technique in the first time period for the identified document.

Mt

The mean value of the observations obtained using the treatment technique for the identified document.

SDt

The standard deviation of the observations obtained using the treatment technique for the identified document.

Nt

The number of participants using the treatment technique in the first time period for the identified document.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBrereton.DocData

Description

This data is used in the paper: Barbara Kitchenham, Lech Madeyski and Pearl Brereton. Meta-analysis for Families of Experiments: A Systematic Review and Reproducibility Assessment, Empirical Software Engineering (2019) doi:10.1007/s10664-019-09747-0. This file holds the descriptive data for each experiment which include the mean, standard deviation and sample size for the control and treatment techniques. Note in the case of studies 3, 7 and 11, which reported descriptive data for each time period (or equivalently each document) separately, the values for of the descriptive data were obtained by analysing the data reported in the DocData file.

Usage

KitchenhamMadeyskiBrereton.ExpData

Format

A text file with variables:

Study

This field includes the study identifier of each of the 13 primary studies which were included in the systematic review.

Exp

This identifies the experiment to which the descriptive data belongs.

Source

Always set to Rep. This identifies that the data was as reported by the primary study authors.

Mc

The mean value of the observations obtained using the control technique.

SDc

The standard deviation of the observations obtained using the control technique.

Nc

The number of participants using the control technique in the first time period.

Mt

The mean value of the observations obtained using the treatment technique.

SDt

The standard deviation of the observations obtained using the treatment technique.

Nt

The number of participants using the treatment technique in the first time period.

r

The correlation between repeated measures. NA if not reported. Note only study 13 reported this correlation.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBrereton.ExpData

Description

This data is used in the paper: Barbara Kitchenham, Lech Madeyski and Pearl Brereton. Meta-analysis for Families of Experiments: A Systematic Review and Reproducibility Assessment (to be submitted). This data set reports the meta-analysis results reported by the authors of the 13 primary studies included in the systematic review.

Usage

KitchenhamMadeyskiBrereton.MetaAnalysisReportedResults

Format

A text file file with variables:

Study

This field includes the study identifier of each of the 13 primary studies which were included in the systematic review.

Type

This identifies the type of effect size used by the study authors. d or g refer to d_IG and g_IG, P is the aggregated p values, if the repeated measures estimate was obtained it is appropriately specified, r refers to the point bi-serial correlation.

Source

Always set to Rep. This identifies that the data was as reported by the primary study authors.

mean

The overall mean effect size reported by the study authors

pvalue

The one-sided p-value associated with the overall mean reported by the study authors. NA means the authors did not report this statistic.

UB

The upper bound of the confidence interval of the overall mean as reported by the primary study authors. NA means the authors did not report this statistic.

LB

The lower bound of the confidence interval of the overall mean as reported by the primary study authors. NA means the authors did not report this statistic.

QE

The heterogeneity statistic associated with the meta-analysis as reported by the study authors. NA means the authors did not report this statistic.

Qep

The p-value of the heterogeneity statistic associated with the meta-analysis as reported by the study authors. NA means the authors did not report this statistic.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBrereton.MetaAnalysisReportedResults

Description

This data is used in the paper: Barbara Kitchenham, Lech Madeyski and Pearl Brereton. Meta-analysis for Families of Experiments: A Systematic Review and Reproducibility Assessment, Empirical Software Engineering (2019) doi:10.1007/s10664-019-09747-0. This file holds the individual effect sizes for each experiment, as reported by 13 primary studies in the systematic review.

Usage

KitchenhamMadeyskiBrereton.ReportedEffectSizes

Format

A text file with variables:

Study

This field includes the study identifier of each of the 13 primary studies which were included in the systematic review.

Type

This identifies the type of effect size used by the study authors. d or g refer to dIG and gIG, p is the p-value used for aggregation, if the repeated measures estimate was obtained it is appropriately specified as gRM, r refers to the point bi-serial correlation.

Source

Always set to Rep. This identifies that the data was as reported by the primary study authors.

Design

The refers to the design method used by the study author. 4GroupCO is a 4-group crossover design. Mixed means different experiments in a particular family used different methods (only S3 used mixed methods and 4 experiments used the 4 group crossover and one used an independent groups design). ABBACO is the standard 2-group crossover design. IndGroups is the independent groups design also called between groups design or a randomised design. PrePost is pretest and posttest design with a post test control.

Exp1

This is the reported standardized effect size for the first experiment in the family.

Exp2

This is the reported standardized effect size for the second experiment in the family.

Exp3

This is the reported standardized effect size for the third experiment in the family.

Exp4

This is the reported standardized effect size for the fourth experiment in the family. NA means there was no fourth experiment in the family.

Exp5

This is the reported standardized effect size for the fifth experiment in the family. NA means there was no fifth experiment in the family.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBrereton.ReportedEffectSizes

Description

If you use this data set please cite this R package and the following paper when accepted: Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong, 'Robust Statistical Methods for Empirical Software Engineering', Empirical Software Engineering, vol. 22, no.2, p. 579-630, 2017. DOI: 10.1007/s10664-016-9437-5 (https://dx.doi.org/10.1007/s10664-016-9437-5), URL: https://madeyski.e-informatyka.pl/download/KitchenhamMadeyskiESE.pdf

Usage

KitchenhamMadeyskiBudgen16.COCOMO

Format

A data frame with variables:

Project

Project ID

Type

A categorical variable describing the type of the project

Year

The year the project was completed

Lang

A categorical variable describing the development language used

Rely

Ordinal value defining the required software reliability

Data

Ordinal value defining the data complexity / Data base size

Cplx

Ordinal value defining the complexity of the software / Process complexity

Aaf

??

Time

Ordinal value defining the stringency of timing constraints / Time constraint for cpu

Stor

Ordinal value defining the stringency of the data storage requirements / Main memory constraint

Virt

Virtual Machine volatility

Turn

Turnaround time

Type2

A categorical variable defining the hardware type: mini, max=mainframe, midi

Acap

Ordinal value defining the analyst capability

Aexp

Ordinal value defining the analyst experience / application experience

Pcap

Ordinal value defining the programming capability of the team / Programmers capability

Vexp

Ordinal value defining the virtual machine experience of the team

Lexp

Ordinal value defining the programming language experience of the team

Cont

??

Modp

/ Modern programming practices

Tool

Ordinal value defining the extent of tool use / Use of software tools

ToolCat

Recoding of Tool to labelled ordinal scale

Sced

Ordinal value defining the stringency of the schedule requirements / Schedule constraint

Rvol

Ordinal value defining the requirements volatility of the project

Select

Categorical value calculated by BAK for an analysis example

Rvolcat

Recoding of Rvol to a labelled ordinal scale

Modecat

Mode of the projects: O=Organic, E=Embedded, SD-Semi-Detached

Mode1

Dummy variable calculated by BAK: 1 if the project is Organic, 0 otherwise

Mode2

Dummy variable calculated by BAK: 1 if the project is Semi-detached, 0 otherwise

Mode3

Dummy variable calculated by BAK: 1 if the project is Embedded, 0 otherwise

KDSI

Product Size Thousand of Source Instructions

AKDSI

Adjusted Product Size for Project in Thousand Source Instructions - differs from KDSI for enhancement projects

Effort

Project Effort in Man months

Duration

Duration in months

Productivity

Productivity of project calculated by BAK as AKDSI/Effort, so the the larger the value the better the productivity

Details

Data set collected at TRW by Barry Boehm see: B.W. Boehm. 1981. Software Engineering Economics. Prentice-Hall.

Explanations by Barbara Kitchenham / https://terapromise.csc.ncsu.edu:8443/!/#repo/view/head/effort/cocomo/cocomo1/nasa93/nasa93.arff

COCOMO.txt: pro type year Lang Rely Data CPLX aaf time store virt turn type2 acap aexp pcap vexp lexp cont modp TOOL TOOLcat SCED RVOL Select rvolcat Modecat Mode1 Mode2 Mode3 KDSI AKDSI Effort Dur Productivity

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBudgen16.COCOMO

Description

If you use this data set please cite this R package and the following paper when accepted: Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong, 'Robust Statistical Methods for Empirical Software Engineering', Empirical Software Engineering, vol. 22, no.2, p. 579-630, 2017. DOI: 10.1007/s10664-016-9437-5 (https://dx.doi.org/10.1007/s10664-016-9437-5), URL: https://madeyski.e-informatyka.pl/download/KitchenhamMadeyskiESE.pdf

Usage

KitchenhamMadeyskiBudgen16.DiffInDiffData

Format

A data frame with variables:

Abstract

The abstract identifier

Site

A numeric identifier of the site

Treatment

A three character alphanumeric identifying the journal and time period of the abstract

Journal

The journal in which the abstract was published: IST or JSS

Timeperiod

The time period in which the abstract: 1 or 2

J1

The identifier for the judge who made the next 2 assessments

J1Completeness

The average completeness made by judge J1 based on the 8 completeness questions

J1Clarity

The clarity assessment made by judge J1

J2

The identifier for the judge who made the next 2 assessments

J2Completeness

The average completeness made by judge J2 based on the 8 completeness questions

J2Clarity

The clarity assessment made by judge J2

J3

The identifier for the judge who made the next 2 assessments

J3Completeness

The average completeness made by judge J3 based on the 8 completeness questions

J3Clarity

The clarity assessment made by judge J3

J4

The identifier for the judge who made the next 2 assessments

J4Completeness

The average completeness made by judge J4 based on the 8 completeness questions

J4Clarity

The clarity assessment made by judge J4

MeanCompleteness

The mean of J1Completeness, J2Completeness, J3Completeness, J4Completeness

MedianCompleteness

The median of J1Completeness, J2Completeness, J3Completeness, J4Completeness

MedianClarity

The median clarity of J1Clarity, J2Clarity, J3Clarity, J4Clarity

MeanClarity

The mean clarity of J1Clarity, J2Clarity, J3Clarity, J4Clarity

VarCompleteness

The variance of J1Completeness, J2Completeness, J3Completeness, J4Completeness

VarClarity

The variance clarity of J1Clarity, J2Clarity, J3Clarity, J4Clarity

Details

Data set was derived from the data reported in the SubjectData data set (subjectdata.txt). It contains the summary completeness and clarity data from 4 judges who assessed the same abstract. Only the initial 5 sites are included.

dinddata.txt

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBudgen16.DiffInDiffData

Description

If you use this data set please cite this R package and the following paper when accepted: Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong, 'Robust Statistical Methods for Empirical Software Engineering', Empirical Software Engineering, vol. 22, no.2, p. 579-630, 2017. DOI: 10.1007/s10664-016-9437-5 (https://dx.doi.org/10.1007/s10664-016-9437-5), URL: https://madeyski.e-informatyka.pl/download/KitchenhamMadeyskiESE.pdf

Usage

KitchenhamMadeyskiBudgen16.FINNISH

Format

A data frame with variables:

Project

Project ID

DevEffort

Development Effort measured in hours

UserEffort

Effort provided by the customer/user organisation measured in hours

Duration

Project duration measured in months

HWType

A categorical variable defining the hardware type

AppType

A categorical variable defining the application type

FP

Function Points measured using the TIEKE organisation method

Co

A categorical variable defining the company

Details

Data set collected from 9 Finish companies by Mr Hanna M\'aki from the TIEKE organisation see Barbara Kitchenham and Kari Kansala, Inter-item correlations among function points, Proceedings ICSE 15, 1983, pp 477-480

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBudgen16.FINNISH

Description

If you use this data set please cite this R package and the following paper when accepted: Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong, 'Robust Statistical Methods for Empirical Software Engineering', Empirical Software Engineering, vol. 22, no.2, p. 579-630, 2017. DOI: 10.1007/s10664-016-9437-5 (https://dx.doi.org/10.1007/s10664-016-9437-5), URL: https://madeyski.e-informatyka.pl/download/KitchenhamMadeyskiESE.pdf

Usage

KitchenhamMadeyskiBudgen16.PolishData

Format

A data frame with variables:

Abstract

The abstract identifier

Site

Numeric identifier for the site

Treatment

The first three characters of the Abstract field which identifies the journal and time period of the abstract

Journal

An acronym for the journal from which the abstract was obtained: IST or JSS

Timeperiod

The Time period in which the abstract was found: 1 or 2

J1

The identifier for the judge who made the next 2 assessments

J1Completeness

The average completeness made by judge J1 based on the 8 completeness questions

J1Clarity

The clarity assessment made by judge J1

J2

The identifier for the judge who made the next 2 assessments

J2Completeness

The average completeness made by judge J2 based on the 8 completeness questions

J2Clarity

The clarity assessment made by judge J2

J3

The identifier for the judge who made the next 2 assessments

J3Completeness

The average completeness made by judge J3 based on the 8 completeness questions

J3Clarity

The clarity assessment made by judge J3

J4

The identifier for the judge who made the next 2 assessments

J4Completeness

The average completeness made by judge J4 based on the 8 completeness questions

J4Clarity

The clarity assessment made by judge J4

MedianCompleteness

The median of J1Completeness, J2Completeness, J3Completeness, J4Completeness

MedianClarity

The median of J1Clarity, J2Clarity, J3Clarity, J4Clarity

Details

Data set derived from PolishSubjects data set collected at Wroclaw University. It summarizes the completeness and clarity data collected from 4 judges about the same abstract.

PolishData.txt

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBudgen16.PolishData

Description

If you use this data set please cite this R package and the following paper when accepted: Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong, 'Robust Statistical Methods for Empirical Software Engineering', Empirical Software Engineering, vol. 22, no.2, p. 579-630, 2017. DOI: 10.1007/s10664-016-9437-5 (https://dx.doi.org/10.1007/s10664-016-9437-5), URL: https://madeyski.e-informatyka.pl/download/KitchenhamMadeyskiESE.pdf

Usage

KitchenhamMadeyskiBudgen16.PolishSubjects

Format

A data frame with variables:

Judge

The identifier for each subject

Abstract

The identifier for each abstract - the code starts with a three alphanumeric string that defines the source of the abstract

OrderViewed

Each judge assessed 4 abstracts in sequence, this data item identifies the order in which the subject viewed the specified abstract

Completness1

Assessment by judge of question 1:Is the reason for the project clear? Can take values: Yes/No/Partly

Completness2

Assessment by judge of question 2: Is the specific aim/purpose of the study clear? Can take values: Yes/No/Partly

Completness3

Assessment by judge of question 3: If the aim is to describe a new or enhanced software technology (e.g. method, tool, procedure or process) is the method used to develop this technology defined? Can take values: Yes/No/Partly/NA

Completness4

Assessment by judge of question 4: Is the form (e.g. experiment, general empirical study, data mining, case study, survey, simulation etc.) that was used to evaluate the technology made clear? Can take values: Yes/No/Partly

Completness5

Assessment by judge of question 5: Is there a description of how the evaluation process was organised? Can take values: Yes/No/Partly

Completness6

Assessment by judge of question 6: Are the results of the evaluation clearly described? Can take values: Yes/No/Partly

Completness7

Assessment by judge of question 7: Are any limitations of the study reported?: Yes/No/Partly

Completness8

Assessment by judge of question 8: Are any ideas for future research presented?: Yes/No/Partly

Clarity

Assessment by judge of question regarding the overall understandability of the abstract: Please give an assessment of the clarity of this abstract by circling a number on the scale of 1-10 below, where a value of 1 represents Very Obscure and 10 represents Extremely Clearly Written.

Completness1NumValue

A numerical value for completeness question 1 where 0=No, Partly=0.5, yes =1

Completness2NumValue

A numerical value for completeness question 2 where 0=No, Partly=0.5, yes =1, NA means not applicable

Completness3NumValue

A numerical value for completeness question 3 where 0=No, Partly=0.5, yes =1, NA means not applicable or not answered

Completness4NumValue

A numerical value for completeness question 4 where 0=No, Partly=0.5, yes =1, NA means not applicable

Completness5NumValue

A numerical value for completeness question 5 where 0=No, Partly=0.5, yes =1, NA means not applicable

Completness6NumValue

A numerical value for completeness question 6 where 0=No, Partly=0.5, yes =1, NA means not applicable

Completness7NumValue

A numerical value for completeness question 7 where 0=No, Partly=0.5, yes =1, NA means not applicable

Completness8NumValue

A numerical value for completeness question 8 where 0=No, Partly=0.5, yes =1, NA means not applicable

Sum

The sum of the numerical completeness questions excluding those labelled NA

TotalQuestions

The count of the number of question related to completeness excluding questions considered not applicable

Completeness

Sum/TotalQuestions

Details

Data set collected at Wroclaw University of Technology (POLAND) by Lech Madeyski includes separate entries for each abstract assessed by a judge, that is 4 entries for each judge. Data collected from 16 subjects recruited from Wroclaw University of Technology who were each asked to assess 4 abstracts.

Note Only completeness question 2 was expected to be context dependent and have a NA (not applicable) answer, if other completeness answers were left blank, BAK coded the answer as NA

polishsubjects.txt

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBudgen16.PolishSubjects

Description

If you use this data set please cite this R package and the following paper when accepted: Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong, 'Robust Statistical Methods for Empirical Software Engineering', Empirical Software Engineering, vol. 22, no. 2, pp. 579–630, 2017. DOI: 10.1007/s10664-016-9437-5 (https://dx.doi.org/10.1007/s10664-016-9437-5), URL: https://madeyski.e-informatyka.pl/download/KitchenhamMadeyskiESE.pdf

Usage

KitchenhamMadeyskiBudgen16.SubjectData

Format

A data frame with variables:

Judge

Alphanumeric identifier for each judge

Institution

Numerical value identifying each site from which data was collected

JudgeID

Numerical value identifying each judge

Age

Age of the judge in years

Eng1st

Whether the judge's first language was Enlish: Yes/No

YearsStudy

The number of years have student been studying computing at University: 1, 2, 3, 4

AbstractsRead

Number of abstracts the judge had read prior to the study' 0, 1 to 10, 10+

AbstractsWritten

Whether the judge had ever written an abstract for a scientific report/article

AbstractID

Alphanumeric identifier for an abstract. The first character identifies the journal, I=IST, J=JSS, the third digit identifies the time period as 1 or 2, the remaining digits identify the abstract number within the set of abstracts found for the specified journal and time period

Treat

The initial 3 characters of AbstractID

TreatID

A numeric identifier for the journal and time period, 1=IB1, 2=IB2, 3=JB1, 4=JB2

Order

The order in which the judge should have viewed the specified abstract

Completness1NumValue

The numeric answer to completeness question 1

Completness2NumValue

The numeric answer to completeness question 2

Completness3NumValue

The numeric answer to completeness question 3

Completness4NumValue

The numeric answer to completeness question 4

Completness5NumValue

The numeric answer to completeness question 5

Completness6NumValue

The numeric answer to completeness question 6

Completness7NumValue

The numeric answer to completeness question 7

Completness8NumValue

The numeric answer to completeness question 8

Clarity

The response to the clarity question or NA if not answered

NumberOfAnsweredCompletnessQuestions

The number of completeness questions excluding those with NA

TotalScore

Sum of the numeric values of the 8 completeness questions

MeanScore

Sum of the completeness questions 1 to 8 divided by TotalScore

Site

The name of the site which provided the data. HongKong refers to the Polytechnic University, HongKong.2 refers to the City University

Details

Data set collected from 16 judges assessing 4 abstracts at 6 sites: Lincoln University NZ=1, Hong Kong Polytechnic University=2, PSu Thailand=3, Durham=4, Keele=5, Hong Kong City University=6

subjectdata.txt: Judge Institution JudgeID age eng1st years.study abs.read Absid Treat TreatID Order Com.1 Com.2 Com.3 Com.4 Com.5 Com.6 Com.7 Com.8 Clarity num.questions total.score av.score Site

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

KitchenhamMadeyskiBudgen16.SubjectData

Description

Returns a sample of N observations from a Laplace distribution with specified mean and spread.

Usage

LaplaceDist(N, mean, spread, max = 0.5, min = -0.5)

Arguments

Value

N values from a Laplace distribution

Author(s)

Barbara Kitchenham and Lech Madeyski

Examples

set.seed(123)
LaplaceDist(10, 0, 1)
#  [1] -0.55311564  0.85946218 -0.20094937  1.45258293  2.12808209 -2.39565480  0.05785263
#   [8]  1.53636446  0.10855453 -0.09076809

Description

If you use this data set please cite: Marian Jureczko and Lech Madeyski, 'Cross-project defect prediction with respect to code ownership model: An empirical study', e-Informatica Software Engineering Journal, vol. 9, no. 1, pp. 21-35, 2015. DOI: 10.5277/e-Inf150102 (https://dx.doi.org/10.5277/e-Inf150102) URL: https://madeyski.e-informatyka.pl/download/JureczkoMadeyski15.pdf)

Usage

Madeyski15EISEJ.OpenProjects

Format

A data frame with variables:

PROP

The percentage of classes of proprietary (i.e., industrial) projects that must be tested in order to find 80% of defects in case of software defect prediction models built on open source projects.

NOTOPEN

The percentage of classes of projects which are not open source projects that must be tested in order to find 80% of defects in case of software defect prediction models built on open source projects.

STUD

The percentage of classes of student (i.e., academic) projects that must be tested in order to find 80% of defects in case of software defect prediction models built on open source projects.

OPEN

The percentage of classes of open source projects that must be tested in order to find 80% of defects in case of software defect prediction models built on open source projects.

Details

This paper presents an analysis of 84 versions of industrial, open-source and academic projects. We have empirically evaluated whether those project types constitute separate classes of projects with regard to defect prediction. The predictions obtained from the models trained on the data from the open source projects were compared with the predictions from the other models (built on proprietary, i.e. industrial, student, open source, and not open source projects).

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

Madeyski15EISEJ.OpenProjects

Description

If you use this data set please cite: Marian Jureczko and Lech Madeyski, 'Cross-project defect prediction with respect to code ownership model: An empirical study', e-Informatica Software Engineering Journal, vol. 9, no. 1, pp. 21-35, 2015. DOI: 10.5277/e-Inf150102 (https://dx.doi.org/10.5277/e-Inf150102) URL: https://madeyski.e-informatyka.pl/download/JureczkoMadeyski15.pdf)

Usage

Madeyski15EISEJ.PropProjects

Format

A data frame with variables:

NOTPROP

The percentage of classes of non-proprietary (i.e., non-industrial) projects that must be tested in order to find 80% of defects in case of software defect prediction models built on proprietary (i.e., industrial) projects.

OPEN

The percentage of classes of open source projects that must be tested in order to find 80% of defects in case of software defect prediction models built on proprietary (i.e., industrial) projects.

STUD

The percentage of classes of student (i.e., academic) projects that must be tested in order to find 80% of defects in case of software defect prediction models built on proprietary (i.e., industrial) projects.

PROP

The percentage of classes of proprietary (i.e., industrial) projects that must be tested in order to find 80% of defects in case of software defect prediction models built on proprietary (i.e., industrial) projects.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

Madeyski15EISEJ.PropProjects

Description

If you use this data set please cite: Marian Jureczko and Lech Madeyski, 'Cross-project defect prediction with respect to code ownership model: An empirical study', e-Informatica Software Engineering Journal, vol. 9, no. 1, pp. 21-35, 2015. DOI: 10.5277/e-Inf150102 (https://dx.doi.org/10.5277/e-Inf150102) URL: https://madeyski.e-informatyka.pl/download/JureczkoMadeyski15.pdf)

Usage

Madeyski15EISEJ.StudProjects

Format

A data frame with variables:

PROP

The percentage of classes of proprietary (i.e., industrial) projects that must be tested in order to find 80% of defects in case of software defect prediction models built on student (i.e., academic) projects.

NOTSTUD

The percentage of classes of projects which are not student projects that must be tested in order to find 80% of defects in case of software defect prediction models built on student (i.e., academic) projects.

STUD

The percentage of classes of student (i.e., academic) projects that must be tested in order to find 80% of defects in case of software defect prediction models built on student (i.e., academic) projects.

OPEN

The percentage of classes of open source projects that must be tested in order to find 80% of defects in case of software defect prediction models built on student (i.e., academic) projects.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

Madeyski15EISEJ.StudProjects

Description

If you use this data set please cite: Lech Madeyski and Marian Jureczko, 'Which Process Metrics Can Significantly Improve Defect Prediction Models? An Empirical Study,' Software Quality Journal, vol. 23, no. 3, pp.393-422, 2015. DOI: 10.1007/s11219-014-9241-7

Usage

Madeyski15SQJ.NDC

Format

A data frame with variables:

Project

In case of open source projects this field includes the name of the project as well as its version. In case of industrial projects this field includes the string 'proprietary' (we were not allowed to disclose the names of the analyzed industrial software projects developed by Capgemini Polska).

simple

The percentage of classes that must be tested in order to find 80% of defects in case of simple defect prediction models, i.e., using only software product metrics as predictors.

advanced

The percentage of classes that must be tested in order to find 80% of defects in case of advanced defect prediction models, using not only software product metrics but also the NDC (Number of distinct committers) process metric.

Details

'This paper presents an empirical evaluation in which several process metrics were investigated in order to identify the ones which significantly improve the defect prediction models based on product metrics. Data from a wide range of software projects (both, industrial and open source) were collected. The predictions of the models that use only product metrics (simple models) were compared with the predictions of the models which used product metrics, as well as one of the process metrics under scrutiny (advanced models). To decide whether the improvements were significant or not, statistical tests were performed and effect sizes were calculated. The advanced defect prediction models trained on a data set containing product metrics and additionally Number of Distinct Committers (NDC) were significantly better than the simple models without NDC, while the effect size was medium and the probability of superiority (PS) of the advanced models over simple ones was high (p=.016, r=-.29, PS=.76), which is a substantial finding useful in defect prediction. A similar result with slightly smaller PS was achieved by the advanced models trained on a data set containing product metrics and additionally all of the investigated process metrics (p=.038, r=-.29, PS=.68). The advanced models trained on a data set containing product metrics and additionally Number of Modified Lines (NML) were significantly better than the simple models without NML, but the effect size was small (p=.038, r=.06). Hence, it is reasonable to recommend the NDC process metric in building the defect prediction models.' [https://dx.doi.org/10.1007/s11219-014-9241-7]

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

Madeyski15SQJ.NDC

Description

If you use this data set please cite this R package and the paper where we analyze the data set: Lech Madeyski and Barbara Kitchenham, 'Effect Sizes and their Variance for AB/BA Crossover Design Studies', Empirical Software Engineering, vol. 24, no.4, p. 1982-2017, 2018. DOI: 10.1007/s10664-017-9574-5

Usage

MadeyskiKitchenham.EUBASdata

Format

A data frame with variables:

ID

Project ID

TimePeriod

Period of time (run): R1, R2

SequenceGroup

Sequence group: G1, G2, G3, G4

System

Software system identifier indicates the system (i.e., S1 or S2) used as the experimental object: S1. A software system to sell and manage CDs/DVDs in a music shop, S2. A software system to book and buy theater tickets

Technique

The independent variable. It is a nominal variable that can assume the following two values: AM (analysis models plus source code) and SC (source code alone)

Comp_Level

This denotes the comprehension level of the source code achieved by a software engineer

Modi_Level

This denotes the capability of a maintainer to modify source code

Details

Data set comes from an experiment conducted in Italy at the University of Basilicata (with 24 first-year students from the Master's Program in Computer Science) to answer the question 'Do the software models produced in the requirements analysis process aid in the comprehensibility and modifiability of source code?', see G. Scanniello, C. Gravino, M. Genero, J. A. Cruz-Lemus, and G. Tortora, 'On the Impact of UML Analysis Models on Source-code Comprehensibility and Modifiability,' ACM Transactions on Software Engineering and Methodology, vol. 23, pp. 13:1-13:26, Apr. 2014. However, the inconsistent subject data for subject 2 was removed, see the aforementioned paper by Madeyski and Kitchenham.

Source

https://madeyski.e-informatyka.pl/reproducible-research/

Examples

MadeyskiKitchenham.EUBASdata

Description

Data form a set of primary studies on reading methods for software inspections. They were analysed by Lech Madeyski and Barbara Kitchenham, 'How variations in experimental designs impact the construction of comparable effect sizes for meta-analysis', 2015.

Usage

MadeyskiKitchenham.MetaAnalysis.PBRvsCBRorAR

Format

A data frame with 17 rows and 26 variables:

Study

Name of empirical study

Ref.

Reference to the paper reporting primary study or experimental run where data were originally reported

Teams

The number of teams including both, PBR and Control teams

DesignDesc

Experimental design description: Before-after, Between-groups, Cross-over

ExpDesign

Experimental design: between-groups (BG), within-subjects cross-over (WSCO), within-subjects before-after (WSBA)

M_PBR

The average proportion of defects found by teams using PBR

M_C

The average proportion of defects found by teams using Control treatment: Check-Based Reading (CBR) or Ad-Hoc Reading (AR)

Diff

The difference between M_PBR and M_C, i.e. Diff = M_PBR - M_C

Inc

The percentage increase in defect rate detection, i.e. Inc=100*[(M_PBR-M_C)/M_C]

SD_C_ByAuthors

The standard deviation of the control group values reported by the original Authors, i.e., obtained from the papers/raw data

SD_C

The standard deviation of the control group values equals SD_C_ByAuthors for studies for which the data was available OR the weighted average of SD_C_ByAuthors (i.e., 0.169) for studies where SD_C_ByAuthors is missing.

V_C

The variance of the Control group observations, i.e., the variance obtained from the teams using the Control method V_C=SD_C^2

V_D

The variance of the unstandardized mean difference D (between the mean value for the treatment group and the mean value for the Control group)

SD_C_Alt

This is the equivalent of SD_C (the standard deviation of the control group) based on a different variance for the student studies or the practitioner studies depending on the subject type of the study with the missing value.

V_Alt

The variance of the mean difference in the meta-analysis based on SD_C_Alt

SS_C

The sum of squares of the Control group values. For within subjects studies SS=V_C*(n-1). For between subjects studies SS=V_C*(n_C-1)

n_PBR

The number of PBR teams

n_C

The number of Control (CBR or AR) teams

ControlType

Type of Control treatment: CRB or AR

ParticipantsType

Type of participants: Engineers or Students

TeamType

Type of team: Nominal or Real

TwoPersonTeamVsLargerTeam

Reflects size of the teams: 2-PersonTeam or LargerTeam

ArtefactType

The type of artefact: Requirements or Other

AssociatedWithBasili

Whether study is associated with Basili (the forerunner): Yes or No

ControlType_Basili

Combined ControlType and AssociatedWithBasili: AH_AssociatedWithBasili, CBR_AssociatedWithBasili, CBR_NotAssociatedWithBasili