Documentation

Reports are generated for each group and for each field selected by the configuration. Each report presents benchmarks in a single group with a single column presenting a single field. If there are m fields and n groups selected by the configuration then a total of m x n reports are generated. Output files are named using -estimator-groupname-fieldname as suffix.

One report is generated for each field selected by the configuration. Each report presents a field with all the groups selected by the configuration as columns or clusters. Output files are named using -estimator-fieldname as suffix.

One report is generated for each group selected by the configuration. Each report presents a group with all the fields selected by the configuration as columns or clusters. Output files are named using -estimator-groupname as suffix.

Show absolute values of the field for all groups

Show baseline group as absolute values and values for the subsequent groups as difference from the baseline

If the value of the group being compared is higher than the baseline then display the difference as percentage of baseline otherwise display the difference as a percentage of the group being compared.

If the value of the group being compared is higher than the baseline then display +(value / baseline value) otherwise display -(baseline value / value). This provides a normalized comparison independent of the absolute value of a benchmark. Note that Multiples can be directly computed using PercentDiff and vice-versa.

Size of a tick, the unit is microseconds for time fields, and bytes for space fields.

Total number of ticks in the range spread.

Specify the index of the sort column. Index 0 corresponds to the first value column. In a textual report, the very first column consists of benchmark names, therefore index 0 addresses the second column of the report.

Specify the column using the name of the group or the field it represents, and the runId. When just the name is enough to uniquely identify the sort column the Left constructor can be used, otherwise the Right constructor is used which can use the runId to disambiguate. In a Fields presentation, just the field name is enough. In a Groups presentation, when there is a single benchmark run in the input file, just the group name is enough to identify the group, the runId defaults to 0. However, when there are multiple runs, a group needs to specify a runId as well.

Report the median, outliers and outlier variance using box-plot method. This is the most robust indicator with respect to outliers when successive runs of benchmarks are compared.

Report the mean and the standard deviation from the mean. This is less robust than median but more precise.

Report the coefficient of regression, discarding the constant factor, arrived at by linear regression using ordinary least square method. The R-square goodness-of-fit estimate is also reported. It works better when larger number of samples are taken. This cannot be used when the number of samples is less than 2, in that case a mean value is reported instead.

Use a single estimator to compute the difference between the baseline and the candidate. The estimator that is provided in the Config is used.

Use Mean, Median and Regression estimators for both baseline and candidate, and report the estimator that shows the minimum difference. This is more robust against random variations.