criterion performance measurements
overview
want to understand this report?
| alpmestan.com | |
| links/50 | |
| links/500 | |
| links/5000 | |
| google search | |
| youtube |
alpmestan.com/tagsoup
5.0 ms 5.5 6.0 6.5 7.0 7.5 8.0
mean |
5.0 ms 5.5 6.0 6.5 7.0 7.5 8.0
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 5.636 ms | 5.793 ms | 5.976 ms |
| Standard deviation | 760.9 μs | 867.4 μs | 974.5 μs |
Outlying measurements have severe (89.4%) effect on estimated standard deviation.
alpmestan.com/taggy
1.2 ms 1.4 1.6 1.8 2.0
mean |
1.2 ms 1.4 1.6 1.8 2.0
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 1.303 ms | 1.343 ms | 1.392 ms |
| Standard deviation | 191.5 μs | 226.6 μs | 266.4 μs |
Outlying measurements have severe (91.5%) effect on estimated standard deviation.
links/50/tagsoup
0.9 ms 1.0 1.1 1.2 1.3 1.4 1.5 1.6
mean |
0.9 ms 1.0 1.1 1.2 1.3 1.4 1.5 1.6
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 1.011 ms | 1.040 ms | 1.076 ms |
| Standard deviation | 136.7 μs | 164.1 μs | 196.9 μs |
Outlying measurements have severe (90.5%) effect on estimated standard deviation.
links/50/taggy
300 μs 325 350 375 400 425
mean |
300 μs 325 350 375 400 425
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 328.2 μs | 331.4 μs | 336.5 μs |
| Standard deviation | 13.86 μs | 20.18 μs | 29.18 μs |
Outlying measurements have severe (58.5%) effect on estimated standard deviation.
links/500/tagsoup
12 ms 13 14 15 16 17 18 19
mean |
12 ms 13 14 15 16 17 18 19
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 13.96 ms | 14.26 ms | 14.60 ms |
| Standard deviation | 1.460 ms | 1.643 ms | 1.865 ms |
Outlying measurements have severe (84.1%) effect on estimated standard deviation.
links/500/taggy
3.0 ms 3.5 4.0 4.5
mean |
3.0 ms 3.5 4.0 4.5
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 3.002 ms | 3.061 ms | 3.149 ms |
| Standard deviation | 273.7 μs | 366.9 μs | 473.1 μs |
Outlying measurements have severe (85.2%) effect on estimated standard deviation.
links/5000/tagsoup
170 ms 180 190 200 210 220 230
mean |
170 ms 180 190 200 210 220 230
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 180.3 ms | 181.4 ms | 183.3 ms |
| Standard deviation | 5.039 ms | 7.356 ms | 13.19 ms |
Outlying measurements have moderate (37.6%) effect on estimated standard deviation.
links/5000/taggy
34 ms 36 38 40 42 44
mean |
34 ms 36 38 40 42 44
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 37.88 ms | 38.33 ms | 38.80 ms |
| Standard deviation | 2.081 ms | 2.352 ms | 2.703 ms |
Outlying measurements have severe (58.5%) effect on estimated standard deviation.
google search/tagsoup
120 ms 125 130 135 140 145 150
mean |
120 ms 125 130 135 140 145 150
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 139.2 ms | 140.6 ms | 141.7 ms |
| Standard deviation | 5.176 ms | 6.301 ms | 7.670 ms |
Outlying measurements have moderate (42.5%) effect on estimated standard deviation.
google search/taggy
8.0 ms 9.0 10.0 11.0 12.0
mean |
8.0 ms 9.0 10.0 11.0 12.0
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 9.172 ms | 9.385 ms | 9.629 ms |
| Standard deviation | 1.039 ms | 1.163 ms | 1.294 ms |
Outlying measurements have severe (85.2%) effect on estimated standard deviation.
youtube/tagsoup
300 ms 320 340 360 380
mean |
300 ms 320 340 360 380
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 321.0 ms | 324.1 ms | 327.5 ms |
| Standard deviation | 13.92 ms | 16.56 ms | 21.15 ms |
Outlying measurements have moderate (49.4%) effect on estimated standard deviation.
youtube/taggy
35 ms 36 37 38 39 40
mean |
35 ms 36 37 38 39 40
|
| lower bound | estimate | upper bound | |
|---|---|---|---|
| Mean execution time | 36.04 ms | 36.25 ms | 36.51 ms |
| Standard deviation | 983.4 μs | 1.197 ms | 1.455 ms |
Outlying measurements have moderate (28.7%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. In each section, we display two charts, each with an x axis that represents measured execution time. These charts are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. Measurements are displayed on the y axis in the order in which they occurred.
Under the charts is a small table displaying the mean and standard deviation of the measurements. We use a statistical technique called the bootstrap to provide confidence intervals on our estimates of these values. The bootstrap-derived upper and lower bounds on the mean and standard deviation let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.