criterion performance measurements

overview

want to understand this report?

map/inline

370
375
380
385
390
395
400
405
map/inline time densities
mean
200
300
400
500
600
100 iters
100
150
200
250
0 s
50 ms
regression
map/inline times
lower bound estimate upper bound
OLS regression 372 μs 376 μs 384 μs
R² goodness-of-fit 0.998 0.999 1.000
Mean execution time 372 μs 373 μs 377 μs
Standard deviation 105 ns 5.16 μs 11.8 μs

Outlying measurements have slight (6.3%) effect on estimated standard deviation.

map/transformers

372
372
372
372
372
372
map/transformers time densities
mean
200
300
400
500
600
100 iters
100
150
200
250
0 s
50 ms
regression
map/transformers times
lower bound estimate upper bound
OLS regression 372 μs 372 μs 372 μs
R² goodness-of-fit 1.000 1.000 1.000
Mean execution time 372 μs 372 μs 372 μs
Standard deviation 47.6 ns 91.8 ns 164 ns

Outlying measurements have slight (1.1%) effect on estimated standard deviation.

map/transformers+inline

370
375
380
385
390
395
400
405
map/transformers+inline time densities
mean
200
300
400
500
600
100 iters
100
150
200
250
0 s
50 ms
regression
map/transformers+inline times
lower bound estimate upper bound
OLS regression 372 μs 375 μs 381 μs
R² goodness-of-fit 0.998 0.999 1.000
Mean execution time 373 μs 373 μs 377 μs
Standard deviation 1.34 μs 5.08 μs 11.0 μs

Outlying measurements have slight (6.3%) effect on estimated standard deviation.

drop/inline

100
102
104
106
108
110
drop/inline time densities
mean
2
3
4
5
6
7
8
9
1 iters
400
600
800
0 s
200 ms
1 s
regression
drop/inline times
lower bound estimate upper bound
OLS regression 99.6 ms 105 ms 112 ms
R² goodness-of-fit 0.991 0.995 0.999
Mean execution time 103 ms 106 ms 108 ms
Standard deviation 2.46 ms 3.40 ms 4.69 ms

Outlying measurements have slight (9.9%) effect on estimated standard deviation.

drop/transformers

512
514
516
518
520
522
524
drop/transformers time densities
mean
1
2
2
3
3
4
4
0.5 iters
2
0 s
500 ms
1 s
1.5
2.5
regression
drop/transformers times
lower bound estimate upper bound
OLS regression 439 ms 501 ms 545 ms
R² goodness-of-fit 0.994 0.998 1.000
Mean execution time 512 ms 518 ms 521 ms
Standard deviation 0 s 5.67 ms 6.49 ms

Outlying measurements have moderate (18.8%) effect on estimated standard deviation.

drop/transformers-inline

100
102
104
106
108
110
drop/transformers-inline time densities
mean
2
3
4
5
6
7
8
9
1 iters
400
600
800
0 s
200 ms
1 s
regression
drop/transformers-inline times
lower bound estimate upper bound
OLS regression 99.4 ms 105 ms 111 ms
R² goodness-of-fit 0.990 0.995 0.999
Mean execution time 103 ms 105 ms 108 ms
Standard deviation 2.44 ms 3.52 ms 4.92 ms

Outlying measurements have slight (9.9%) effect on estimated standard deviation.

map . drop . map/inline

260
270
280
290
300
map . drop . map/inline time densities
mean
2
3
4
5
1 iters
2
0 s
500 ms
1 s
1.5
regression
map . drop . map/inline times
lower bound estimate upper bound
OLS regression 226 ms 287 ms 358 ms
R² goodness-of-fit 0.975 0.978 1.000
Mean execution time 261 ms 274 ms 292 ms
Standard deviation 4.39 ms 18.1 ms 24.2 ms

Outlying measurements have moderate (17.1%) effect on estimated standard deviation.

map . drop . map/transformers

960
970
980
990
1 s
1.01
map . drop . map/transformers time densities
mean
1
2
2
3
3
4
4
0.5 iters
1
2
3
4
0 s
regression
map . drop . map/transformers times
lower bound estimate upper bound
OLS regression 817 ms 932 ms 1.04 s
R² goodness-of-fit 0.993 0.998 1.000
Mean execution time 959 ms 985 ms 1.00 s
Standard deviation 0 s 24.1 ms 27.4 ms

Outlying measurements have moderate (18.7%) effect on estimated standard deviation.

map . drop . map/transformers-inline

260
265
270
275
map . drop . map/transformers-inline time densities
mean
2
3
4
5
1 iters
500
750
0 s
250 ms
1 s
1.25
1.5
regression
map . drop . map/transformers-inline times
lower bound estimate upper bound
OLS regression 227 ms 264 ms 291 ms
R² goodness-of-fit 0.982 0.992 1.000
Mean execution time 261 ms 268 ms 274 ms
Standard deviation 2.49 ms 7.34 ms 10.0 ms

Outlying measurements have moderate (16.0%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.

We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.