Pinewood Derby Race Method Evaluation
A CASE STUDY

by Stan Pope

Problem Summary

A group of 28 Cub Scouts will be ready to race their Pinewood Derby Cars in hopes of winning one of the 4 Trophies their pack offers and in hopes of being one of the 4 Scouts who will represent our pack at the big District Races. The Pack Committee has asked me to recommend a racing plan that will do a accurate job of awarding the trophies and selecting our pack's representatives.

In addition to accuracy, the committee would like for all of the Cubs to be involved for most of the racing. They asked for something other than a classic elimination style competition because of the relatively early "fall-out" of so many participants. They think that since the boys put so much effort into building their cars, they ought to get to race them more than 3 or 4 times.

Finally, they would like for the racing to last about 60 to 90 minutes!

We have a 4-lane track with an electronic finish line that registers finish places 1 through 4 for the heat. If the contestants for each heat were known in advance, about 90 4-car heats could be completed in the available time.

The following are some cases I evaluated:

1. Stearns: 13 rounds, 7 heats per round (91 heats).
   
   
2. Stearns: 10 rounds, 7 heats per round (70 heats).
   Select 13 top scorers as participants in Perfect 13-4 (13 heats). Total 83 heats.
   
   
3. Stearns: 10 rounds, 7 heats per round (70 heats).
   Select 7 top scorers as participants in Perfect 7-3 (7 heats). Total 77 heats.
   
   
4. Stearns: 10 rounds, 7 heats per round (70 heats).
   Select 7 top scorers as participants in Complementary-Perfect 7-3 (14 heats). Total 84 heats.
   
   
5. Young and Pope: 2 rounds, 28 heats per round (56 heats).
   Select 7 top scorers as participants in Complementary-Perfect 7-3 (14 heats). Total 70 heats.

The 10-Round Stearns could be the first 10 rounds of a 13-round Chart. But how well would it work? It all depends upon how well the Stearns chart fragment could select the fastest 4 cars as a group of 7 or 13 cars. As it turns out, the 13-round Stearns chart and the 10-round Stearns fragment perform much better for screening than for trophy assignment or representative selection.

Each of these cases has the flexibility to adapt to other initial numbers of participants.

Summary of Results

Method	4-Trophy Error	Top-4 Error	Total Time (minutes)	Finals Time (minutes)	% of Time with Full Participation
13-Round Stearns	28.10%	7.30%	91	0	100%
10-Round Stearns plus P13-4 (1)	23.12%	5.32%	83	13	84%
10-Round Stearns plus P7-3 (1)	19.30%	3.77%	77	7	91%
10-Round Stearns plus CP7-3 (2)	12.21%	2.72%	84	14	83%
2-Round Young and Pope plus CP7-3 (2)	12.19%	2.70%	70	14	80%

Case 1: 13-Round (91 heat) Stearns

From the 13-Round Stearns simulation, we immediately see the following information:

Accuracy measures: 13-Round 28-Car Stearns
Trophy		Representation
4-Trophy:	71.90%	Top-4:	92.70%

Error measures: 13-Round 28-Car Stearns
Trophy		Representation
4-Trophy:	28.10%	Top-4:	7.30%

Error Measures are "the other side" of accuracy measures. If I got 72% right, then I got 100% - 72% = 28% wrong! (I like to use error measures because the impact of decisions is easier to visualize. For instance if the method error is 28%, then about one in three of the results is wrong! If I use a method with 28% error to award the 4 top trophies, then one or more of the boys (on the average) did not get a trophy that he deserved. For me, thinking this way changes the analysis from a mathematical exercise to a study of real effects on real, live boys.)

On the plus side, all of the scouts were involved in the racing from beginning to end.

Case 2: 10-Round (70 heat) Stearns plus
(13 heat) Perfect 13-4 (1)

Use a 10 round Stearns Chart to select 13 finalists; Race the finalists using a Perfect 13-4 (1) Chart.

From the 10-Round Stearns simulation, we immediately see the following information:

Accuracy measures: 10-Round 28-Car Stearns
Trophy		Representation		Screening
13-Trophy:	(ignored)	Top-13:	(ignored)	4n13:	100.00%

From the Perfect 13-4 (1) simulation, we immediately see the following information:

Accuracy measures: P13-4 (1)
Trophy		Representation
4-Trophy:	76.88%	Top-4:	94.68%

To form the composite accuracies, the two relevant measures are multiplied... 100% of the deserving advanced and 76.88% of those present were awarded the correct trophy.

Accuracy measures: Composite
Trophy		Representation
4-Trophy:	76.88%	Top-4:	94.68%

Again, error measures tell the real tale...

Error measures: Composite
Trophy		Representation
4-Trophy:	23.12%	Top-4:	5.32%

The accuracy is better. About a quarter of the trophies are, on the average, awarded to the wrong boy.

This method will keep all of the Scouts racing for the first 80+% of the time (10 races each, 70 minutes). The last few minutes will be spent running the finals: 13 heats with 4 races for each of 13 scouts.

Case 3: 10-Round (70 heat) Stearns
plus (7 heat) Perfect 7-3 (1)

Use a 10-round Stearns Chart to select 7 finalists; Race the finalists using a Perfect 7-3 (1) Chart.

From the 10-Round Stearns simulation, we immediately see the following information:

Accuracy measures: 10-Round 28-Car Stearns
Trophy		Representation		Screening
7-Trophy:	(ignored)	Top-7:	(ignored)	4n7:	99.88%

From the Perfect 7-3 (1) simulation, we immediately see the following information:

Accuracy measures: P7-3 (1)
Trophy		Representation
4-Trophy:	80.80%	Top-4:	96.35%

To form the composite accuracies, the two relevant measures are multiplied... 99.88% of the deserving advanced and 80.80% of those present were awarded the correct trophy. So 99.88% of 80.80% yields a composite average of 80.70%.

Accuracy measures: Composite
Trophy		Representation
4-Trophy:	80.70%	Top-4:	96.23%

The error measures say ...

Error measures: Composite
Trophy		Representation
4-Trophy:	19.30%	Top-4:	3.77%

The accuracy is better. About a fifth of the trophies are, on the average, awarded to the wrong boy.

This method will keep all of the Scouts racing for the first 90+% of the time (10 races each, 70 minutes). The last few minutes will be spent running the finals: 7 heats with 3 races for each of 7 scouts.

Case 4: 10-Round (70 heat) Stearns
plus (14 heat) Complementary-Perfect 7-3 (2)

Use a 10-round Stearns Chart to select 7 finalists; Race the finalists using a Complementary-Perfect 7-3 (2) Chart.

From the 10-Round Stearns simulation, we again see the following information:

Accuracy measures: 10-Round 28-Car Stearns
Trophy		Representation		Screening
7-Trophy:	(ignored)	Top-7:	(ignored)	4n7:	99.88%

From the Complementary-Perfect 7-3 (2) simulation, we immediately see the following information:

Accuracy measures: CP7-3 (2)
Trophy		Representation
4-Trophy:	87.90%	Top-4:	97.40%

To form the composite accuracies, the two relevant measures are multiplied... 99.88% of the deserving advanced and 87.90% of those present were awarded the correct trophy. So 99.88% of 87.90% yields a composite average of 87.79%

Accuracy measures: Composite
Trophy		Representation
4-Trophy:	87.79%	Top-4:	97.28%

The error measures say ...

Error measures: Composite
Trophy		Representation
4-Trophy:	12.21%	Top-4:	2.72%

The accuracy is better. About one eighth of the trophies are, on the average, awarded to the wrong boy. The representation error has been reduced to less than one in thirty!

This method will keep all of the Scouts racing for the first 80+% of the time (10 races each, 70 minutes). The last few minutes will be spent running the finals: 14 heats with 6 races for each of 7 scouts.

Case 5: 2-Round (56 heat) 2-Round Young and Pope
plus (14 heat) Complementary-Perfect 7-3 (2)

Use a 2-round Young and Pope (Partial Perfect N) Chart to select 7 finalists; Race the finalists using a Complementary-Perfect 7-3 (2) Chart.

From the 2-round Young and Pope simulation, we see the following information:

Accuracy measures: 2-Round 28-Car Young and Pope
Trophy		Representation		Screening
7-Trophy:	(ignored)	Top-7:	(ignored)	4n7:	99.90%

From the Complementary-Perfect 7-3 (2) simulation, we immediately see the following information:

Accuracy measures: CP7-3 (2)
Trophy		Representation
4-Trophy:	87.90%	Top-4:	97.40%

To form the composite accuracies, the two relevant measures are multiplied... 99.90% of the deserving advanced and 87.90% of those present were awarded the correct trophy. So 99.90% of 87.90% yields a composite average of 87.81%

Accuracy measures: Composite
Trophy		Representation
4-Trophy:	87.81%	Top-4:	97.30%

The error measures say ...

Error measures: Composite
Trophy		Representation
4-Trophy:	12.19%	Top-4:	2.70%

The accuracy is about the same as Case 4. About one eighth of the trophies are, on the average, awarded to the wrong boy. The representation error has been reduced to less than one in thirty!

This method will keep all of the Scouts racing for the first 80% of the time (8 races each, 56 minutes). The last few minutes will be spent running the finals: 14 heats with 6 races for each of 7 scouts.

Reference: Analysis Tool, Parameters and Explanation

constructed a very nice piece of software for chart simulation, "DerbySim". It is available in the "Software" section of his Cub Scout Pack's Website, The results are more or less consistent with what I produced, but his program is much more sophistocated than what I built for my C-64. The results are not directly comparable because, with his mathematics and programming prowess, his more powerful computer and his better programming languages, he was able to bridge many of my simplifying assumptions.

The following table shows the parameters that I used with "DerbySim" program for this study. These are not necessarily the best numbers to model the track and competitors. They seem reasonable, but runs using other models may be useful in validating the overall strategy.

Parameters	Count	Mean	Std. Deviation
Car Speed			10.0"
Lane Speed			1.0"
Lub Breakin		.5"	.2"
Lub Degradation		.5"	.2"
Threshold	2		1
Random element			.5"
Reduction	4
Trials	1000
Tie-Break	Head to Head
Scoring	Linear: 4-3-2-1

Here is a brief description of the numbers:

Car Speed

Finish location in inches relative to an imaginary "reference car" attributable to the car's speed. Randomized at the beginning of each trial.

Lane Speed

Finish location in inches relative to an imaginary "reference car" attributable to the each lane's characteristics. Randomized at the beginning of each trial.

Lub Breakin

Finish location in inches relative to an imaginary "reference car" attributable to the performance improvement from "breakin" of the lubricant. Randomized at the beginning of each trial.

Lub Degradation

Finish location in inches relative to an imaginary "reference car" attributable to the performance degradation from loss of the lubricant. Randomized at the beginning of each trial.

Threshold

The number of runs at which the lubricant transitions from breakin to degradation.

Random element

From run to run a car will experience a variation from its average performance within +/- 0.5 inches 67% of the time. and within +/- 1.0 inch 96% of the time. This factor is randomized for each run of each car.

Reduction

Number of top cars required from the screening.

Trials

How many times the chart was simulated to produce the averages quoted.

Tie-Break

Simple head-to-head racing (one heat) is used to break ties.

Scoring

Linear: 1st place, 4 points; 2nd place, 3 points; etc.

Reference: Analysis of 13-Round Stearns

This is a 13-Round Stearns Chart, as published at fine website. The chart was created using Peronto's implementation of Stearns Method.

Chart:
Within each round, rows are heats, columns are lanes. For example in Round 1 Car 7 races in lane 2 of heat 4.

Round 1					Round 2					Round 3					Round 4
17	14	24	2		20	25	4	24		2	9	27	18		27	11	26	10
11	20	15	22		14	3	28	11		21	1	16	15		18	13	25	14
25	21	6	13		26	8	13	5		8	24	14	4		19	15	8	17
4	7	3	8		7	12	21	1		13	26	10	3		23	2	20	7
5	18	23	16		22	27	18	6		12	17	11	25		3	5	9	28
1	28	12	26		15	16	2	9		28	19	7	20		24	4	22	21
9	10	19	27		10	23	17	19		6	22	5	23		16	6	1	12
Round 5					Round 6					Round 7					Round 8
21	9	11	8		18	21	3	20		21	23	26	2		24	16	26	19
5	17	27	7		5	1	24	15		10	7	22	24		4	10	6	8
18	10	28	15		27	28	23	14		27	16	8	20		23	1	20	18
1	3	19	25		7	13	16	25		5	14	12	19		2	11	5	25
13	23	24	12		9	22	17	26		18	4	1	17		22	3	12	27
20	6	26	14		12	2	8	10		28	6	25	9		17	13	28	21
16	4	2	22		11	6	4	19		15	13	11	3		7	14	9	15
Round 9					Round 10					Round 11					Round 12
6	24	3	2		28	24	8	9		9	12	20	23		15	23	3	11
10	20	5	21		17	3	20	13		22	14	1	8		27	24	18	8
23	8	25	22		4	5	23	12		21	27	4	26		13	4	9	20
17	16	11	7		16	14	10	1		19	13	2	28		21	14	7	26
13	27	1	9		26	15	25	27		17	6	15	18		10	16	28	22
4	28	12	15		21	19	2	22		24	11	16	3		1	2	19	5
14	19	18	26		6	7	18	11		5	25	10	7		6	17	12	25
Round 13
26	13	22	15
3	16	27	21
20	24	9	5
25	23	8	28
1	11	17	2
14	19	6	4
12	18	7	10

Although the chart has some imperfections, it does seem to conform to the Stearns Specifications. It also has the Stearn's strengths, in that every car races 13 times!

The chart is scored with linear point assignment... 1st place in heat = 4 points, 2nd place = 3 points, etc. Other simulations indicate that this scoring method provides more accurate results than the "1st place in heat = 1 point, rest of cars 0 points" scoring recommended in the Stearns program documentation.

The accuracy measures are explained in good detail in the documentation for DerbySim. In summary, they are ...

N-Trophy

Percentage of time that a car among the N objectively fastest receives correct trophy.

Top-N

Percentage of time that a car among the N objectively fastest placed in top N.

NnX

Percentage of time that a car among the N objectively fastest placed in top X. This measures the chart's utility as a "screening" tool.

Accuracy measures: 13-Round 28-Car Stearns
Trophy		Representation		Screening
1-Trophy:	90.40%	Top-1:	90.40%
2-Trophy:	83.70%	Top-2:	91.50%
3-Trophy:	77.03%	Top-3:	91.90%
4-Trophy:	71.90%	Top-4:	92.70%	4n4:	92.70%
5-Trophy:	67.40%	Top-5:	93.04%	4n5:	98.38%
6-Trophy:	62.90%	Top-6:	92.65%	4n6:	99.50%
7-Trophy:	59.49%	Top-7:	93.79%	4n7:	99.88%
8-Trophy:	57.06%	Top-8:	94.65%	4n8:	99.97%
9-Trophy:	55.26%	Top-9:	95.24%	4n9:	100.00%
10-Trophy:	53.47%	Top-10:	95.21%	4n10:	100.00%
11-Trophy:	51.68%	Top-11:	95.28%	4n11:	100.00%
12-Trophy:	50.08%	Top-12:	95.71%	4n12:	100.00%
13-Trophy:	49.05%	Top-13:	96.11%	4n13:	100.00%

These are measures of the specific Stearns chart in the example, and not for Stearns charts in general. It is likely that other runs of the Stearns generator with the same parameters would produce charts with other accuracy characteristics. It seems likely that those measures would not vary markedly from what is depicted here.

The most important measures of this chart, according to its proposed usage, will be its ability to accurately assign trophies for first through fourth place and to select the four representatives to the district races.

Measures through 13 are included for comparison to the measures of the 10-round partial chart below.

Reference: Analysis of 10-Round Stearns

The chart for this alternative is made using the first 10 rounds of the 13-Round Stearns chart above. As would be expected, the accuracy measures fall off slightly with the reduction in Rounds.

This Stearns fragment may not satisfy all of the requirements for a Stearns Method Chart. In particular, it may be inferior to a properly generated 10-Round Stearns Chart. It is used in this study because it should have many of the same performance characteristics as its 13-Round parent, and so will introduce fewer variables into the comparisons.

Like all Stearns Charts, all the cars participate regularly throughout the competition. Each car races 10 times.

Accuracy measures: 10-Round 28-Car Stearns
Trophy		Representation		Screening
1-Trophy:	90.30%	Top-1:	90.30%
2-Trophy:	82.60%	Top-2:	91.15%
3-Trophy:	75.50%	Top-3:	90.67%
4-Trophy:	69.30%	Top-4:	90.85%	4n4:	90.85%
5-Trophy:	64.08%	Top-5:	91.92%	4n5:	97.38%
6-Trophy:	60.18%	Top-6:	92.75%	4n6:	99.38%
7-Trophy:	57.04%	Top-7:	93.47%	4n7:	99.88%
8-Trophy:	54.69%	Top-8:	93.55%	4n8:	99.95%
9-Trophy:	52.43%	Top-9:	94.24%	4n9:	99.97%
10-Trophy:	50.44%	Top-10:	94.61%	4n10:	100.00%
11-Trophy:	48.84%	Top-11:	94.81%	4n11:	100.00%
12-Trophy:	47.40%	Top-12:	95.40%	4n12:	100.00%
13-Trophy:	46.18%	Top-13:	95.58%	4n13:	100.00%

These are measures of the specific 10-Round Stearns chart fragment in the example, and not for Stearns charts or fragments in general. See additional comments under the 13-Round Stearns chart accuracy measures.

The most important measures of this chart, according to its proposed usage, will be its ability to include the top 4 cars in its selection of the Top N, 4n6, 4n7, etc.

Reference: Analysis of 2-Round Young and Pope

The chart for this alternative is made using the publicly available Young and Pope (Partial Perfect N) Chart Generator, which was created by Cory Young and Stan Pope. Accuracy measures are comparable to the 10-Round Stearns chart fragment.

Chart Generation parameters used were:

• Number of lanes: 4
• Number of cars: 28
• Number of rounds: 2
• Heat Ordering Options: all "Medium"

2-Round 28-Car Young and Pope
Round 1					Round 2
1	5	10	16		28	4	9	15
2	6	11	17		19	26	18	16
3	7	12	18		25	1	6	12
4	8	13	19		8	15	7	5
5	9	14	20		23	2	22	20
24	3	23	21		14	21	13	11
28	7	27	25		1	8	28	26
11	15	20	26		10	17	9	7
13	17	22	28		27	6	26	24
6	10	15	21		16	20	25	3
14	18	23	1		5	12	4	2
25	4	24	22		19	23	28	6
12	16	21	27		11	18	10	8
15	19	24	2		15	22	14	12
22	26	3	9		20	27	19	17
8	12	17	23		9	13	18	24
10	14	19	25		7	11	16	22
26	2	7	13		17	21	26	4
9	16	8	6		3	10	2	28
18	22	27	5		16	23	15	13
4	11	3	1		27	3	8	14
21	28	20	18		26	5	25	23
23	27	4	10		2	9	1	27
20	24	1	7		24	28	5	11
12	19	11	9		7	14	6	4
21	25	2	8		13	20	12	10
6	13	5	3		18	25	17	15
17	24	16	14		22	1	21	19

Like the Stearns Charts, all the cars participate regularly throughout the competition. Each car races 8 times.

Accuracy measures: 2-Round 28-Car Young and Pope
Trophy		Representation		Screening
1-Trophy:	90.70%	Top-1:	90.70%
2-Trophy:	82.30%	Top-2:	90.00%
3-Trophy:	76.30%	Top-3:	92.27%
4-Trophy:	71.93%	Top-4:	92.72%	4n4:	92.72%
5-Trophy:	68.08%	Top-5:	93.38%	4n5:	98.03%
6-Trophy:	65.18%	Top-6:	94.03%	4n6:	99.45%
7-Trophy:	62.57%	Top-7:	94.64%	4n7:	99.90%
8-Trophy:	60.36%	Top-8:	95.11%	4n8:	100.00%
9-Trophy:	58.70%	Top-9:	95.33%	4n9:	100.00%
10-Trophy:	57.13%	Top-10:	95.60%	4n10:	100.00%
11-Trophy:	55.61%	Top-11:	95.98%	4n11:	100.00%
12-Trophy:	54.27%	Top-12:	96.20%	4n12:	100.00%
13-Trophy:	53.22%	Top-13:	96.61%	4n13:	100.00%

The most important measures of this chart, according to its proposed usage, will be its ability to include the top 4 cars in its selection of the Top N, 4n6, 4n7, etc.

Reference: Analysis of Perfect 13-4 (1)

This is a Perfect-13 Chart. It was produced using the publicly available Perfect-N Chart Generator, which was created by Cory Young and Stan Pope.

P13-4 (1)
1	2	4	10
2	3	5	11
3	4	6	12
4	5	7	13
5	6	8	1
6	7	9	2
7	8	10	3
8	9	11	4
9	10	12	5
10	11	13	6
11	12	1	7
12	13	2	8
13	1	3	9

Accuracy measures: P13-4 (1)
Trophy		Representation		Screening
1-Trophy:	90.90%	Top-1:	90.90%
2-Trophy:	84.85%	Top-2:	92.90%
3-Trophy:	80.37%	Top-3:	93.97%
4-Trophy:	76.88%	Top-4:	94.68%	4n4:	94.68%

The most important measures of this chart, according to its proposed usage, will be its ability to accurately assign trophies for first through fourth place and to select the four representatives to the district races.

Reference: Analysis of Perfect 7-3 (1)

This is a Perfect-7 Chart. It was produced using the publicly available Perfect-N Chart Generator, which was created by Cory Young and Stan Pope.

P7-3 (1)
1	2	4
2	3	5
3	4	6
4	5	7
5	6	1
6	7	2
7	1	3

Accuracy measures: P7-3 (1)
Trophy		Representation		Screening
1-Trophy:	90.60%	Top-1:	90.60%
2-Trophy:	85.65%	Top-2:	94.45%
3-Trophy:	82.87%	Top-3:	95.43%
4-Trophy:	80.80%	Top-4:	96.35%	4n4:	96.35%

The most important measures of this chart, according to its proposed usage, will be its ability to accurately assign trophies for first through fourth place and to select the four representatives to the district races.

Reference: Analysis of Complementary-Perfect 7-3 (2)

This is a Complementary Perfect-7 Chart. It was produced using the publicly available Perfect-N Chart Generator, which was created by Cory Young and Stan Pope.

Note that each head to head matchup occurs twice, in alternating lanes.

CP7-3 (2)
1	2	4
6	5	3
1	7	5
3	4	6
6	7	2
4	3	1
2	1	6
4	5	7
2	3	5
7	6	4
3	2	7
5	6	1
7	1	3
5	4	2

Accuracy measures: CP7-3 (2)
Trophy		Representation		Screening
1-Trophy:	95.20%	Top-1:	95.20%
2-Trophy:	92.25%	Top-2:	96.75%
3-Trophy:	89.93%	Top-3:	97.13%
4-Trophy:	87.90%	Top-4:	97.40%	4n4:	97.40%

The most important measures of this chart, according to its proposed usage, will be its ability to accurately assign trophies for first through fourth place and to select the four representatives to the district races.

Acknowledgements

This study has benefitted, in large measure, from frequent counsel, ideas and evaluation software provided by Cory Young. His patient, thoughtful responses to my inquiries have helped bring the project to a swift completion.

Latest update:
4/8/2001: Added Case 5 using Partial Perfect N as Screening Method
1/6/98: New simulations using 1/5/98 version of DerbySim

Copyright 1997, 2001 © by Stan Pope. All rights reserved.