Experimental Design

Experimental Design has been a Science Olympiad event for many years in both divisions. In this event, competitors will design, execute, and write-up an experiment based on the topic and materials provided by the event supervisor.

Statement of Problem
The statement of problem is a question posed that will be explored in an experiment. One of the formats that can be used for almost any experiment is "How does (Independent Variable) affect (Dependent Variable)?" Ex. How does the height a ball is dropped from (1, 2, 3 meters) affect its rebound height (cm)? Tips: The statement of problem should not be a simple yes or no question. The key words are "how" and "why".

Hypothesis
The most common way a hypothesis is used in scientific research is as a tentative, testable, and falsifiable statement that explains some observed phenomenon in nature. This kind of statement is more specifically called an explanatory hypothesis. However, a hypothesis can also be a statement that describes an observed pattern in nature. In this case, the statement is called a generalizing hypothesis. The hypothesis statement is followed by a specific, measurable prediction that can can be made if the hypothesis is valid. Thus, in science the hypothesis is thought of as an explanation or generalization on trial.

A prediction in science is a prophecy, a specific and measurable event that is likely to happen in the future as the result of an experiment if the hypothesis is valid.

Teaching the Hypothesis Incorrectly Many teachers and even many textbooks teach the hypothesis in a way that makes it no different from a prediction. They teach students to write “If – then” statements for their hypotheses. This approach results in the incorrect form: If I do X, then Y will happen. There is no hypothesis here. This is simply a method (if I do X) followed by a prediction (then Y will happen). Some teachers and textbooks add “…because…” at the end of the “If…, then…” statement. The because statement is often close to the hypothesis that is being tested, but it still does not carefully delineate the hypothesis from the prediction. Indeed, even professional scientists can make mistakes.

The "because" part of the hypothesis is often referred to as the Rationale.

In short, to receive full points for this section, the hypothesis should be written like this: If I change _____(the IV), then the DV will _____(what changes) because ________(the rationale).

Ex. If I drop a ball from different heights (1, 2, 3 meters), then the rebound heights (in centimeters) for the higher drop heights will be greater than the lower drop heights because of Issac Newton's 3rd law (For every action, there is an equal and opposite reaction). This law applies to this experiment because when the drop height is greater, there is more force in the action of the ball falling to the floor, and thus the rebound height (the equal and opposite reaction) will be greater.

Tips: Make sure you have all of the parts for this section complete and explained thoroughly. Sometimes, a rationale may sound complete in a competitor's head, but in reality, it is not fully explained on paper, and the competitor would lose points on the basis of silly mistakes.

Variables
There are three different kinds of variables to be defined in the lab write-up: the independent variable, the dependent variable, and controlled variables.

Tips: Make sure to operationally define (units) all of your variables. Also make them clear and concise to make sure you get all of the points in a point-heavy part like this.

Independent Variable (IV)
The independent variable is the variable that is changed to examine its effect on the dependent variable. There should only be one IV, which should be listed with units. The IV must be operationally defined (in terms of the experiment) and empirically defined (in general for future variations of the experiment). Additionally, a minimum of three different levels of the independent variable must be listed excluding the control level.

Ex. the drop height (1, 2, 3 meters)

Dependent Variable (DV)
The dependent variable is what is affected by the independent variable. There should only be one IV, which should be listed with units. The dependent variable must be operationally and empirically defined. Do not include levels of the DV as that is what will be determined through the experiment. Ex. rebound height (in centimeters)

Controlled Variables (CV)
Controlled variables are factors which could affect the dependent variable but are kept constant throughout the experiment. Multiple controlled variables should be listed, but only four need to be listed to receive full credit for this section.

Ex. size of ball, material of ball, shape of ball, material of dropping surface, method of releasing ball (dropping, throwing, etc.)

Standard of Comparison or "Control"
The standard of comparison (SOC) is the "normal trial", or the one that hasn't been changed at all (there is only one). It serves as a neutral comparison for the other trials. A rationale for the SOC should be included.

Ex. The SOC for this experiment would be the 1 meter IV level. This SOC was chosen because it is the level closest to zero. Since the hypothesis predicted an increase in rebound heights with increasing drop heights, this IV level would either prove or disprove the hypothesis and would be useful for analyzing the experiment afterwards.

Tips: Changing the IV to zero or using the highest or lowest possible numeric value of the IV make good SOC's

Materials
The materials list is what it sounds like. It's a list of the materials used in the experiment. All materials used in the experiment should be listed (provided materials that are not used should not be included). The materials list should be as specific as possible; the quantity of the materials to be used, as well as brand names should be listed. A person should be able to look at the list and gather the exact materials used in the original experiment. The material of the independent variable should be listed once with the levels after it in parenthesis.

Some competitions want measuring devices to be listed, while others may take off points for it. The event supervisor should distinguish what they want.

Ex. 3 Penn racquetballs, 3 meter sticks

Tips: Make sure all materials are listed. Sometimes accidents happen and competitors lose points on what may be the simplest section ever. If time is leftover after finishing an experiment, all competitors should check that all materials are listed.

Procedure
The procedure is a list of the steps in the experiment and includes labeled diagrams (at least three) of how the experiment was performed. The procedure is included in write-ups so that other scientists reproducing the experiment know exactly how it was done the first time. The steps should be listed clearly as well as very specifically, and include three trials for each level of the IV.

Ex. 1. Gather all necessary materials. 2. Drop one ball at a height of 1 meter. Drop heights: 3. Record the rebound height. | 3m - O                                                                                         4. Repeat steps 2-3 twice more. |            "Drop height:" -> O 5. Drop the second ball at a height of 2 meters. |                              |             O<-"Rebound height" 6. Record the rebound height. | 2m - O  Where you drop       V             ^ 7. Repeat steps 5-6 twice more. |         the ball                           | How high the ball bounces 8. Drop the last ball at a height of 3 meters. |                  Floor->  ________       _______ after you drop it                                                      9. Record the rebound height. | 1m - O                                                  10. Clean up your workspace.

Tips: Steps such as, "Repeat steps X to Y", can be used to save time. Also, the very last step of the procedure should be "Clean up your workspace." Also, remember to clearly explain things so that anyone could read and replicate the experiment. Sometimes this is hard, like if the objective to explain how to fold a paper airplane, but still, this is an important section that is worth many points and is a possible tiebreaker section.

Qualitative Observations
There are three types of observations that must be made to receive full credit: observations about the procedure, results, and anything not related to the DV. Additionally, observations must be made over the course of the experiment about the three.

Typically, observations about the procedure are about noticing flaws in the experiment that went unnoticed before it was performed. These can include flaws in measurement technique, flaws in building/maintaining an experimental unit, and flaws in performing the actual experiment. Observations about the procedure carry over to experimental errors and practical applications, so be sure that you thoroughly explain what went wrong.

Ex. About the Procedure: Every time a competitor was preparing to drop a ball, his/her hands wobbled the slightest bit, maybe influencing the DV the slightest bit. Also, competitors noticed that it was hard to measure exactly where the ball rebounded to, especially in the 3 meter trials (lots of force => harder to see) since the ball was constantly in motion. About the results: During the experiment, when competitors measured the rebound height, they often had to estimate because it was sometimes hard to tell the exact height of the rebound. Also, the results showed that on average, the ball rebounded to about 5/8 of its original drop height. Other: When the ball rebounded after the initial rebound, the ball's path of bouncing wasn't exactly vertical, making the retrieving of the ball harder to do. Also, when each of the balls hit the floor at different heights, they made different sounds. The 3 meter ball was louder that the 2 and 1 meter balls.

Tips: DO NOT confuse these with errors. It is easy to combine qualitative observations and errors, and that will take points off of the experiment.

Quantitative Data
One way to organize the data is to make two tables. For the first, make a table of four rows and four columns. The first column should consist of, from top to bottom, a blank box, IV 1, IV 2, and IV 3. The second column should be labeled "Trial 1", and following boxes filled accordingly to the data. The next two columns follow the same layout as the Trial 1 box, but with Trials 2 and 3. Title the graph as seen fit for the data. Next to that table, draw a one column, four row condensed table (to the right). Name it "Average" (or AVG for short), and average the data for each IV. Put arrows from the second row of the first table to the second row of the condensed table, and so forth. Give a sample calculation for the average ((Trial 1+Trial 2+Trial 3)/3), located below the table or on one of the arrows. Remember to title both of your tables.

Also be sure use significant figures if you are in C Division, and be sure to keep them consistent and logical. You do not want to have a number down to three decimals when your ruler can only accurately measure to one decimal. See Significant Figures for additional info about significant figures.

Tips: Make sure the sample calculation and a condensed table are included. Those are worth many points and will not take much time. Also label your table properly.

Graph
A standard bar, line, or scatter-plot graph works almost universally at any competition level. Even so, always be sure to use the correct type for your data. Also, if your data starts off at a non-zero amount, you can draw a zigzag atop of the 0 mark to skip to your data.

Ex.  |  How the Drop Height of a Penn Racquetball R 230 |         affects its Rebound Height E 220 |                                     *O B 210 |                                   * O 200 |                                 *        KEY: U 190 |                               * N 180 |                             * D 170 |                           *                   O = data point 160 |                        O H 150 |                       *                     *** = line of best fit E 140 |                     * I 130 |                   * G 120 |                 * H 110 |               * T 100 |             * 90 |           *                                                                                                                              C  80 |          O M    _\       * 0 \__________________________________________________                                                                                                       |              |             |                                                                                                                       1m             2m            3m Drop Heights (m)

Tips: Label your axis (x+y), title the graph, use the DV as the y value and IV as the x value, title the individual axis, connect the data points or draw a line of best fit, only include the averages of the data for each IV

Statistics
Take the common statistics - mean, mode, range, median. Also include any other relevant statistics and show work. The best idea is to put all statistics in a neat table.

Your table of data should be neat- a ruler helps a lot. Be sure to keep writing your units.

Once your common statistics are done, make sure to do some more. Standard deviation is a very good statistic to include. The equation for calculating standard deviation is: [math]\sigma = \sqrt{avg((value - mean)^{2})}[/math]. For a better visual equation and an explanation of what standard deviation is (which you will need to know to explain the statistic), see Standard Deviation. Actually doing trials is necessary, as a standard deviation of a sample size of 1 is clearly stupid. A key point that is easy to miss is the deviation has to be squared. If you don't, your result will always be 0, and though this may look pretty, it should be obvious that your data does not have a standard deviation of 0.

As of 2015, both Division B and Division C will be expected to do more with the data, whereas before only C Division needed to. One essential aspect of the graph will be to create a regression, or line of best fit. Since both divisions are now permitted to bring any type of calculator, this would be a good time to invest in a nice TI-84 or similar graphing calculator because linear, logarithmic, and many other types of regression can be calculated with them. To calculate a linear regression on a TI calculator, start by putting your data in a list. Press STAT and go to the EDIT menu. Press 1 to edit your list. Then, exit and press STAT again. Go to the CALC menu to select the type of regression most suited for your data (which in most cases will be LinReg). Place the list containing your X values in Xlist, and repeat for Y values in Ylist. Scroll down and press Calculate, which will then give you the constants for your regression. Draw in the line on your graph and label it.

If you cannot get a graphing calculator, the best fit line of dubious accuracy is made by drawing a straight line with a ruler that you think seems to go as close to all the points on the graph. Once you do, find the y-intercept, and calculate the slope. Make sure to consider which outliers are significant, and which are experimental errors. If you make these kinds of judgment calls, make sure to point it out and explain it in the Analysis section.

Also, make sure you always have the same units through-out the experiment, if you are using milliseconds in the data table continue using milliseconds for everything else, DO NOT change to seconds or any other units. Division C competitors should remember to use significant figures in their statistics.

Tips: Graph your data. Make the graph neat, legible. Use a legend if need be. Label the axes (with units) and make a title for the graph (including units here as well is a good idea)

Analysis
This section should be one extended paragraph which at least touches on all data points and expands on outliers. Look at the data and draw some reasonable conclusions about the experiment. There should be trends; point them out and explain them. Discuss your statistics and again, explain them. Guesses are okay, even if they're wrong; they show your thought process. If you have any outliers or random "bad" data points, don't ignore them - again, write about them. Was there anything you did wrong that time, or was it just a fluke? Conclude by stating if the IV is directly, inversely, or not clearly proportional to the DV.

This section is the time for you to shine! Show the event supervisors that you are worthy of that gold medal and elaborate on your data, describing every last atom, and getting full points.

Ex: For our first IV level (1 meter), our rebound heights were 78 cm, 96 cm (outlier), and 69 cm, averaging 81 cm. For our second IV level (2 meters), our rebound heights were 160 cm, 171 cm, and 162 cm, averaging around 163.3 cm. for our last IV level (3 meters), our rebound heights were 220 cm (mode), 227 cm, and 220 cm (mode), averaging around 222.3 cm. Overall, it seems that this brand of racquetball (Penn) has about a 3:4 ratio of rebound height to drop height on a linoleum floor. This trend is clearly shown in the graph with a positive slope of about 3/4. Since a linoleum floor is mostly hard, the floor does not absorb much energy and sends the ball back with most of its energy ,relating to sir Issac Newton's 3rd Law of motion: for every action, there is an equal and opposite reaction. One might think that if Newton's third law was true, then the ball would rebound to the original drop height, but that is not the case. Many forces make the "equal and opposite reaction," such as gravity, friction/air resistance, and the floor, and in this case the equal and opposite reactions will most often send the ball to about 3/4 of its drop height. In the 1st IV level, there was an outlier. In the 2nd trial, the rebound height was 96 cm, which was significantly higher than average. Most likely, the height fluctuated a bit and caused the rebound height to be higher, which affected the average, which would've been 73.5, just under what the trend suggested. Otherwise, this IV level would be just what was expected. In the 2nd IV level, our results were all higher than what the trend showed (the trend being 3/4 of the drop height, in this case 150 cm). This may have been due to the fact that most people are shorter than 2 meters, and have a hard time measuring the exact rebound height. Although, even with the increased rebound heights, these results were still expected. In the last IV level, there was a mode of 220 cm. Probability-wise, the chance of this is very rare, due to the high drop height and high rebound height, but the dropper was able to get a brief approximation of what the rebound height was. It looked like the first and third trials both had a rebound height of 220 cm, so they were approximated. This IV level, with an average of about 223.3, fit the trend perfectly, only being about 1.7 cm under the 3:4 mark of 3 meters, 225 cm, and was just what was expected. To sum everything up, there was a trend of the ratio of rebound height to drop height being about 3:4, and all three IV levels fit it perfectly. Overall, the IV was directly proportional to the DV.

Tips: Make sure you explain everything very clearly. This is a point-heavy portion that depends on clarity. Also, explain everything. Everything in an experiment could be relevant to the Analysis, so include everything.

Possible Sources of Experimental Error
Look for all the things wrong with your experimental setup. How might they have caused inaccuracies in your data? This is extremely useful, because it can redeem mistakes made earlier in the event by showing that you are aware of them. Sometimes points can even be regained. Try to stay away from, but still use, human errors (what humans cause) and try to focus on experimental sources of errors like you can say things that have to do with temperature. The container of your object may have insulated it. Say any possible thing that could change the outcome of the experiment. Then, explain how the errors are believed to affect the data: increase from normal, decrease from normal, or either.

Also, this section can be written before any data is actually collected, and just added to if there are glaring errors in data collection that you didn't predict. If you're not sure how the experiment is going to turn out, this is a good thing to write first, though any errors that are included in your qualitative observations should be commented on.

Ex. This experiment had a few errors in it. First of all, the rebound height wasn't measured so that it was spot-on; it was guesstimated. Since the balls were moving at high velocities, often through the speed, we couldn't make out the exact heights. This is an experimental error because there wasn't an easy way to measure without using electronics. This sometimes increased, sometimes decreased the rebound heights, usually averaging out in the average but making the standard deviation higher. Another error was that the person who dropped the ball often had shaky arms, so sometimes, the speed of the ball would increase and sometimes the drop height would fluctuate. This is a human error, as the dropper made the mistake of not dropping the ball at the same velocity and height. This would mainly increase the rebound height, but would also decrease it rarely, if the dropper's hand was lower than normal.

Tips: Don't confuse errors with qualitative data. It is an easy way to make your total score of your experiment decrease. Also, even though only one error is required, more than one should be included if possible.

Conclusion
DO NOT ever say that your hypothesis was right or wrong. After one experiment, a hypothesis can either be supported or not supported by the data. Restate your hypothesis minus the explanation before concluding in either way. Then, explain why you came to that conclusion with your data as support. Never extrapolate anything; stick to what you observed even if you think the results were wrong. You may attempt to explain why your data varied from your hypothesis using proper scientific terminology that was not considered while writing your hypothesis, but the vast majority of the explanation should be data-based.

Ex. If a ball is dropped from different heights, then the ball dropped at the greatest height will have a greater rebound height. The data supported this hypothesis, with 1 meter drop heights averaging 81 cm, 2 meters averaging about 163 cm, and 3 meters averaging about 222 cm. Also, the graph showed a positive trend that had greater rebound heights when the drop height was greater. From these clear reasons, the hypothesis was accepted.

Tips: Make sure to include viable reasons. The reasons are an important part of this section. Also, make sure all statements are true and supported by the data.

Applications and Recommendations for Further Experimentation
When writing this section, consider variations of your experiment that would produce more accurate results. There should be three variations listed: one to improve a certain aspect of your experiment, one to approach the hypothesis in a different way, and one for a future experiment related to the DV. Finally, consider a practical application for the experiment. This section can also be written without any knowledge of how the experiment will turn out, and so it can also be written before data has been collected.

Ex. For further experimentation, a few things can be done to this experiment. First of all, since the measuring of the rebound height is hard to do, providing a way to measure more accurately should be provided. This would make the results more accurate and the data more reliable. To change the IV of this experiment, maybe instead of changing drop heights, the size of the ball could be the IV. This would provide a different experiment than this one and would probably be more useful in the real world. To change the DV, instead of measuring the rebound height, the measured thing could be how many times the ball bounces. This would provide an interesting trend and could apply to many things, such as sports. This experiment could be useful in the outside world in a few ways. First of all, it could be applied to basketball dribbling in the sport of basketball. If a ball is dribbled high, then it will bounce high and therefore be harder to control. If a ball is dribbled low, then it will rebound low, and it will be hard to move. This could be used to teach aspiring basketball players to dribble well. Also in the sport of basketball, this experiment could be useful in another way. The term "rebound" is a basketball term referring to the distance a basketball travels after it has hit the backboard of a basketball hoop and missed the hoop. When a ball is hit hard against the backboard, more force is being applied, so more force is in the reaction, propelling the ball greater distances. For example, if a basketball player knows s/he cannot make a shot, s/he can calculate what the perfect amount of force is to rebound off of the backboard and to another teammate's control. Finally, this experiment could be used in the tennis industry. Many tennis players have had balls fly over the court when hit with too much force to the ground. All tennis players know to not hit too hard or too soft, a principle demonstrated in this experiment.

Tips: Make sure to include applications and recommendations that are actually useful. Points may be deducted if you say something irrelevant.

Common Strategies
Knowing the scoring rubric is the key to success in Experimental Design. The rubric is a set of guidelines used for scoring experiments. When experimenters are aware of what is expected in each section, it becomes much easier to work efficiently.

At the start of the event, start by brainstorming possible experiments. Expect to get a handful of seemingly random items to test with, along with the possibility of a topic or prompt to design the experiment around. If each team member is familiar with general scientific concepts, designing an experiment should not be too difficult. Focus on execution and write-up, not on preparation. Spend no more than 5 minutes on this.

Keep your experiment simple. Too many variables can mean a lot of writing. Consider an example experience from one regional tournament. 3 balls (different colors), 2 rubber bands, a foot of masking tape, a metric stick, and a mini catapult were given. Naturally, one would want to experiment with the fanciest item given (catapult, in this case), but there would be so many variables to consider. Instead, performing a dropping experiment on how a rubber band affects the time it takes for a ball to drop. This is much simpler and, in this scenario, an idea that definitely paid off. Teams that used the catapult had balls flying everywhere throughout the event, and team members had to run around searching for them; wasting time. On the other hand, teams that utilized the other equipment achieved third out of thirty teams. Moral of the story is, ignore the urge to fiddle around with the complex stuff. Keep it simple- experiments will be simpler to write and test, saving time.

However, be sure to have enough trials. Having 3-5 trials for each variable ensures that data is sound and statistics have merit. This way if there is a possibility of strange data (one test being too high/low/fast/slow) there is the "experimental errors" section to comment on that. Only having 1-3 trials means there may not be enough data to show that a data point is strange, because there are not enough points to compare it to.

Know who is doing each section before the competition. All 3 people don't need to be doing the lab; only 1 or 2 people should be experimenting. Don't spend the whole time doing the lab either, 15-20 minutes should be plenty to get a significant amount of data for an experiment. There are many ways to divvy up the work on this event. Find something that works, each group is different.

If the experiment goes horribly wrong and all the data is skewed, focus on the report as much as possible. Make sure to explain why the experiment was bad and where it went wrong. This is where "Possible Experimental Errors" really counts-- be sure to write and explain every error which caused the experiment to go wrong. Having a bad experiment but a very good report  can, in some cases, cancel out the fact that the experiment didn't work.

Just like any other event, practice! Have a fellow teammate or coach gather materials and come up with a possible topic. Spend 50 minutes and come up with, test, and write up a lab report. Have the coach/teammate then grade the lab based on the rubric. This gives great insight into time usage and where improvements need to be made.

General Tips

 * Keep the experiment reasonable. There's a limited amount of time, so keep the scope small enough for a 50 minute event.
 * Come prepared, equipped with several different writing utensils, a ruler, a stopwatch and any type of calculator as long as it cannot access the Internet and does not have a camera.
 * Study the rubric: just to be safe, always look over the rubric before the competition. It may be very useful for designing the experiment.
 * Write neatly. If the judges can't read the experiment, they are not going to accept it. There will not be enough time for one person to write up everything perfectly, so writing neatly is crucial.
 * Think outside of the box - don't do exactly what every other team is inclined to do. Build a unique and intelligent experiment, which will set you apart from other teams and offer a better chance at medaling.
 * Be efficient: sometimes speed is extremely important due to the limited time of the event.
 * Be precise, specially when labeling the list of materials. Nothing is "too specific" for this event!
 * Keep it simple, stupid (KISS). Although not literally, never design an experiment that is too complicated, given the tight time limit.

Links

 * Experimental Design National Page (Div. C)
 * The rubric for Experimental Design The rubric rarely varies from year to year but has changed for the 2015 season.
 * Sample Experiments
 * Sample Experiment from Minnesota Div. C (Note: PDF)
 * Sample Experiment from Minnesota Div. B (Note: PDF)
 * Test Exchange