Experiment Design

Introduction

For every cause, there is an effect. The two variables in this relationship are the causal variable and the effect variable. The causal variable is considered to be independent, and the effect variable is understood to be the dependent variable. Few cause and effect relationships are given below.

Scenario	Independent Variable (Cause)	Dependent Variable (Effect)
Sales increase after ad campaign	Advertising spend	Sales growth
Employee productivity rises after training	Training program	Productivity
More online sales after website redesign	Website redesign	Sales increase
Higher profits after hiring new manager	New leadership	Profitability
Increase in app downloads after promotion	Marketing campaign	App downloads

However, there could be multiple other factors impacting the effect variable apart from the cause variable. Hence, there are certain necessary conditions under which this relationship should be studied.

The concomitant or confounding variable effect

The confounding variable is an extraneous variable that changes with the independent variable and also affects the dependent variable. It comes with cause. In the presence of a concomitant variable, it becomes complex to study the true cause-and-effect relationship as it creates a bias in the study.

If we do not consider the concomitant variable(s) while studying the cause-and-effect relationship, the conclusion drawn is faulty, and the internal validity of the research is reduced.

Scenarioa	Independent Variable (Cause)	Dependent Variable (Effect)	Concomitant Variable
Sales increase after ad campaign	Advertising spend	Sales growth	Seasonal demand (e.g., festive season)
Employee productivity rises after training	Training program	Productivity	Employee motivation level
More online sales after website redesign	Website redesign	Sales increase	Simultaneous discount offers
Higher profits after hiring new manager	New leadership	Profitability	Market growth conditions
Increase in app downloads after promotion	Marketing campaign	App downloads

 Occurrence of the variables

The cause variable must occur before or simultaneously with the effect relationship. If the cause has not happened before the effect. It is impossible to study the relationship.

Cause (Happens First)	Effect (Happens Later)	Time Order Valid?
New marketing campaign launched	Increase in sales after 2 weeks	Ö
Employee training conducted	Improvement in performance next month	Ö
Price reduction announced	Immediate rise in demand	Ö
Increase in sales observed	Then advertising increased	X
High profits reported	Then new strategy introduced	X

Other factors are absent

This condition simply means that the effect has purely occurred because of the cause, and there is no other external factor involved. In other words, no alternative explanation can be given for the relationship under study. Even if the cause has occurred before the effect, the relationship does not stand valid. Some of the examples are given below

Observed Relationship	Possible Third Factor (Confounder)	Conclusion
Advertising ↑ → Sales ↑	Festive season demand	Not pure causation
Training → Productivity ↑	Employee motivation	Doubtful causation
Website redesign → Sales ↑	Discounts running simultaneously	Confounded
New manager → Profit ↑	Overall market growth	Not isolated effect
Social media campaign → Brand awareness ↑	Influencer promotions	Mixed effect

There are multiple ways in which the influence of other factors can be reduced significantly, such as

Method	Purpose
Controlled experiments	Keep other variables constant
Randomization	Distribute unknown factors evenly
Control group	Compare with untreated group
Statistical control (regression)	Isolate effect of variables
A/B testing	Test one variable at a time

Terminologies used in Experiment Design

Independent Variables

There are multiple terms used for independent variables, such as causal variables, explanatory variables, and treatments at times. These variables are manipulated or varied to understand the respective effect on the explained or the effect variable.

Dependent Variables

The dependent variable also called the explained variable, response variable or effect variable measure the effect of the treatment of independent variables. As mentioned above in various examples, these are post or simultaneous effects of the occurrence of the causal variable.

Test Unit

The treatment is applied to organisations, people, individuals, geographical areas, etc. These entities are known as test units. These test units undergo the treatment, and the effect of the treatment is measured through these units. For instance, if a group of people is given a specific training (treatment) the group of people will be called the test units.

Experiment

The researcher measures the effect of the cause variable on the effect variable by altering the cause variable while eliminating the intervention of extraneous variables. This process of measuring effect is referred to as an experiment in business research.

Extraneous variables

As mentioned earlier, some variables exist with cause variables and impact the cause-and-effect relationship. These variables reduce the internal validity of the experiment as explained in the next section. The researcher always tries to eliminate the effect of extraneous variables while studying the cause-and-effect relationship.

Validity of an Experiment

The internal and external validity of an experiment is regarded as the truthfulness and usefulness of the experiment. The internal validity focuses on whether the effect has purely occurred due to the cause, or if there is some intervention of an extraneous variable. The question asked in internal validity is, “Is it true?” For example, an A/B test showing a jump in sales due to higher discounts offered has higher internal validity than the experiment, in which sales increased after the discount, given that the festival season is also going on.

External validity focuses on the extent to which the experiment results can be generalised to real life situation. The question answered in external validity is, “Is it applicable? For example, if a lab experiment that shows a particular type of advertisement campaign works only for a specific type of people has low external validity compared to a field survey that explains the effect of advertising across different geographies.

Experimental Designs Classification

The following chart explains the classification of Experimental Designs

Pre-Experimental Design

These are the simplest experimental designs with no random selection of test units, no control groups, and a very limited control over the variables. These designs are often quick to implement, carry a low internal validity as they have very low control over external variables. These designs are also low on cost.

One Shot Case Study

In this design, the test units are not selected from the treatment group at random but by the researcher.

X	O
Treatment	Observation
A Social Media Campaign Launched by and Apparel Company	Sales have increased by 30%

Social Media Campaign (X) → Sales Measurement (O)

One-group pre-test-post-test design

No random selection of test units is done.

O1	X	O2
Observation 1	Treatment	Observation 2
Employee Productivity Measured	Training Given	Productivity is measured again

Employee Productivity Measured (O₁) → X (Training Given) → Employee Productivity Measured Again (O₂)

Static Group Comparison

In this design, two groups are selected non-randomly, and one group is given the treatment, while the other group is not given the treatment. The observations are taken for both groups at the end.

Group 1	X (Treatment)	O1 (Observation Group 1)
Group 2	-	O2 (Observation Group 1)

Example
Branch A	Branch A of the business is using CRM X (Treatment)	O1 (Observation Branch A)
Branch B	Branch B doesn’t use CRM	O2 (Observation Branch B)

Summary of Pre-Experimental Design

Design Type	Structure	Key Limitation
One-shot case study	X → O	No baseline, no control
One-group pre-post	O₁ → X → O₂	External factors affect results
Static group comparison	X → O₁ / O₂	No randomization

 

Quasi-Experiment Design

The term “quasi” means “almost” or “as if”. So, quasi experimental design means almost an experiment but not a complete experiment. It lacks the most important feature of an experiment, i.e., random assignment.

The quasi-experimental designs are used in business research where it is not feasible to select test units randomly, yet the cause-and-effect relationship is to be studied over time. These designs are used for Pricing Strategies, Marketing campaigns and policy impact studies.

The quasi-experiment designs can be studied as time series and multiple time series experiments. The time series quasi-designs study the change in the dependent variable over the period of time, and multiple time series show a comparative analysis for studying causality in depth.

Time Series Design

In a time, series quasi-design, the pre- and post-experiment observations are taken at multiple points in time. In the diagram below, O₁…. O₄ are the observations taken before the treatment, and O₅… O₈ are the observations taken after the treatment.

O₁ O₂ O₃ O₄ → X → O₅ O₆ O₇ O₈

Example

The company's monthly sales are tracked, and a price cut is introduced after April. There is a clear jump in sales figures after the price cut, and hence the treatment (price cut) effect on sales can be observed. Here, the independent variable, or cause variable, is the price cut, and the dependent variable, or effect variable, is sales.

Month	Sales
Jan	100
Feb	110
Mar	105
Apr	108
Price Cut (X)
May	140
Jun	150
Jul	145
Aug	155

The limitations of the designs are.

a.     There is no control group and,

b.     The extraneous factors, like seasonality and market trends, may affect the relationship under study

Multiple Time Series Design

As it can be observed from the diagram below, there are two groups in this design under study. One is the control group, and the other is expertimental or treatment group. The pre and post-observations are taken for both groups at the same point in time.

Treatment Group:   O₁ O₂ O₃ O₄ → X → O₅ O₆ O₇ O₈ 

Control Group:     O₁ O₂ O₃ O₄ →     O₅ O₆ O₇ O₈

Example

A company studies the sales of two regions North and South. The North region is considered the treatment group where the campaign is launched or the treatment is given. While the South region is considered the control group, where the campaign did not run, i.e., no treatment is given. Pre- and post-observations show a clear comparison of the two groups. The North region shows a jump in sales, while the South region shows stable growth in sales.

	North Region (Treatment)	South Region (Control)
Jan	100	98
Feb	105	100
Mar	102	99
Apr	104	101
	Campaign (X)
May	140	105
Jun	150	107
Jul	148	106
Aug	152	108

 Comparison of Time Series and multiple time series quasi experimental design

Feature	Time Series	Multiple Time Series
Groups	One	Two or more
Control group	No	Yes
Internal validity	Moderate	Higher
Ability to control external factors	Limited	Better

True Experimental Designs

True experimental designs are experiments in a true sense, as they use random assignment of subjects (R), have a control group, take observations (O), and use manipulation of the independent or cause variable, also known as treatment (X). True experimental designs are rigorous and scientifically valid designs; they study a clear cause-and-effect relationship. Due to their robust design, the internal validity of these designs is high.

There are various types of true experimental designs

Pretest- Posttest Control Group Design

The following diagrammatic structure explains the design. The test units for both groups are assigned at random. The observations O1 are taken for both groups pre-treatment. The treatment is given to the experimental group or test group. No treatment is given to the control group. The post treatment observations O2 are taken for both groups.

R  O₁  X  O₂

R  O₁     O₂

Example

The objective is to study the impact of the causal variable, i.e. the discount (X), on the effect variable, i.e. sales. The two test groups are randomly selected. Group A is treated as the experimental group, and Group B is treated as the control group. The pre and post observations in terms of sales level are taken for both groups.

Group	Before	Treatment	After
Group A	Sales level	Discount (X)	Sales increased
Group B	Sales level	—	No major change

The true effect of discounts can be studied here, as there are two groups available for comparison of the cause-and-effect relationship.

These designs have high internal validity, and it controls the initial difference.

Post Test Only Control Group Design

In this design, the random assignment is given to the groups, while the pre observation to the treatment is not taken. Only post-treatment observations are recorded for the experimental and controlled group. The following structure diagram provides a higher clarity.

R X  O₁

R      O₂

This design avoids the pre-test bias. It is simple and effective in nature.

Example

The objective is to know the impact of a new advertisement on the purchase intention of people. The experimental group is selected at random and shown the new advertisement, while the control group is still viewing the old advertisement. The observation after showing the new advertisement to the experimental group is collected.

Solomon Four – Group Design

As the diagram structure explains, there are four groups involved in this design. The test units are selected at random for all four groups. It is a mix of the first two designs as explained. Observations are taken pre and post treatment for first two groups, while for next two groups the observations are taken only post treatment.

R  O₁  X  O₂

R  O₁     O₂

R      X  O₂

R          O₂

If the researcher intends to check if the pre-testing influences the results, this design comes in picture. This is a robust design with a high internal validity. 

Statistical Experimental Design

These designs are statistical in nature and use statistical principles at the core. The statistical principles are used for controlling variability, improving accuracy and testing cause-effect relationship effectively.  These designs are widely used in business analytics, operations, marketing and quality control.

In these designs, the randomisation of the test and repeating the experiment to improve reliability and controlling external validation remain at the centre.  

Completely Randomised Design (CRD)

In this design, the test groups are assigned to different treatments. It is one of the simplest statistical designs. The following structure diagram shows the functioning of the experiment.

Example

Three different groups are shown three different advertisements then the purchase intentions are captured for the three groups to compare the advertisement impact.

Group	Ad Type
Group 1	Ad A
Group 2	Ad B
Group 3	Ad C

This design is suitable when there is homogeneity amongst the test units.

Randomize Bolock Design

In this design, different blocks of test units are created based on the basis of similar attributes. The randomisation is kept intact within each block.

Example

Testing advertisements across different age groups. The variation in this design is controlled due to a similar age group. This reduces external factor errors.

Block (Age Group)	Ad A	Ad B
18–25	✔	✔
26–40	✔	✔

Factorial Design

In this design, two or more factors are studied simultaneously. It largely examines interaction effects.

Example

We may wish to study the impact of different price levels and different advertisements at the same time on sales. The factorial design is most suitable in such conditions.

Testing:

• Price (Low/High)

• Advertisement (A/B)

Price	Ad Type	Outcome
Low	A	Sales
Low	B	Sales
High	A	Sales
High	B	Sales

We can always draw an insight into whether the price impact depends on the advertisement type.

Latin Square Design

This design controls two extraneous variables simultaneously. This is an effective technique to control or keep extraneous variables constant while studying the cause-and-effect relationship.

Example

A retail store may test the different layouts by keeping the location and time of day controlled.

Fractional Factorial Design

This design usage subset of the full factorial combination.

Example

Testing multiple website features without testing all combinations

Comparison of Statistical Experimental Designs

Design	Key Feature	Use Case
CRD	Full randomization	Simple experiments
RBD	Blocking	Control one external factor
Factorial	Multiple variables	Interaction effects
Latin Square	Two controls	Complex environments
Fractional Factorial	Reduced combinations	Cost/time efficiency

Comparison of all research Designs

Structure	Pre-Experimental Design	Quasi-Experimental Design	True Experimental Design	Statistical Experimental Design
Basic Nature	Simplest, exploratory	Semi-controlled, real-world	Fully controlled experiment	Advanced, statistically structured
Randomization	No	No	Yes	Yes (essential principle)
Control Group	Usually absent	Present (non-equivalent)	Present	Present
Number of Groups	One or two	Two or more	Two or more	Two or more (systematically designed)
Control over variables	Very low	Moderate	High	Very high (through design techniques)
Internal Validity	Low	Moderate	High	Very high
External Validity	Low	High (real-world setting)	Moderate	Moderate to high
Causality Strength	Weak	Moderate	Strong	Very strong
Handling of extraneous variables	Poor	Limited	Good	Excellent (blocking, factorial control)
Use of statistical tools	Minimal	Basic	Moderate	Advanced (ANOVA, DOE, etc.)
Complexity	Very simple	Moderate	High	Very high
Cost & Time	Low	Moderate	High	High
Typical Designs	One-shot, Pre-post	Time series, non-equivalent groups	RCT, Solomon design	CRD, RBD, Factorial, Latin Square
Business Application	Pilot testing, quick insights	Field studies (HR, marketing)	A/B testing, pricing experiments	Process optimization, Six Sigma, analytics
Example	Ad campaign → measure sales	Region A vs Region B campaign	Randomly test two ads	Test price × ad × layout combinations