A hands-on course introducing modern approaches to experimental design, from traditional factorial methods to automated closed-loop experimentation using active learning and information theory.
This course teaches you how to design efficient experiments that maximize information gain while minimizing resources (time, participants, trials). You'll progress from classical experimental design methods to cutting-edge automated experimentation using the AutoRA framework.
By the end of this course, you will be able to:
- Design and implement synthetic experiments for testing experimental design strategies
- Apply different sampling strategies (random, Latin hypercube, factorial)
- Use information-theoretic principles to optimize experimental designs
- Implement uncertainty-based active learning for intelligent sampling
- Build automated closed-loop experiments using AutoRA
- Evaluate and compare different experimental design strategies
-
Session 1: Introduction to Experimental Design
- Definition and terminology
- Examples from cognitive science (Stroop, 2AFC, Multi-Armed Bandit)
- Limitations and resource constraints
-
Session 2: Simulation-Based Experimentation
- Implementing synthetic experiments
- Ground truth functions and noise
- Model recovery and validation
- Tutorial: syntheticexperiments.ipynb
-
Session 3: Random Sampling Methods
- Normal, uniform, and Latin hypercube sampling
- Coverage analysis and comparison
- When to use each strategy
- Tutorial: randomsampling.ipynb
-
Session 4: Factorial Experimental Design
- Full factorial designs
- Main effects vs. interactions
- Factorial explosion and fractional designs
- Tutorial: factorialdesign.ipynb
-
Session 5: Introduction to AutoRA & Closed-Loop Experimentation
- The AutoRA framework (Experimentalist → Experiment Runner → Theorist)
- State management and workflow
- Simple experimentalists (grid, random)
- Building your first closed-loop experiment
- Tutorial: autora_intro.ipynb
-
Session 6: Information Theory Basics
- Entropy and information content
- Conditional entropy and mutual information
- Application to experimental design
- Tutorial: informationtheory.ipynb
-
Session 7: Uncertainty-Based Active Learning
- Uncertainty sampling principles
- AutoRA uncertainty experimentalist
- Comparing random vs. uncertainty-based sampling
- Tutorial: autora_uncertainty.ipynb
-
Session 8: Advanced Active Learning
- Model disagreement and ensembles
- AutoRA disagreement experimentalist
- Comparing uncertainty vs. disagreement strategies
- Tutorial: autora_advanced.ipynb
- Student paper presentations (40% of grade)
- Group project kickoff
- Q&A and consultation
- Python 3.8 or higher
- Basic knowledge of Python programming
- Familiarity with NumPy and Matplotlib
- Basic understanding of statistics and machine learning
- Clone the repository
git clone https://github.com/yourusername/CourseOptimizingExperimentalDesign.git
cd CourseOptimizingExperimentalDesign- Create a virtual environment (recommended)
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate- Install required packages
pip install -r requirements.txt- Install AutoRA with experimentalists
pip install -U "autora[experimentalist-uncertainty]"
pip install -U "autora[experimentalist-inequality]"
pip install -U "autora[experimentalist-novelty]"
# Or install all experimentalists at once:
# pip install -U "autora[all-experimentalists]"- Verify installation
python -c "import autora; import sklearn; import torch; print('All packages installed successfully!')"See installation_instructions.md for detailed troubleshooting.
Located in ~/Downloads/pdf/:
01_Introduction_OED.pdf- Introduction to experimental design02_RandomSamplingMethods.pdf- Random sampling strategies03_FactorialExperimentalDesign.pdf- Factorial designs04_InformationTheory.pdf- Information theory basics05_AutoRA.pdf- AutoRA framework and active learning
All tutorials are Jupyter notebooks in the tutorials/ directory:
| Tutorial | Topic | Duration |
|---|---|---|
syntheticexperiments.ipynb |
Synthetic experiment setup | 60 min |
randomsampling.ipynb |
Random sampling comparison | 45 min |
factorialdesign.ipynb |
Factorial designs | 30 min |
autora_intro.ipynb |
AutoRA basics & closed-loop | 60 min |
informationtheory.ipynb |
Information theory concepts | 45 min |
autora_uncertainty.ipynb |
Uncertainty-based sampling | 60 min |
autora_advanced.ipynb |
Disagreement & ensembles | 60 min |
resources/synthetic.py- Synthetic experiment utilitiesresources/regressors.py- Neural network modelsresources/sampler.py- Sampling strategies
Part 1: Paper Presentation (40%)
- Due: January 25, 2025
- Choose a paper on optimal experimental design or active learning
- Present: content, applicability, benefits, limitations (20 min + 10 min discussion)
Part 2: Python Implementation (60%)
- Due: February 25, 2025
- Apply the presented approach to the 2AFC demo experiment
- Compare efficiency/effectiveness vs. baseline random sampling
- Document thoroughly with comments, text sections, and plots
Bonus: Paper Discussion (up to 10%)
- Active participation in paper presentation discussions
- Mandatory course attendance
- All group project components submitted on time
- Clear documentation and reproducible code
The course uses a 2-Alternative Forced Choice experiment as a running example:
Task: Participants view a grid of colored tiles and identify which color is more prevalent
Controllable Factors:
ratio: Proportion of blue vs. orange tiles (0 = all orange, 1 = balanced)scatteredness: Spatial randomness of tiles (0 = segregated, 1 = random)
Observations:
- Response time (continuous)
- Accuracy (binary)
This experiment allows testing various design strategies in a controlled setting.
- Factors: Types of stimuli (e.g., ratio, scatteredness)
- Levels: Stimuli intensity values
- Treatment: Specific combination of factor levels
- Design Space: All possible treatment combinations
- Run: Single execution of an experimental unit
- Sample Size: Total number of runs
- Query Strategy: Method for selecting informative samples
- Uncertainty Sampling: Select samples with highest prediction uncertainty
- Disagreement Sampling: Select samples where models disagree most
- Closed-Loop: Automated cycle of experimentation → modeling → new experiments
- Entropy: Expected information content
- Conditional Entropy: Remaining uncertainty given conditions
- Mutual Information: Shared information between variables
- Musslick et al. (2024). AutoRA: Automated Research Assistant for Closed-Loop Empirical Research. Journal of Open Source Software.
- Musslick et al. (2023). An Evaluation of Experimental Sampling Strategies for Autonomous Empirical Research in Cognitive Science. Proceedings of CogSci.
- Settles, B. (2009). Active Learning Literature Survey. University of Wisconsin-Madison.
- AutoRA - Automated research assistant
- SweetPea - Experimental design constraints
- PsychoPy - Psychology experiment builder
Found an issue or have suggestions? Please:
- Check existing issues
- Open a new issue with details
- Or submit a pull request
Instructor: [Your Name] Email: [your.email@university.edu] Office Hours: [Days/Times]
This course material is licensed under [LICENSE TYPE]. Feel free to use and adapt for educational purposes with attribution.
This course builds on:
- The AutoRA framework by Musslick et al.
- modAL active learning library
- Examples from cognitive science literature
- Student feedback from previous offerings
Last Updated: January 2025 Version: 2.0 (AutoRA integration)