Skip to content

Commit

Permalink
Merge pull request #571 from AutoResearch/556-doc-check-and-update-us…
Browse files Browse the repository at this point in the history 
…er-guide

docs: updated docs for installation, theorists, experiment runners, and experimentalists
  • Loading branch information
@chadcwilliams
chadcwilliams authored Oct 31, 2023
2 parents 5ce58b0 + 3ec545a commit 8e93185
Show file tree
Hide file tree
Showing 4 changed files with 58 additions and 48 deletions.
10 changes: 5 additions & 5 deletions docs/experiment-runner/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Experiment Runner Overview

AutoRA includes tools for running synthetic and real-world experiments.
``autora`` includes tools for running synthetic and real-world experiments.

![Overview](../img/experiment_runner.png)

For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between AutoRA and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants.
For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between ``autora`` and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants.

Experiment runner tools may take in information about new experimental conditions or entire experiments. The following table includes the various experiment runner tools currently implemented in AutoRA.
Experiment runner tools may take in information about new experimental conditions or entire experiments. The following table includes the various experiment runner tools currently implemented in ``autora``.

| Name | Links | Description |
|---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Synthetic | [Package](https://pypi.org/project/autora-synthetic/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/synthetic/) | A compendium of ground-truth models across psychology, psychophysics, behavioral economics, and other disciplines. |
| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between AutoRA and an experiment on Firebase. |
| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using AutoRA. |
| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between ``autora`` and an experiment on Firebase. |
| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using ``autora``. |
| Firebase-Prolific (experiment runner) | [Package](https://pypi.org/project/autora-experiment-runner-firebase-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/firebase-prolific/) | An experiment runner that combines the Firebase experimentation manager and the Prolific recruitment manager to automatically run human behavioral experiments. |

45 changes: 16 additions & 29 deletions docs/experimentalist/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,41 +1,28 @@
# Experimentalist Overview

The primary goal of an experimentalist is to identify experiments that yield
scientific merit. AutoRA implements techniques for automating the identification
scientific merit. ``autora`` implements techniques for automating the identification
of novel experiments.

An experiment consists of a series of **experimental conditions** $\vec{x} \in X$.
The experimental variables manipulated in each experimental condition
are defined as **factors**, and the values of each variable to be sampled
in the experiment are defined as **levels** of the corresponding **factors**.
As an example, consider a visual discrimination tasks in which participants are presented
with two lines of different lengths, and are asked to indicate which line is longer.
There are two factors in this experiment: the length of the first line and
the length of the second line. Instances of the two line lengths
(e.g., 2.0 cm for the first line and 2.1 cm for the second line)
can be considered levels of the two factors, respectively. Thus, *an experimental condition is a vector of values that
corresponds to a specific combination of experiment levels $x_i$,
each of which is an instance of an experiment factor.*

Experimentalists in AutoRA serve to identify novel
experimental conditions $\vec{x} \in X$, where $x_i$ corresponds
to the level of an experimental factor $i$.
An experiment consists of a series of **conditions** $\vec{x} \in X$. The variables manipulated in each condition
are defined as **independent variables**. As an example, consider a visual discrimination task in which participants are presented
with two lines of different lengths, and are asked to indicate which line is longer. There are two independent variables in this experiment: the length of the first line and the length of the second line, which each have values (e.g., 2.0 cm for the first line and 2.1 cm for the second line). Thus, *a condition is a vector of values that corresponds to a specific combination of values of the independent variables $x_i$.*

![Overview](../img/experimentalist.png)

Experimentalists may use information about candidate models $M$ obtained from a theorist,
experimental conditions that have already been probed $\vec{x}' \in X'$, or
respective dependent measures $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented
in AutoRA.
conditions that have already been probed $\vec{x}' \in X'$, or
respective observations $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented
in ``autora``.

| Name | Links | Description | Arguments |
|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|
| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/core/docs/experimentalists/sampler/random/) | An experimentalist with pooling and sampling functionality. The pooler creates combinations of conditions from lists of discrete values using random selection; the sampler draws from a pool of conditions without replacement using uniform random sampling. | |
| Novelty | [Package](https://pypi.org/project/autora-experimentalist-sampler-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/novelty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric applied to existing conditions $\vec{x} \in X$. | $X'$ |
| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/uncertainty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to model uncertainty, which can be calculated in three different ways. | $M$ |
| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/model-disagreement/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric between theorist models, $P_{M_{i}}(\hat{y}, \vec{x}')$. | $M$ |
| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | An experimentalist with pooling and sampling functionality which generates and samples from novel experimental conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ |
| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | A sampler that uses a mixture of specified sampling strategies to identify novel experimental conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ |
| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | A sampler which returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ |
| Leverage | [Package](https://pypi.org/project/autora-experimentalist-sampler-leverage/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/leverage/) | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ |
| Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | A sampler that uses a pairwise distance metric to compare and select new experimental conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ |
| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/random/) | An experimentalist with pooling and sampling functionality. The pooler creates combinations of conditions from lists of discrete values using random selection; the sampler draws from a pool of conditions without replacement using uniform random sampling. | |
| Novelty | [Package](https://pypi.org/project/autora-experimentalist-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/novelty/) | Identifies conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric applied to existing conditions $\vec{x} \in X$. | $X'$ |
| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/uncertainty/) | Identifies conditions $\vec{x}' \in X'$ with respect to model uncertainty, which can be calculated in three different ways. | $M$ |
| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/model-disagreement/) | Identifies conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric between theorist models, $P_{M_{i}}(\hat{y}, \vec{x}')$. | $M$ |
| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/falsification/) | An experimentalist with pooling and sampling functionality that generates and samples from novel conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ |
| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/mixture/) | Uses a mixture of specified sampling strategies to identify novel conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ |
| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/nearest-value/) | Returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ |
| Leverage | [Package](https://pypi.org/project/autora-experimentalist-leverage/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/leverage/) | Identifies conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M, X', Y'$ |
| Inequality | [Package](https://pypi.org/project/autora-experimentalist-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/inequality/) | Uses a pairwise distance metric to compare and select new conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ |
Loading

0 comments on commit 8e93185

Please sign in to comment.