# When the Tutor Leaves — reproduction package

Original exploratory secondary analysis by EuraStudy, 8 September 2026.
Not preregistered, not peer reviewed, and not a new experiment. The original
experiment and individual data belong to Bastani and colleagues. This package
contains our analysis code, aggregate results, and original scientific figures.

Article: https://eurastudy.com/research/when-the-tutor-leaves

## Source

Bastani et al. (2025), *Generative AI without guardrails can harm learning:
Evidence from high school mathematics*. PNAS 122(26), e2422633122.
https://doi.org/10.1073/pnas.2422633122

Public repository: https://github.com/obastani/GenAICanHarmLearning

- Pinned commit: `2f63dae1a01d51453826fe07ef5cf6678e339588`
- Input: `main_regressions/final_data.csv`
- SHA-256: `430055b4633eaaf87879bb38eb4b1c9425b91e1d7f529029dc1883e33d91df0a`
- Honors restriction checked against `main_regressions/main_analysis.R`.

The script refuses an input with a different checksum. Individual records and
student conversations are not redistributed here. Download the original data
from its authors; this package does not relicense their data or article.

## Run

Use Python 3.11 or later; the published results were generated with Python 3.13,
NumPy 2.3.5 and pandas 2.2.3. From a directory containing
`analyze.py` and `requirements.txt`:

```sh
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
git clone https://github.com/obastani/GenAICanHarmLearning.git source
git -C source checkout 2f63dae1a01d51453826fe07ef5cf6678e339588
python analyze.py --data source/main_regressions/final_data.csv --out reproduced
```

The analysis and figure rendering scripts do not use the network. Install and download
steps above do. In the EuraStudy repository, the scripts live under
`public/research/tutor-removal/`; quantitative tests are in
`docs/research/tutor-removal/test_analysis.py` and can be run with the same Python.

## Figure rendering

The article uses EuraStudy’s Hairline Atelier engine and the same monochrome
strokes, typography, and figure frames as the earlier Lab articles. There is no
hand-authored SVG or raster image in the article. `figure-specs.ts` derives typed
specifications from `results.json`; `render-figures.cjs` validates, renders, and
sanitizes them. The display namespaces SVG IDs separately for every instance.
No renderer code is sent to the browser.

From the EuraStudy repository root, after installing its pinned npm dependencies and Playwright’s Chromium browser:

```sh
node public/research/tutor-removal/render-figures.cjs
```

The renderer reads `public/research/tutor-removal/results.json`. To reproduce
figures from a fresh analysis, copy that analysis’s results to this path first.
The SVG downloads use the engine’s font fallback when the publication fonts are
not installed. PNGs use the same browser-resolved fallback. The command rebuilds all five figures in wide and narrow layouts: dark inline
SVGs in `figures/engine.json`, and light SVG/PNG exports for printing or reuse.
The shared engine lives under `src/lib/figures/` in the EuraStudy repository;
rendering the exact publication appearance requires that checkout. The Python
analysis is independently runnable from the downloads above.

The confidence intervals use engine-drawn schematic primitives with the exact
bootstrap endpoints. They are deliberately not converted to symmetric errors.
Figure 04 uses engine matrices; Figure 02 and Figure 05 use node graphs. The
previous Matplotlib script is retained as a legacy alternate export method,
but does not produce the figures currently displayed or downloaded. To run that
alternate script, separately install `matplotlib==3.10.6`.

## Unit of observation and fields

One row is one observed student-session pair, not one unique student.

| Field in original CSV | Use |
| --- | --- |
| `Student ID` | Identify repeated observations; count unique students |
| `Class` | Cluster, with arm assignment constant within class |
| `Year` | Grades 9, 10, and 11 |
| `Session` | Session 1, 2, 3, or 4 |
| `Part2Tot` | Practice score, already a fraction in [0, 1] |
| `Part3Tot` | Subsequent unaided exam score, already a fraction in [0, 1] |
| `Treatment arm` | `control`, `vanilla` (GPT Base), `augmented` (GPT Tutor) |
| `GPTBase`, `GPTTutor` | Cross-check arm indicators |
| `Honors` | Main sample retains `0`; all-students sensitivity also uses `1` |

Main sample: 2,899 pairs, 839 students, 44 classes. The source has 3,255 pairs,
943 students, 50 classes. Excluding honors removes 356 pairs and 104 students
in six classes. No score missingness or duplicate student-session key occurs
in this source. The file does not represent every initially enrolled student
at every scheduled session; no absent records are imputed.

## Estimands

With practice P and exam E, the primary conditional quantity is
`count(P >= .80 and E < .50) / count(P >= .80)`, calculated within assigned arm.
Exactly 80% qualifies; exactly 50% on the exam is not below the exam threshold.
Values within 1e-12 of a cutoff are treated as equal to it, using zero relative
tolerance. This avoids interpreting binary floating-point artifacts such as
0.4999999999999999 as scores below 50%. Means use the original unrounded values.
These cutoffs are analyst choices, not school passing or validated mastery rules.

Also report raw means, qualifying share among all pairs, and joint share among
all pairs. Repeated observed sessions receive equal weight in the main analysis.
The conditional comparison selects on a treatment-affected variable and is
descriptive, not a causal treatment effect. No regression or hypothesis test
replicating the original trial is claimed.

Uncertainty: 10,000 bootstrap draws of whole classes, independently within arm,
retaining all observed sessions whenever a class is drawn. Sample the original
number of classes per arm with replacement. Recompute ratios of pooled counts,
not equal-weight averages of classroom percentages. Generator: NumPy
`default_rng(20260908)`; arm order control, vanilla, augmented; sorted class IDs.
Intervals: linear-interpolated 2.5th and 97.5th percentiles. All primary
conditional replicates have nonzero denominators. This approximate bootstrap
does not reconstruct the original randomization strata or model transportability.

## Outputs and sensitivity

- `results.json`: provenance, sample accounting, means, rates, 95% intervals,
  descriptive contrasts, full threshold grid, subgroup and sample variants,
  leave-one-class-out minimum and maximum.
- `summary.csv`: one row per arm; scores and probabilities are fractions, not
  percentages. Interval columns end in `_ci_lower` or `_ci_upper`.
- `thresholds.csv`: practice thresholds .60, .70, .80, .90 crossed with exam
  thresholds .40, .50, .60, for each arm. Includes counts and denominators.
- `subgroups.csv`: each grade and each session. Small denominators remain visible.
- `sample-sensitivity.csv`: include honors; weight each session by the reciprocal
  of its student's observed session count; restrict to four-session attendees.
  In the equal-student-weight variant, integer count columns remain raw counts,
  while means and rates use weights. Do not divide the raw counts to reproduce
  that weighted rate. Weighting is applied before selecting high practice scores.
- `figures/`: five SVG plates and PNG exports, each in wide and mobile layouts.
  Figure 05 is a conceptual diagram. Other figures depict source counts or
  computed aggregates. No simulated learner outcomes appear.

All twelve threshold combinations were specified before the new aggregate
calculations, but after the source schema and original findings were inspected.
The four-session-attendee restriction is an additional exploratory sensitivity
check, not part of the initial plan. No sensitivity result is a causal comparison
of selected learners, and the cells are not independent replications.

The original experiment's practice and exam instruments differ. Their score
difference does not measure learning lost. Same-session testing does not establish
delayed retention or far transfer. No outcome here validates EuraStudy or another
current product. The article's prospective independent-check study is a proposal.