Metadata-Version: 2.4
Name: colm-final
Version: 0.1.0
Summary: Reproducible COLM 2026 strategic-reasoning analysis pipelines
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
Requires-Dist: httpx<0.28,>=0.27
Requires-Dist: numpy<2,>=1.26
Requires-Dist: openai<2,>=1.54
Requires-Dist: openpyxl<4,>=3.1
Requires-Dist: pandas<3,>=1.5
Requires-Dist: scipy<2,>=1.12
Provides-Extra: test
Requires-Dist: pytest<9,>=8.2; extra == "test"

# COLM 2026 stage-first code repository

This repository reorganizes the COLM 2026 price-game project into an active,
offline-testable pipeline while preserving the retained non-plotting historical
scripts and reported artifacts. It does **not** claim that every paper result
has been freshly regenerated.

## Citation

Jiannan Xu†, Yongkang Duan†, Jane Yi Jiang, Jiding Zhang. “Escaping the
Nash Trap: Structural Estimation and Alignment of Strategic Reasoning in Large
Language Models”, *Third Conference on Language Modeling (COLM 2026)*, 2026.

## Architecture

The active workflow has five layers:

1. `configs/` and `src/specs/`: experiment, game, order, Nash, and
   provenance definitions.
2. `src/generation/` and `src/extraction/`: condition-specific prompt
   packs, OpenRouter boundary, raw JSONL records, choice extraction, and
   aggregation.
3. `src/estimation/`: shared Level-0/Level-1/Level-2/Nash likelihood
   core, LLM cell fitting, human MAP fitting, and human finite mixtures.
4. `src/analysis/`: model-free Nash ratios, expected payoff, and
   repeated-game round aggregation.
5. `results/` and `provenance/`: fresh outputs, migrated reported artifacts,
   hashes, conventions, and known gaps.

`_archive/legacy_code/` retains the historical result-processing scripts but
excludes plotting-only files and bytecode caches. Active code does not import
it.

The root is intentionally limited to:

```text
colm_final/
├── configs/       # experiments, game matrices/Nash fields, standard prompts
├── src/           # active package
├── tests/         # unit and real-data offline smoke tests
├── data/          # raw/interim stages and processed inputs
├── results/       # fresh outputs and immutable reported results
├── provenance/    # manifest, scientific conventions, protocols, known gaps
└── _archive/      # retained historical Python sources only
```

## Environment

Python 3.12.13 is the locally verified interpreter; `pyproject.toml` accepts
Python `>=3.12,<3.13`. Point `$python312Base` at a Python 3.12 executable on the
local machine. Keep the virtual environment beside `colm_final`, because the
repository intentionally rejects generated tool directories inside its root:

```powershell
cd colm_final
$python312Base = "C:\path\to\Python312\python.exe"  # replace this path
& $python312Base -m venv ..\.venv_colm_final
$python312 = (Resolve-Path ..\.venv_colm_final\Scripts\python.exe).Path
& $python312 -m pip install -r requirements.txt
$env:PYTHONPATH = (Resolve-Path .\src).Path
function colm-final { & $python312 -m cli @args }
$env:PYTHONDONTWRITEBYTECODE = "1"
& $python312 -m pytest -p no:cacheprovider -q
colm-final validate --all
```

`requirements.txt` is the single dependency file and records the exact direct
runtime and test dependency versions used for verification. `httpx==0.27.2`
is pinned because the retained `openai==1.54.3` client is incompatible with
httpx 0.28 at client-construction time.

For a later shell session, restore the interpreter, `PYTHONPATH`, and the small
command wrapper explicitly:

```powershell
$env:PYTHONPATH = (Resolve-Path .\src).Path
$python312 = (Resolve-Path ..\.venv_colm_final\Scripts\python.exe).Path
function colm-final { & $python312 -m cli @args }
colm-final validate --all
```

## Data stages

- `data/raw/`: new per-run API JSONL records and retained exact historical raw
  workbooks; existing paths are never overwritten.
- `data/interim/`: extraction results with explicit extraction status.
- `data/processed/`: migrated, analysis-ready inputs.
- `results/generated/`: outputs from active commands.
- `results/reported/`: immutable historical outputs; active commands refuse
  to write here.

The processed main human inputs contain only:

- participant table: `participant_id`, `context`, `excluded`,
  `exclusion_reason`;
- choice table: `participant_id`, `game_id`, `choice`.

The main loader enforces one valid choice for every one of the 12 games and
returns 97 participants after excluding participant IDs 66 and 105: 44 in the
context condition and 53 in no-context.

## 1. Prompt/API, extraction, and aggregation

Generation requires five explicit selections:

- `--experiment` selects one experiment configuration and therefore its game
  set, permitted conditions, and permitted model IDs;
- repeatable `--condition` selects only the requested standard prompt
  condition;
- repeatable `--game` selects only the requested prompt game ID;
- repeatable `--model` selects only the requested configured model;
- `--protocol historical|standardized` selects the decoding rule and has no
  default;
- `--replicates N` controls only this invocation and defaults to 1.

The authoritative game files are `configs/games/original_3x3.json` and
`configs/games/revised_4x4.json`, plus the later 24-game catalog
`configs/games/new_game_3x3.json`. Each game records the payoff matrix,
`nash_metric_actions`, and `structural_nash_belief`. These two Nash fields are
deliberately distinct. The later catalog additionally records whether each
game belongs to the Nash-trap or no-trap group.

The shared prompt skeleton is `configs/prompts/base_game_prompt.txt`. The
condition instructions and request parameters are in
`configs/prompts/<experiment>.json`. The allowed selector values themselves
are in `configs/experiments/<experiment>.json`. See
`configs/prompts/README.md` for the full mapping.

### Decoding protocols

`configs/decoding_protocols.json` is the only active source of temperature
values. Temperature is deliberately absent from the prompt packs.

| Experiment         | `--protocol historical`                                              | `--protocol standardized`        |
| ------------------ | ---------------------------------------------------------------------- | ---------------------------------- |
| Early experiments  | omit`temperature`                                                    | non-Gemini`0.75`; Gemini `1.0` |
| `repeated_price` | non-Gemini`0.75`; Gemini omitted (expected provider default `1.0`) | non-Gemini`0.75`; Gemini `1.0` |
| `new_game`       | preserve ordinary/Gemini/Qwen condition-specific settings              | non-Gemini`0.75`; Gemini `1.0` |

For `new_game`, historical ordinary-model reasoning omits temperature,
ordinary no-reasoning uses `0.75`, Gemini uses `1.0`, and the special Qwen
on/off runs use `0.6`/`0.7`. The standardized protocol applies `0.75` to every
non-Gemini model, including Qwen. `level0_anchors` reuses existing choices for
analysis and rejects generation.

Preview one exact request without an API key or network call:

```powershell
colm-final generate `
  --experiment baseline_3x3 `
  --condition context_direct `
  --game 1 `
  --model openai/gpt-5.2 `
  --protocol historical `
  --replicates 1 `
  --dry-run `
  --output data/raw/baseline_game1_preview.jsonl
```

Preview one later 24-game request with its recorded historical decoding rule:

```powershell
colm-final generate `
  --experiment new_game `
  --condition no_reasoning `
  --game 1 `
  --model openai/gpt-5.4 `
  --protocol historical `
  --dry-run `
  --output data/raw/new_game_game1_preview.jsonl
```

For the special Qwen models, use `reasoning` rather than
`reasoning_low`/`reasoning_high`. To make a new cross-model rerun, change the
protocol explicitly to `--protocol standardized`; never reuse an output path
from a historical-protocol run.

For one complete original-game count matrix, select all 12 prompt games
explicitly. First review the dry-run output; then remove `--dry-run`, set the
key, and add `--confirm-live`:

```powershell
$env:OPENROUTER_API_KEY = "<key>"
colm-final generate `
  --experiment baseline_3x3 `
  --condition context_direct `
  --game 1 --game 7 --game 2 --game 8 --game 3 --game 9 `
  --game 4 --game 10 --game 5 --game 11 --game 6 --game 12 `
  --model openai/gpt-5.2 `
  --protocol historical `
  --replicates 30 `
  --output data/raw/baseline_context_direct_gpt52.jsonl `
  --model-revision "<provider revision>" `
  --confirm-live

colm-final extract `
  --input data/raw/baseline_context_direct_gpt52.jsonl `
  --output data/interim/baseline_context_direct_gpt52.csv `
  --game-set original_3x3

colm-final aggregate `
  --input data/interim/baseline_context_direct_gpt52.csv `
  --output results/generated/baseline_context_direct_gpt52_counts.csv `
  --game-set original_3x3
```

The aggregate output uses `prompt_game_id`; `fit-llm` accepts that active
pipeline column as well as the historical numeric `game_id` column.

Generation is the only network-enabled command. Live mode requires all of:

- `OPENROUTER_API_KEY`;
- explicit `--condition`, `--game`, and `--model` selectors;
- explicit `--protocol historical` or `--protocol standardized`;
- an explicit `--output`;
- `--confirm-live`.

`--dry-run` requires no key and writes the exact prompt, resolved request
parameters, prompt provenance, protocol, policy id, temperature
mode/value/note, and null response/timestamp fields. A provider-default
temperature is omitted from the
actual `request_parameters`. No live request is made by tests or repository
validation. Historical raw responses are not invented from aggregate counts.

## 2. LLM fitting

An LLM count input must contain one model-condition cell with exactly one row
per prompt/workbook game:

```text
prompt_game_id,Choice_A,Choice_B,Choice_C
```

Historical files using numeric `game_id` remain supported.

The original 3x3 order is `1,7,2,8,3,9,4,10,5,11,6,12`; the revised 4x4 order
is `1,13,2,14,...,12,24`; the later `new_game_3x3` order is `1,...,24`.
Missing, duplicate, extra, or four-row historical inputs fail before
optimization.

```powershell
colm-final fit-llm `
  --input results/generated/one_model_condition_counts.csv `
  --output results/generated/fit_uniform.json `
  --game-set original_3x3 `
  --anchor uniform

colm-final fit-llm `
  --input results/generated/one_revised_model_condition_counts.csv `
  --output results/generated/fit_maximax.json `
  --game-set revised_4x4 `
  --anchor maximax
```

The shared core returns `gamma`, type log-likelihoods, normalized posterior,
MAP type, objective value, and optimizer status. Uniform and maximax use the
same game definitions and likelihood implementation. The Python API also
accepts an explicit `linear` Level-0 action distribution; recovered historical
linear-weight results remain under legacy/reported provenance.

## 3. Human fitting

Participant-level MAP fitting:

```powershell
colm-final fit-human-map `
  --choices data/processed/baseline_3x3/human_choices.csv `
  --participants data/processed/baseline_3x3/human_participants.csv `
  --output results/generated/human_map.json `
  --anchor uniform
```

Population finite-mixture fitting:

```powershell
colm-final fit-human-mixture `
  --choices data/processed/baseline_3x3/human_choices.csv `
  --participants data/processed/baseline_3x3/human_participants.csv `
  --output results/generated/human_mixture.json `
  --anchor uniform
```

The LLM and human entry points share the same type probabilities, likelihood,
posterior, Nash belief, and Level-k recursion. Only the observation aggregation
differs.

## 4. Other analyses

```powershell
colm-final nash-ratio `
  --input data/interim/choices.csv `
  --output results/generated/nash_ratio.json `
  --game-set original_3x3

colm-final payoff `
  --matrix matrix.json `
  --llm "0.2,0.3,0.5" `
  --human "0.4,0.4,0.2" `
  --output results/generated/payoff.json

colm-final repeated-summary `
  --input data/interim/repeated_rounds.csv `
  --output results/generated/repeated_summary.csv `
  --group-columns "model,condition" `
  --round-column round
```

Nash ratio is deliberately model-free and uses the declared paper action set.
Repeated-game aggregation retains round identifiers and never calls the
one-shot type estimator.

## Scientific conventions

The original paper's model-free Nash labels and the payoff-derived structural
Nash beliefs disagree for original Games 1/7 and 2/8. They are separate fields,
not aliases. Revised 4x4 games are the default basis for Level-0 anchor
robustness. Details are in `provenance/scientific_conventions.md`.

## Provenance and result status

`provenance/source_manifest.csv` contains one row per migrated file with source
and destination paths, SHA-256 values, size, source modification time,
experiment, stage, status, and transformation. It currently covers 106 exact
or explicitly transformed migrations:

- 41 unchanged retained non-plotting legacy Python files under
  `_archive/legacy_code/`;
- 41 unchanged reported-result files;
- 22 processed inputs, including transformed privacy-minimized human tables;
- 2 exact `new_game` raw-response workbooks.

Treat status labels literally:

- generated/recomputed/exact-after-rounding: recoverable by the stated path;
- reported transcription/reconstructed: paper-derived historical evidence;
- diagnostic: a check, not a published-result reproduction;
- missing: not available and not fabricated.

See `provenance/known_gaps.md` before making any reproduction claim.

## Verified offline checks

The repository test suite covers:

- all three game sets and their workbook order;
- the historical `(4,3)` versus expected `(12,3)` failure;
- separate model-free and structural Nash fields;
- uniform, maximax, and linear probability normalization;
- the symmetric-column maximax orientation regression;
- extraction, aggregation, and injected fake transport;
- explicit generation selectors and key-free request previews;
- direct aggregate-to-fit `prompt_game_id` compatibility;
- 97-person main data loading and privacy-minimized schemas;
- individual and mixture fitting;
- original 3x3 and revised 4x4 real-workbook LLM smoke fits;
- later 24-game checked-in-choice fit smoke tests;
- revised 4x4 uniform and maximax smoke fits;
- real human MAP/mixture and intervention-payoff smoke tests;
- all manifest destination hashes;
- protected reported/processed/archive directories;
- CLI discovery, invalid-selector rejection, and missing-key network boundary;
- absence of plotting implementations and plotting dependencies.

Third-party deprecation warnings from the pinned openpyxl/dateutil stack
do not indicate a test failure. No internet access, live API request, paper PDF
rewrite, or missing-result reconstruction is part of verification.
