# Agentic Env Index: for agents - Pick sources from `meta/catalog.parquet` (one row per source, about 50 KB; do not download `meta/` whole, `meta/duplicates.parquet` is about 4 GB). Columns: `source`; `domain`, a list of codes D1 to D10 or leaves such as D3.5, or `multi`, `unclassified`; `form`, such as `test-suites` or `spreadsheets-with-answers`; `units` (a unit is one test case or one workbook; a raw corpus has one unit per row), `answers` (units that have an answer, the expected results kept apart) and `files` (files of every role); `language` (the language codes of the rows' labels, what the default profile filters on; `not recorded` when a row has none); `contamination.level` (`not scanned` when no comparison is recorded for the source); `license.ship`; `pii.level`; `bytes`. The vocabulary is `meta/vocabulary.json`: every domain code with its name, what it covers and its parent, and every form with what it means. - Task-ready sources are collections whose rows carry the roles unit, companion and answer (`answers` above 0, or `units` below `files`): each unit comes with its expected results. Every other source is a raw corpus to mine, one unit per row. - Fetch with one command: `uv run https://huggingface.co/datasets/Battam/agentic-env-index/resolve/main/tools/fetch.py --domain D3 --form spreadsheets-with-answers --sample 3 --out ./data`. Without uv: `pip install duckdb`, download `tools/fetch.py` and run it with `python`. Select with `--source`, `--domain`, `--form`, `--tag`, `--ext`; `--sample N` takes N units per source, `--dry-run` only counts and writes `_meta/fetch_manifest.jsonl` with one line per file of the selection (status selected, removed with the criterion in removed_by, or not_taken), as many lines as the files the summary counts. - It pins the commit it reads and prints it; pass `--revision ` to get the same files later. A name that does not exist stops with exit code 2 and the closest names. - The default profile removes duplicate copies, rows that matched a benchmark (copies of benchmark data, collections awaiting a ruling), rows not compared with the benchmark list yet (counted apart, with their own reason), files whose license does not allow shipping (license.ship yes and conditional stay), rows recorded in a language other than English, personal data (catalog pii.level present or suspected, rows marked sensitive), and files above 20 MB. Before downloading it prints how many units each criterion removed; `--all` turns the profile off, and each criterion has its own option (`--keep-duplicates`, `--allow-benchmark`, `--allow-not-compared`, `--any-license`, `--any-language`, `--allow-personal-data`, `--any-size`; `--allow-benchmark` also keeps rows not compared yet). This release records no comparison with the benchmark list: none of its 196 sources has been compared. The profile also removes every row whose data file was not compared (`--allow-benchmark` keeps them, unchecked). - Output: `./data///...` holds the units and their companion files (README, LICENSE); `./data/answer//...` holds the answers and expected results, apart from the material; `./data/_meta/manifest.jsonl` has one line per file (source, path, url, sha256, bytes, spdx, licence_url, publisher, retrieved, answer, origin_kind, leaf, role, unit_group, answer_for); `./data/_meta/sizes.tsv` one line per source. Every file is checked against the sha256 the index records. A unit stored inside a container (such as a disk image inside a zip) is written as the unit itself: fetch.py takes the member out and checks it against the unit's recorded sha256, so no second tool is needed; only units marked expand (archives meant to be used as folders) are unpacked by `tools/to_pool.py`. - Never put `answer/` into the environment the agent works in: the answers are there to check its result, not to be read by it. Each kind of answer, and what else to keep out: - Answer files (role `answer`; an answer in JSON names its kind in the field `kind` or `answer_kind`): the expected result of one unit, such as an output, a value or a verdict. Keep out of the environment also any copy of it, and any file that holds the same result. - `key_formula_values` (spreadsheets with answers): the answer is the value of the workbook's key formula cells, and the published workbook already holds those values, cached, so a workbook put in as fetched hands the agent its answer. Strip the cached values first: in .xlsx and .xlsm the `` of every formula cell (`xl/worksheets/*.xml`), the chart caches (`c:numCache` and `c:strCache` in `xl/charts/*.xml`), the pivot caches (`xl/pivotCache/`, definitions and records) and the external link caches (`xl/externalLinks/`); in .ods the value attributes (`office:value` and the like) and the text of formula cells; in .xls the cached result of every FORMULA record and its STRING record. - Reported results of a package (`./data/answer///reported.json`, for reproductions and evaluation results): the numbers the authors published. Keep out also the files of the package that hold the same numbers, such as result tables and logs. - Expected results inside the unit (task-ready sources without answer files, such as test suites, detection rules with their tests and rules with worked examples): the unit carries its own check (expected outputs, expected findings, the worked result). Keep those files or parts out of the environment and use them only to check the agent's result. - The form of a task built from these files (such as a Terminal-Bench style task with `environment/`, `solution/` and `tests/`): - `solution/solve.sh` is in bash, with the work spread over top-level commands (8 or more), each command that changes the environment on its own line; not wrapped in a function, `main`, a large heredoc or a single `python` call. - The task runs without network; its data is inside `environment/`. - Written files carry no timestamps, random values or host names; versions are pinned. - No background service that has to outlive a step, no database service. - Tests check the final state, not only that files exist.