earth2studio-create-datasource

Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores. Do NOT use for…

Create and Validate Data Source

Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to Earth2Studio's async data fetching infrastructure — from analysis through implementation, testing, validation, and PR submission.

Prerequisites

Earth2Studio dev environment with uv (uv run python must work)
Git configured with fork (origin) and upstream (upstream) remotes
Access to the target remote data store (credentials if private)
Python 3.10+

⚠️ Credential Safety: If the remote store requires authentication, credentials must be passed via environment variables only. Never hard-code secrets in source files. Before any git commit, verify no credentials are staged (check git diff --cached for API keys, tokens, or passwords).

Workspace

Use the directory containing pyproject.toml. For Harbor evals, write to /workspace/output/ preserving paths. Never read evals/targets/.

Instructions

Python Environment: Always use uv run python or the local .venv. Never use the system Python directly.

Follow every step in order.

[CONFIRM] gates: Step 1 (Source Type), Step 12 (Sanity-Check Plots), and Step 13 (Ready to Submit) require explicit user approval before proceeding. All other [CONFIRM] markers are advisory — present decisions inline and proceed without blocking.

Deliverables first: Write the source file and test file (Steps 6–7) before extended exploration, documentation, registration, CHANGELOG, or PR work. Skip Steps 8–14 when the user asks for implementation only.

Before you finish: Run verification commands in the repo root so results appear in the session log:
uv run pytest test/data/test_<source>.py -x
make format && make lint
Be concise: Avoid long architecture reports; summarize decisions in a few sentences and move on to file writes.

Hangs or User Feedback If agent becomes stuck or user provides a correction during this skills use, conservatively review relevant part of the skill and improve. Be concise.

One source type per invocation. Invoke again for companion types.

Reference Files

Load these on demand during the relevant steps:

File	Content	Load at
`references/reference-implementation.py`	Obstore-backed source skeleton with FILL comments	Steps 6–10
`references/reference-lexicon.py`	Lexicon skeleton with FILL comments	Step 4
`references/testing-guide.py`	Test skeleton with FILL comments	Step 11
`references/validation-guide.md`	Plot templates, PR body template, Greptile handling	Steps 12–14 (optional, for templates)

Workflow Overview

Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review

Step 0 — Obtain Remote Data Store Reference

If $ARGUMENTS is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

Please provide a URL, API documentation link, or description of the remote data store. This will be used to understand storage format, access pattern, variable inventory, temporal/spatial resolution.

Step 1 — Determine Source Type

Protocol	Returns	Has `lead_time`?	Use
DataSource	`xr.DataArray`	No	Gridded analysis/reanalysis
ForecastSource	`xr.DataArray`	Yes	Gridded forecast
DataFrameSource	`pd.DataFrame`	No	Sparse/station obs
ForecastFrameSource	`pd.DataFrame`	Yes	Sparse forecast obs

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

[CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

Step 2 — Examine Remote Store & Propose Dependencies

Analyze: storage backend, file format, authentication, access pattern, temporal/spatial resolution, variable inventory.

Prefer Earth2Studio obstore helpers:

Backend / format	Preferred	Avoid
S3/GCS/HTTP object files	`obstore_store_from_url` + `obstore_fetch_to_cache`	raw SDK clients
Cloud Zarr	`obstore_zarr_store`	hand-rolled fsspec/zarr stores
Azure/object auth	`obstore.store.from_url` via shared helpers	provider SDKs directly
HuggingFace	`huggingface_hub` (core dep)	custom scripts

Only fall back to dedicated libraries when obstore/shared helpers cannot access the store.

Check pyproject.toml — only propose packages not already present. Core deps include: obstore, zarr, netCDF4, h5py, pygrib, huggingface-hub, pandas, pyarrow.

[CONFIRM — Dependencies & Access Pattern]

Present: backend, obstore helper/store URL, new packages (with license), auth method.

Step 3 — Add Dependencies

Load references/reference-implementation.py from Step 6 through Step 10. Load references/reference-lexicon.py during Step 4.

If new packages needed:

Present the package name(s), version constraints, and licenses to the user
Confirm the user approves adding them before proceeding
uv add --extra data <package>
uv lock
Add optional dependency imports using OptionalDependencyFailure pattern

Step 4 — Create Lexicon Class

Create earth2studio/lexicon/<source_name>.py with:

metaclass=LexiconType
VOCAB: dict[str, str] mapping E2S names → remote keys
get_item(cls, val) returning tuple[str, Callable]
Use :: separator for structured keys
Use references/reference-lexicon.py as the starting template

Map remote variables against E2STUDIO_VOCAB (282 entries in earth2studio/lexicon/base.py).

[CONFIRM — Lexicon & Variable Mapping]

Present: class name, key format, full mapping table, modifiers, reference URL.

Step 5 — Update E2STUDIO_VOCAB / SCHEMA (if needed)

New vocab: surface = descriptive abbrev; pressure = {name}{level}
New schema fields: DataFrame sources only, check E2STUDIO_SCHEMA

[CONFIRM — Vocabulary & Schema Updates]

Skip if no updates needed.

Step 6 — Create Skeleton Data Source File

Use references/reference-implementation.py as the starting template.

Follow canonical method ordering:

Class constants
SCHEMA
__init__
_async_init
__call__
fetch
_create_tasks
fetch_wrapper
fetch_array
_validate_time
Helpers
cache property
available classmethod

Use async task dataclass pattern for parallel execution.

[CONFIRM — Skeleton]

Present: class name, file path, skeleton code, task dataclass.

Step 7 — Implement the Data Source & Tests

The test file is a co-equal deliverable. Create test/data/test_<filename>.py alongside the source. Add test_<source>_call_mock for async sources.

Sync sources: Use prep_data_inputs/prep_forecast_inputs, direct __call__.

Async sources: See references/reference-implementation.py for required patterns: _sync_async, gather_with_concurrency, async_retry, obstore_fetch_to_cache, obstore_zarr_store for Zarr, try/finally cleanup. Constructor params: cache=True, verbose=True, async_timeout=600, async_workers=None, retries=3. DataFrame sources add time_tolerance.

Step 8 — Register the Source

earth2studio/data/__init__.py — alphabetical import
earth2studio/lexicon/__init__.py — alphabetical import
Verify pyproject.toml deps

Step 9 — Update Documentation

Add to correct RST file (datasources_analysis.rst / _forecast.rst / _dataframe.rst)
Class docstring: Parameters, Warning (download size), Note (reference URLs), Badges (last)
All public methods: NumPy-style docstrings

Step 10 — Update CHANGELOG.md

Add entry under the current unreleased version. See references/reference-implementation.py REGISTRATION CHECKLIST for the format.

One line per source. Do NOT add separate lexicon entries.

Step 11 — Verify Style & Expand Tests

Run make format && make lint && make license. Load references/testing-guide.py for test skeletons. Required tests: test_<source>_fetch (slow), _cache (slow), _call_mock, _exceptions, _available. Target 90%+ coverage with --slow.

[CONFIRM — Tests]

Present test file, functions, coverage.

Step 12 — Validate Variables & Sanity-Check

Validate all lexicon vars against real data (run script, do NOT commit)
Remove variables with < 10% valid data
Create sanity-check plot (gridded or sparse template)
Tell user the plot path and ask for visual confirmation

[CONFIRM — Sanity-Check Plots]

User MUST visually inspect plots. Do not proceed without confirmation.

Step 13 — Branch, Commit & Open PR

⚠️ WARNING: This step performs irreversible git operations (push, PR creation). Do NOT proceed without explicit user confirmation. Present all planned git commands to the user and wait for approval.

Create branch feat/data-source-<name>
Commit (do NOT add sanity-check script/images)
Present full command list to user for review
Push to fork (only after user approves)
gh pr create --repo NVIDIA/earth2studio (only after user approves)
Immediately post sanity-check validation as PR comment with:
- Variable coverage table (name, count, range, unit)
- Data validation summary (regions, storms/stations, time range)
- Key findings (physically reasonable values, conversions verified)
- Full validation script in <details> block
- Image placeholder:

[CONFIRM — Ready to Submit] (MANDATORY)

This gate is MANDATORY and blocks all git push/PR operations. Present the following to the user and wait for explicit approval:

List of all files to be committed
Full git commands that will be executed (push, PR create)
Target repository and branch
Warning: "These operations are irreversible and will be visible publicly"

Do NOT push or create PRs until the user explicitly confirms.

Step 14 — Automated Code Review

Poll for Greptile review (5 min timeout)
Categorize feedback (bug/style/perf/docs/suggestion/false-positive)
Present triage table to user
Wait for user to approve which comments to address and which to dismiss
Implement accepted fixes
Respond to PR comments (only with user-approved actions)
Push (with user confirmation)

[CONFIRM — Review Triage]

User approves which comments to address and which to dismiss. Never dismiss code review feedback autonomously.

Examples

User: Add a data source for the NOAA GFS analysis on S3
Agent: [loads skill, proceeds through Steps 0–14]

Limitations

One source type per invocation
Requires network access for validation (Step 12)
SPDX license headers required in all files

Reminders

DO: uv run python, loguru.logger, alphabetical order in __init__.py/RST/CHANGELOG, canonical method ordering, obstore helpers, gather_with_concurrency, async_retry, reference URLs in docstrings, try/finally cleanup.

AVOID: hand-rolled fsspec stores, asyncio.to_thread, bare tqdm.gather, xarray for loading, full file downloads.

NEVER: loop.set_default_executor(), commit secrets, commit sanity-check scripts/images.