earth2studio-create-datasource

Create and Validate Data Source

Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to Earth2Studio's async data fetching infrastructure — from analysis through implementation, testing, validation, and PR submission.

Prerequisites

• Earth2Studio dev environment with uv (uv run python must work)
• Git configured with fork (origin) and upstream (upstream) remotes
• Access to the target remote data store (credentials if private)
• Python 3.10+

⚠️ Credential Safety: If the remote store requires authentication, credentials must be passed via environment variables only. Never hard-code secrets in source files. Before any git commit, verify no credentials are staged (check git diff --cached for API keys, tokens, or passwords).

Workspace

Use the directory containing pyproject.toml. For Harbor evals, write to /workspace/output/ preserving paths. Never read evals/targets/.

Instructions

Python Environment: Always use uv run python or the local .venv. Never use the system Python directly.

Follow every step in order.

[CONFIRM] gates: Step 1 (Source Type), Step 12 (Sanity-Check Plots), and Step 13 (Ready to Submit) require explicit user approval before proceeding. All other [CONFIRM] markers are advisory — present decisions inline and proceed without blocking.

Deliverables first: Write the source file and test file (Steps 6–7) before extended exploration, documentation, registration, CHANGELOG, or PR work. Skip Steps 8–14 when the user asks for implementation only.

Before you finish: Run verification commands in the repo root so results appear in the session log:

uv run pytest test/data/test_<source>.py -x
make format && make lint

Be concise: Avoid long architecture reports; summarize decisions in a few sentences and move on to file writes.

Hangs or User Feedback If agent becomes stuck or user provides a correction during this skills use, conservatively review relevant part of the skill and improve. Be concise.

One source type per invocation. Invoke again for companion types.

Reference Files

Load these on demand during the relevant steps:

File	Content	Load at
references/reference-implementation.py	Obstore-backed source skeleton with FILL comments	Steps 6–10
references/reference-lexicon.py	Lexicon skeleton with FILL comments	Step 4
references/testing-guide.py	Test skeleton with FILL comments	Step 11
references/validation-guide.md	Plot templates, PR body template, Greptile handling	Steps 12–14 (optional, for templates)

---

Workflow Overview

Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review

---

Step 0 — Obtain Remote Data Store Reference

If $ARGUMENTS is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

Please provide a URL, API documentation link, or description of the remote data store. This will be used to understand storage format, access pattern, variable inventory, temporal/spatial resolution.

---

Step 1 — Determine Source Type

Protocol	Returns	Has lead_time?	Use
DataSource	xr.DataArray	No	Gridded analysis/reanalysis
ForecastSource	xr.DataArray	Yes	Gridded forecast
DataFrameSource	pd.DataFrame	No	Sparse/station obs
ForecastFrameSource	pd.DataFrame	Yes	Sparse forecast obs

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

[CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

---

Step 2 — Examine Remote Store & Propose Dependencies

Analyze: storage backend, file format, authentication, access pattern, temporal/spatial resolution, variable inventory.

Prefer Earth2Studio obstore helpers:

Backend / format	Preferred	Avoid
S3/GCS/HTTP object files	obstore_store_from_url + obstore_fetch_to_cache	raw SDK clients
Cloud Zarr	obstore_zarr_store	hand-rolled fsspec/zarr stores
Azure/object auth	obstore.store.from_url via shared helpers	provider SDKs directly
HuggingFace	huggingface_hub (core dep)	custom scripts

Only fall back to dedicated libraries when obstore/shared helpers cannot access the store.

Check pyproject.toml — only propose packages not already present. Core deps include: obstore, zarr, netCDF4, h5py, pygrib, huggingface-hub, pandas, pyarrow.

[CONFIRM — Dependencies & Access Pattern]

Present: backend, obstore helper/store URL, new packages (with license), auth method.

---

Step 3 — Add Dependencies

Load references/reference-implementation.py from Step 6 through Step 10. Load references/reference-lexicon.py during Step 4.

If new packages needed:

1. Present the package name(s), version constraints, and licenses to the user
2. Confirm the user approves adding them before proceeding
3. uv add --extra data <package>
4. uv lock
5. Add optional dependency imports using OptionalDependencyFailure pattern

---

Step 4 — Create Lexicon Class

Create earth2studio/lexicon/<source_name>.py with:

• metaclass=LexiconType
• VOCAB: dict[str, str] mapping E2S names → remote keys
• get_item(cls, val) returning tuple[str, Callable]
• Use :: separator for structured keys
• Use references/reference-lexicon.py as the starting template

Map remote variables against E2STUDIO_VOCAB (282 entries in earth2studio/lexicon/base.py).

[CONFIRM — Lexicon & Variable Mapping]

Present: class name, key format, full mapping table, modifiers, reference URL.

---

Step 5 — Update E2STUDIO_VOCAB / SCHEMA (if needed)

• New vocab: surface = descriptive abbrev; pressure = {name}{level}
• New schema fields: DataFrame sources only, check E2STUDIO_SCHEMA

[CONFIRM — Vocabulary & Schema Updates]

Skip if no updates needed.

---

Step 6 — Create Skeleton Data Source File

Use references/reference-implementation.py as the starting template.

Follow canonical method ordering:

1. Class constants
2. SCHEMA
3. __init__
4. _async_init
5. __call__
6. fetch
7. _create_tasks
8. fetch_wrapper
9. fetch_array
10. _validate_time
11. Helpers
12. cache property
13. available classmethod

Use async task dataclass pattern for parallel execution.

[CONFIRM — Skeleton]

Present: class name, file path, skeleton code, task dataclass.

---

Step 7 — Implement the Data Source & Tests

The test file is a co-equal deliverable. Create test/data/test_<filename>.py alongside the source. Add test_<source>_call_mock for async sources.

Sync sources: Use prep_data_inputs/prep_forecast_inputs, direct __call__.

Async sources: See references/reference-implementation.py for required patterns: _sync_async, gather_with_concurrency, async_retry, obstore_fetch_to_cache, obstore_zarr_store for Zarr, try/finally cleanup. Constructor params: cache=True, verbose=True, async_timeout=600, async_workers=None, retries=3. DataFrame sources add time_tolerance.

---

Step 8 — Register the Source

• earth2studio/data/__init__.py — alphabetical import
• earth2studio/lexicon/__init__.py — alphabetical import
• Verify pyproject.toml deps

---

Step 9 — Update Documentation

• Add the class alphabetically to the appropriate autosummary block in docs/modules/datasources_analysis.md, datasources_forecast.md, or datasources_dataframe.md
• Class docstring: Parameters, Warning (download size), Note (reference URLs), Badges (last)
• Keep the existing region, dataclass, and product badges accurate. Add provider:<name> when the service provider is known, and add dataset:<family> only when the source belongs to a named dataset family; do not infer a family from the provider. Use dataset:gfs for GEFS sources.
• Reuse lowercase badge keys already defined in mkdocs.yml. If a new provider or dataset family is required, add its badge definition there and use hide_in: [autosummary, filter]; the catalog reads these badges even though data-source API tables do not display or filter on them.
• Example: region:global dataclass:reanalysis dataset:era5 product:atmos provider:google
• All public methods: NumPy-style docstrings

---

Step 10 — Update CHANGELOG.md

Add entry under the current unreleased version. See references/reference-implementation.py REGISTRATION CHECKLIST for the format.

One line per source. Do NOT add separate lexicon entries.

---

Step 11 — Verify Style & Expand Tests

Run make format && make lint && make license. Load references/testing-guide.py for test skeletons. Required tests: test_<source>_fetch (slow), _cache (slow), _call_mock, _exceptions, _available. Target 90%+ coverage with --slow.

[CONFIRM — Tests]

Present test file, functions, coverage.

---

Step 12 — Validate Variables & Sanity-Check

1. Validate all lexicon vars against real data (run script, do NOT commit)
2. Remove variables with < 10% valid data
3. Create sanity-check plot (gridded or sparse template)
4. Tell user the plot path and ask for visual confirmation

[CONFIRM — Sanity-Check Plots]

User MUST visually inspect plots. Do not proceed without confirmation.

---

Step 13 — Branch, Commit & Open PR

⚠️ WARNING: This step performs irreversible git operations (push, PR creation). Do NOT proceed without explicit user confirmation. Present all planned git commands to the user and wait for approval.

1. Create branch feat/data-source-<name>
2. Commit (do NOT add sanity-check script/images)
3. Present full command list to user for review
4. Push to fork (only after user approves)
5. gh pr create --repo NVIDIA/earth2studio (only after user approves)
6. Immediately post sanity-check validation as PR comment with:
   • Variable coverage table (name, count, range, unit)
   • Data validation summary (regions, storms/stations, time range)
   • Key findings (physically reasonable values, conversions verified)
   • Full validation script in <details> block
   • Image placeholder: <!-- Drag and drop sanity-check image here -->

[CONFIRM — Ready to Submit] (MANDATORY)

This gate is MANDATORY and blocks all git push/PR operations. Present the following to the user and wait for explicit approval:

1. List of all files to be committed
2. Full git commands that will be executed (push, PR create)
3. Target repository and branch
4. Warning: "These operations are irreversible and will be visible publicly"

Do NOT push or create PRs until the user explicitly confirms.

---

Step 14 — Automated Code Review

1. Poll for Greptile review (5 min timeout)
2. Categorize feedback (bug/style/perf/docs/suggestion/false-positive)
3. Present triage table to user
4. Wait for user to approve which comments to address and which to dismiss
5. Implement accepted fixes
6. Respond to PR comments (only with user-approved actions)
7. Push (with user confirmation)

[CONFIRM — Review Triage]

User approves which comments to address and which to dismiss. Never dismiss code review feedback autonomously.

---

Examples

User: Add a data source for the NOAA GFS analysis on S3
Agent: [loads skill, proceeds through Steps 0–14]

---

Limitations

• One source type per invocation
• Requires network access for validation (Step 12)
• SPDX license headers required in all files

---

Reminders

DO: uv run python, loguru.logger, alphabetical order in __init__.py/RST/CHANGELOG, canonical method ordering, obstore helpers, gather_with_concurrency, async_retry, reference URLs in docstrings, try/finally cleanup.

AVOID: hand-rolled fsspec stores, asyncio.to_thread, bare tqdm.gather, xarray for loading, full file downloads.

NEVER: loop.set_default_executor(), commit secrets, commit sanity-check scripts/images.