synapseml-local-setup

Set up and validate SynapseML locally in WSL or Linux. Use when an agent needs SynapseML working locally, runs sbt compile/test, sees Java 21, Scala 2.12…

SynapseML Local Setup

Use this skill before any local SynapseML build, compile, or test validation.

Important

Always use an explicit SynapseML repo path.
Do not run SynapseML SBT with the machine default Java 21. Use JDK 11: /usr/lib/jvm/java-11-openjdk-amd64
Java 21 can fail before project code compiles with bad constant pool index: 0 while building Scala 2.12 compiler-bridge_2.12.
Compile commands are safe. Some cognitive service tests create, write, list, or delete real Azure resources. Inspect before running those tests and ask for approval if live resources are involved.

Workflow

1. Diagnose the repo and toolchain

Run scripts/synapseml-doctor.sh:

scripts/synapseml-doctor.sh --repo <synapseml-repo>

Capture:

Git branch and dirty state.
Default Java version.
JDK 11 availability.
sbt version and SynapseML Scala/Spark versions.

2. Compile with JDK 11

Run scripts/synapseml-sbt.sh:

scripts/synapseml-sbt.sh --repo <synapseml-repo> -- cognitive/Test/compile

Expected result:

sbt welcome line says Java 11.
core and cognitive main/test classes compile.
Command exits with [success].

3. Run a safe local smoke test

Run scripts/synapseml-smoke-test.sh:

scripts/synapseml-smoke-test.sh --repo <synapseml-repo>

Expected result:

One local Spark test runs.
Output includes All tests passed.

4. Inspect PR-specific tests before running them

Before running service tests, run scripts/check-live-service-tests.sh:

scripts/check-live-service-tests.sh --path <test-file-or-directory>

If it reports live-service hooks, ask the user before running that suite. Do not create or delete Azure Search indexes just to test a PR.

5. Run targeted tests only after safety review

Use the JDK 11 wrapper for any targeted SBT command:

scripts/synapseml-sbt.sh --repo <synapseml-repo> -- '<module>/testOnly <SuiteName> -- -z "<test filter>"'

If tests fail before compiling project code, load references/troubleshooting.md.

Known-good baseline

On 2026-05-05, this setup was validated with:

JDK: /usr/lib/jvm/java-11-openjdk-amd64
Java: openjdk version "11.0.30"
SynapseML: Scala 2.12.17, Spark 3.5.0, sbt 1.10.11
Compile: sbt 'cognitive/Test/compile' succeeded.
Smoke test: UDFTransformerSuite filtered test succeeded.