camera-streaming

โดย facebook

สตรีม, เฟรมวิดีโอ, การจับภาพถ่าย, การกำหนดค่าความละเอียด/อัตราเฟรม

npx skills add https://github.com/facebook/meta-wearables-dat-ios --skill camera-streaming

ดาวน์โหลด ZIP GitHub

552

Camera Streaming (iOS)

Guide for implementing camera streaming and photo capture with the DAT SDK.

For PCM audio delivered with this stream, use the audio-streaming skill.

Key concepts

DeviceSession: The connection to the glasses. Create and start it first; it can stay connected with no camera attached.
Camera: The capability you attach to a started DeviceSession with addCamera(config:). It owns the Stream.
Stream: camera.stream — start/stop streaming, capture photos, observe frames, state, and errors.
VideoFrame: Individual video frames. Use .makeUIImage() to render, or .sampleBuffer when you need the raw CMSampleBuffer (for example to write compressed frames to a file).
StreamConfiguration: Configure video codec, resolution, and frame rate.
PhotoData: Still image captured from glasses (.data plus .format).

Treat "start a session" and "start the preview" as two separate user-visible steps, as the CameraAccess sample does. That keeps the session reusable when the user stops and restarts the preview.

Creating a DeviceSession

import MWDATCamera
import MWDATCore

let wearables = Wearables.shared
let deviceSelector = AutoDeviceSelector(wearables: wearables)
// Or for a specific device: SpecificDeviceSelector(device: deviceId)
let deviceSession = try wearables.createSession(deviceSelector: deviceSelector)
try deviceSession.start()

// Wait for the device session to reach the started state
for await state in deviceSession.stateStream() {
    if state == .started { break }
}

Adding a Camera

Once the DeviceSession is started, add the Camera capability and stream through camera.stream.

Check camera permission first. checkPermissionStatus(.camera) is a silent query, while requestPermission(.camera) switches to the Meta AI app. CameraAccess only calls requestPermission after the user confirms the app switch, so the redirect is never a surprise:

if try await wearables.checkPermissionStatus(.camera) != .granted {
    // Confirm with the user, then:
    guard try await wearables.requestPermission(.camera) == .granted else { return }
}

Then attach the camera:

let config = StreamConfiguration(
    videoCodec: .raw,
    resolution: .medium,  // 504x896
    frameRate: 24
)

guard let camera = try deviceSession.addCamera(config: config) else {
    // DeviceSession must be in the started state before adding a camera
    return
}
let stream = camera.stream

Resolution options

Resolution	Size
`.high`	720 x 1280
`.medium`	504 x 896
`.low`	360 x 640

Frame rate options

frameRate is a UInt. Valid values: 2, 7, 15, 24, 30 FPS.

Lower resolution and frame rate yield higher visual quality due to less Bluetooth compression.

Codec options

VideoCodec has two cases:

.raw — decoded frames. Render them directly with frame.makeUIImage().
.hvc1 — compressed HEVC frames. Use this when you want to write frames to a file in passthrough mode; you must decode frame.sampleBuffer yourself for on-screen preview, as CameraAccess does.

StreamConfiguration() with no arguments defaults to .raw, .medium, and 30 FPS.

Observing stream state

StreamState transitions: stopping → stopped → waitingForDevice → starting → streaming → paused

let stateToken = stream.statePublisher.listen { state in
    Task { @MainActor in
        switch state {
        case .streaming:
            // Stream is active, frames are flowing
        case .waitingForDevice:
            // Waiting for glasses to connect
        case .stopped:
            // Stream ended — release resources
        case .paused:
            // Temporarily suspended — keep connection, wait
        default:
            break
        }
    }
}

Receiving video frames

let frameToken = stream.videoFramePublisher.listen { frame in
    guard let image = frame.makeUIImage() else { return }
    Task { @MainActor in
        self.previewImage = image
    }
}

Observing stream errors

stream.errorPublisher emits typed StreamError values such as .permissionDenied, .deviceNotConnected, .hingesClosed, .thermalHot, .batteryLow, and .photoCaptureFailed. Surface error.localizedDescription rather than maintaining your own error-message map.

let errorToken = stream.errorPublisher.listen { error in
    Task { @MainActor in
        self.errorMessage = error.localizedDescription
    }
}

Keep listener tokens alive for as long as you need callbacks. MWDATCore provides ListenerTokenBag and AnyListenerToken.store(in:) for that:

let streamTokens = ListenerTokenBag()
stream.statePublisher.listen { _ in }.store(in: streamTokens)
stream.videoFramePublisher.listen { _ in }.store(in: streamTokens)
// On teardown:
streamTokens.clear()

Starting and stopping

// Start the stream capability
stream.start()

// Stop the camera — this cascades to its stream child and detaches the camera
// from the session, so a later addCamera() can register a new one.
camera.stop()

// Stop the parent device session when you're done with all capabilities.
// Teardown cascades parent -> child, not child -> parent.
deviceSession.stop()

In-stream photo capture

This is the lightweight capture path used while video is streaming. For standalone high-quality capture with resolution, quality, transfer progress, and its own lifecycle, use the camera-capture skill.

Capture a still photo while streaming. capturePhoto(format:) returns Bool: false means the capture was not started (for example another capture is already in flight), so no photoDataPublisher event will arrive.

// Listen for photo data
let photoToken = stream.photoDataPublisher.listen { photoData in
    let imageData = photoData.data  // photoData.format is .jpeg or .heic
    // Convert to UIImage or save
}

// Trigger capture
let started = stream.capturePhoto(format: .jpeg)
if !started {
    // Re-enable the shutter button and tell the user to try again
}