camera-streaming

द्वारा facebook

स्ट्रीम, वीडियो फ्रेम, फोटो कैप्चर, रिज़ॉल्यूशन/फ्रेम दर कॉन्फ़िगरेशन

npx skills add https://github.com/facebook/meta-wearables-dat-ios --skill camera-streaming

ZIP डाउनलोड करें GitHub

552

Camera Streaming (iOS)

Guide for implementing camera streaming and photo capture with the DAT SDK.

For PCM audio delivered with this stream, use the audio-streaming skill.

Key concepts

DeviceSession: The connection to the glasses. Create and start it first; it can stay connected with no camera attached.
Camera: The capability you attach to a started DeviceSession with addCamera(config:). It owns the Stream.
Stream: camera.stream — start/stop streaming, capture photos, observe frames, state, and errors.
VideoFrame: Individual video frames. Use .makeUIImage() to render, or .sampleBuffer when you need the raw CMSampleBuffer (for example to write compressed frames to a file).
StreamConfiguration: Configure video codec, resolution, and frame rate.
PhotoData: Still image captured from glasses (.data plus .format).

Treat "start a session" and "start the preview" as two separate user-visible steps, as the CameraAccess sample does. That keeps the session reusable when the user stops and restarts the preview.

Creating a DeviceSession

import MWDATCamera
import MWDATCore

let wearables = Wearables.shared
let deviceSelector = AutoDeviceSelector(wearables: wearables)
// Or for a specific device: SpecificDeviceSelector(device: deviceId)
let deviceSession = try wearables.createSession(deviceSelector: deviceSelector)
try deviceSession.start()

// Wait for the device session to reach the started state
for await state in deviceSession.stateStream() {
    if state == .started { break }
}

Adding a Camera

Once the DeviceSession is started, add the Camera capability and stream through camera.stream.

Check camera permission first. checkPermissionStatus(.camera) is a silent query, while requestPermission(.camera) switches to the Meta AI app. CameraAccess only calls requestPermission after the user confirms the app switch, so the redirect is never a surprise:

if try await wearables.checkPermissionStatus(.camera) != .granted {
    // Confirm with the user, then:
    guard try await wearables.requestPermission(.camera) == .granted else { return }
}

Then attach the camera:

let config = StreamConfiguration(
    videoCodec: .raw,
    resolution: .medium,  // 504x896
    frameRate: 24
)

guard let camera = try deviceSession.addCamera(config: config) else {
    // DeviceSession must be in the started state before adding a camera
    return
}
let stream = camera.stream

Resolution options

Resolution	Size
`.high`	720 x 1280
`.medium`	504 x 896
`.low`	360 x 640

Frame rate options

frameRate is a UInt. Valid values: 2, 7, 15, 24, 30 FPS.

Lower resolution and frame rate yield higher visual quality due to less Bluetooth compression.

Codec options

VideoCodec has two cases:

.raw — decoded frames. Render them directly with frame.makeUIImage().
.hvc1 — compressed HEVC frames. Use this when you want to write frames to a file in passthrough mode; you must decode frame.sampleBuffer yourself for on-screen preview, as CameraAccess does.

StreamConfiguration() with no arguments defaults to .raw, .medium, and 30 FPS.

Observing stream state

StreamState transitions: stopping → stopped → waitingForDevice → starting → streaming → paused

let stateToken = stream.statePublisher.listen { state in
    Task { @MainActor in
        switch state {
        case .streaming:
            // Stream is active, frames are flowing
        case .waitingForDevice:
            // Waiting for glasses to connect
        case .stopped:
            // Stream ended — release resources
        case .paused:
            // Temporarily suspended — keep connection, wait
        default:
            break
        }
    }
}

Receiving video frames

let frameToken = stream.videoFramePublisher.listen { frame in
    guard let image = frame.makeUIImage() else { return }
    Task { @MainActor in
        self.previewImage = image
    }
}

Observing stream errors

stream.errorPublisher emits typed StreamError values such as .permissionDenied, .deviceNotConnected, .hingesClosed, .thermalHot, .batteryLow, and .photoCaptureFailed. Surface error.localizedDescription rather than maintaining your own error-message map.

let errorToken = stream.errorPublisher.listen { error in
    Task { @MainActor in
        self.errorMessage = error.localizedDescription
    }
}

Keep listener tokens alive for as long as you need callbacks. MWDATCore provides ListenerTokenBag and AnyListenerToken.store(in:) for that:

let streamTokens = ListenerTokenBag()
stream.statePublisher.listen { _ in }.store(in: streamTokens)
stream.videoFramePublisher.listen { _ in }.store(in: streamTokens)
// On teardown:
streamTokens.clear()

Starting and stopping

// Start the stream capability
stream.start()

// Stop the camera — this cascades to its stream child and detaches the camera
// from the session, so a later addCamera() can register a new one.
camera.stop()

// Stop the parent device session when you're done with all capabilities.
// Teardown cascades parent -> child, not child -> parent.
deviceSession.stop()

In-stream photo capture

This is the lightweight capture path used while video is streaming. For standalone high-quality capture with resolution, quality, transfer progress, and its own lifecycle, use the camera-capture skill.

Capture a still photo while streaming. capturePhoto(format:) returns Bool: false means the capture was not started (for example another capture is already in flight), so no photoDataPublisher event will arrive.

// Listen for photo data
let photoToken = stream.photoDataPublisher.listen { photoData in
    let imageData = photoData.data  // photoData.format is .jpeg or .heic
    // Convert to UIImage or save
}

// Trigger capture
let started = stream.capturePhoto(format: .jpeg)
if !started {
    // Re-enable the shutter button and tell the user to try again
}