LiDAR and RealityKit – Capture a Real World Texture for a Scanned Model

by

Scene Reconstruction

Pity but I am still unable to capture model’s texture in realtime using the LiDAR scanning process. Neither at WWDC20 nor at WWDC22 Apple announced a native API for that (so texture capturing is only possible now using third-party APIs – don’t ask me which ones 🙂 ).

However, there’s good news – a new methodology has emerged at last. It will allow developers to create textured models from a series of shots.

Photogrammetry

Object Capture API, announced at WWDC 2021, provides developers with the long-awaited photogrammetry tool. At the output we get USDZ model with UV-mapped hi-rez texture. To implement Object Capture API you need macOS 12 and Xcode 13.

enter image description here

To create a USDZ model from a series of shots, submit all taken images to RealityKit’s PhotogrammetrySession.

Here’s a code snippet that spills a light on this process:

import RealityKit
import Combine

let pathToImages = URL(fileURLWithPath: "/path/to/my/images/")

let url = URL(fileURLWithPath: "model.usdz")

var request = PhotogrammetrySession.Request.modelFile(url: url, 
                                                   detail: .medium)

var configuration = PhotogrammetrySession.Configuration()
configuration.sampleOverlap = .normal
configuration.sampleOrdering = .unordered
configuration.featureSensitivity = .normal
configuration.isObjectMaskingEnabled = false

guard let session = try PhotogrammetrySession(input: pathToImages, 
                                      configuration: configuration)
else { return 
} 

var subscriptions = Set<AnyCancellable>()

session.output.receive(on: DispatchQueue.global())
              .sink(receiveCompletion: { _ in
                  // errors
              }, receiveValue: { _ in
                  // output
              }) 
              .store(in: &subscriptions)

session.process(requests: [request])

You can reconstruct USD and OBJ models with their corresponding UV-mapped textures.

More Related Contents:

  1. RealityKit vs SceneKit vs Metal – High-Quality Rendering
  2. How to create USDZ file using Xcode converter?
  3. What is ARAnchor exactly?
  4. ARKit – What do the different columns in Transform Matrix represent?
  5. How to improve People Occlusion in ARKit 3.0
  6. What is the real benefit of using Raycast in ARKit and RealityKit?
  7. How to Add Material to ModelEntity programatically in RealityKit?
  8. What’s the difference between using ARAnchor to insert a node and directly insert a node?
  9. Projecting the ARKit face tracking 3D mesh to 2D image coordinates
  10. Adding 3D object to ARGeoAnchor
  11. How do I programmatically move an ARAnchor?
  12. Concatenate dictionary values in Swift
  13. Overriding methods in Swift extensions
  14. What’s the difference between “as?”, “as!”, and “as”?
  15. The “++” and “–” operators have been deprecated Xcode 7.3
  16. Changing the Status Bar Color for specific ViewControllers using Swift in iOS8
  17. Swift extension example
  18. UITextView with hyperlink text
  19. What’s the simplest way to convert from a single character String to an ASCII value in Swift?
  20. How to create local scopes in Swift?
  21. Swift – How to mutate a struct object when iterating over it
  22. Assign value of a Firestore document to a variable
  23. Deletable Table with TextField on SwiftUI
  24. How add tabs programmatically in UITabBarController with swift?
  25. Generating random numbers with Swift
  26. How String Comparison happens in Swift
  27. In Swift 3, how to calculate the factorial when the result becomes too high? [duplicate]
  28. Convert Firebase Firestore Timestamp to Date (Swift)?
  29. SwiftUI 2: the way to open view in new window
  30. Programmatically Screenshot | Swift 3, macOS

Leave a Comment