本系列文章是对 metalkit.org 上面MetalKit内容的全面翻译和学习.
Augmented Reality增强现实提供了一种叠加虚拟内容到摄像头获取到的真实世界视图上的方法.上个月在WWDC2017当看到Apple的新的ARKit框架时我们都很兴奋,这是一个高级API,工作在运行iOS11的A9设备或更新设备上.有些ARKit实验确实非常杰出,比如下面这个:
在ARKit应用中有三种不同的图层:
- Tracking追踪 - 使用视觉惯性里程计来实现世界追踪,无需额外设置.
- Scene Understanding场景理解 - 使用平面检测,点击测试和光照估计来探测场景属性的能力.
- Rendering渲染 - 可以轻松整合,因为
AR视图模板是由SpriteKit和SceneKit提供的,可以用Metal自定义.所有预渲染过程由ARKit处理完成,它还同时负责用AVFoundation和CoreMotion进行图像捕捉.
在本系列的第一章节里,我们将主要关注Metal中的渲染,其余两步将在本系列下一章节中讨论.在一个AR应用中,Tracking追踪和Scene Understanding场景理解是完全由ARKit框架处理的,但渲染可以用SpriteKit, SceneKit或Metal来处理:
开始,我们需要有一个ARSession实例,它是用一个ARSessionConfiguration对象来创建的.然后,我们调用run() 函数来配置.这个会话管理着同时运行的AVCaptureSession和CMMotionManager对象,来获取图像和运动数据来实现追踪.最后,会话将输出当前帧到一个ARFrame对象:
ARSessionConfiguration对象包含了关于追踪类型的信息.ARSessionConfiguration的基础类提供3个自由度的追踪,而它的子类,ARWorldTrackingSessionConfiguration提供6个自由度的追踪(设备位置和旋转方向).
当一个设备不支持世界追踪时,回落到基础配置:
if ARWorldTrackingSessionConfiguration.isSupported {
configuration = ARWorldTrackingSessionConfiguration()
} else {
configuration = ARSessionConfiguration()
}
ARFrame包含了捕捉到的图像,追踪信息和场景信息,场景信息通过包含真实世界位置和旋转信息的ARAnchor对象来获取,这个对象可以轻易从会话中被添加,更新或移除.Tracking追踪是实时确定物理位置的能力.World Tracking,能同时确定位置和朝向,它使用物理距离,与起始位置相关联并提供3D特征点.
ARFrame的最后一个组件是ARCamera对象,它处理变换(平移,旋转,缩放)并携带了追踪状态和相机本体.追踪的质量强烈依赖于不间断的传感器数据,稳定的场景,并且当场景中有大量复杂纹理时会更加精确.追踪状态有三个值:Not Available不可用(相机只有单位矩阵),Limited受限(场景中特征不足或不够稳定),还有Normal正常(相机数据正常).当相机输入不可用时或当追踪停止时,会引发会话打断:
func session(_ session: ARSession, cameraDidChangeTrackingState camera: ARCamera) {
if case .limited(let reason) = camera.trackingState {
// Notify user of limited tracking state
}
}
func sessionWasInterrupted(_ session: ARSession) {
showOverlay()
}
func sessionInterruptionEnded(_ session: ARSession) {
hideOverlay()
// Optionally restart experience
}
Rendering可以在SceneKit中完成,它使用ARSCNView的代理来添加,更新或移除节点.类似的,Rendering也可以在SpriteKit中完成,它使用ARSKView代理来布局SKNodes到ARAnchor对象.因为SpriteKit是2D的,它不能使用真实世界的相机位置,所以它是投影锚点位置到ARSKView,然后在这个被投影的位置作为广告牌(平面)来渲染点精灵的,所以点精灵总是面对着摄像机.对于Metal,没有定制的AR视图,所以这个责任落到了程序员手里.为了处理渲染出的图像,我们需要:
- 绘制背景相机图像(从像素缓冲器生成一个纹理)
- 更新虚拟摄像机
- 更新光照
- 更新几何体的变换
所有这些信息都在ARFrame对象中.为访问这个帧,有两种设置:polling轮询或使用delegate代理.我们将使用后者.我拿出ARKit为Metal准备的模板,并精简到最简,这样我能更好地理解它是如何工作的.我做的第一件事就是移除所有C语言的依赖项,这样就不在需要桥接了.保留这些类型和枚举常量在以后可能会很有用,能用来在API代码和着色器之间共享这些类型和枚举,但是对于本文来说这是不需要的.
下一步,到ViewController中,它将作为我们MTKView和ARSession的代理.我们创建一个Renderer实例,它将与代理协作,实时更新应用:
var session: ARSession!
var renderer: Renderer!
override func viewDidLoad() {
super.viewDidLoad()
session = ARSession()
session.delegate = self
if let view = self.view as? MTKView {
view.device = MTLCreateSystemDefaultDevice()
view.delegate = self
renderer = Renderer(session: session, metalDevice: view.device!, renderDestination: view)
renderer.drawRectResized(size: view.bounds.size)
}
let tapGesture = UITapGestureRecognizer(target: self, action: #selector(self.handleTap(gestureRecognize:)))
view.addGestureRecognizer(tapGesture)
}
正如你看到的,我们将添加一个手势识别器,我们用它来添加虚拟内容到视图中.我们首先拿到会话的当前帧,然后创建一个转换来将我们的物体放到相机前(本例中为0.3米),最后用这个变换来添加一个新的锚点到会话中:
func handleTap(gestureRecognize: UITapGestureRecognizer) {
if let currentFrame = session.currentFrame {
var translation = matrix_identity_float4x4
translation.columns.3.z = -0.3
let transform = simd_mul(currentFrame.camera.transform, translation)
let anchor = ARAnchor(transform: transform)
session.add(anchor: anchor)
}
}
我们使用**viewWillAppear()和viewWillDisappear()**方法来开始和暂停会话:
override func viewWillAppear(_ animated: Bool) {
super.viewWillAppear(animated)
let configuration = ARWorldTrackingSessionConfiguration()
session.run(configuration)
}
override func viewWillDisappear(_ animated: Bool) {
super.viewWillDisappear(animated)
session.pause()
}
剩下的只有响应视图更新或会话错误及打断的代理方法:
func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
renderer.drawRectResized(size: size)
}
func draw(in view: MTKView) {
renderer.update()
}
func session(_ session: ARSession, didFailWithError error: Error) {}
func sessionWasInterrupted(_ session: ARSession) {}
func sessionInterruptionEnded(_ session: ARSession) {}
让我们现在转到Renderer.swift文件中.需要注意的第一件事情是,使用了一个非常有用的协议,它将让我们能访问我们稍后绘制调用中需要的所有MTKView属性:
protocol RenderDestinationProvider {
var currentRenderPassDescriptor: MTLRenderPassDescriptor? { get }
var currentDrawable: CAMetalDrawable? { get }
var colorPixelFormat: MTLPixelFormat { get set }
var depthStencilPixelFormat: MTLPixelFormat { get set }
var sampleCount: Int { get set }
}
现在你可以只需扩展MTKView类(在ViewController中),就让它遵守了这个协议:
extension MTKView : RenderDestinationProvider {}
要想有一个Renderer类的高级视图,下面是它的伪代码:
init() {
setupPipeline()
setupAssets()
}
func update() {
updateBufferStates()
updateSharedUniforms()
updateAnchors()
updateCapturedImageTextures()
updateImagePlane()
drawCapturedImage()
drawAnchorGeometry()
}
像以前一样,我们首先创建管线,这里用setupPipeline() 函数.然后,在setupAssets() 里,我们创建我们的模型,当我们的点击手势识别时就会加载出来.当绘制调用时或需要更新时,MTKView代理将会调用update() 函数.让我们仔细看看它们.首先我们用了updateBufferStates(),它更新我们为当前帧(本例中,我们使用3个空位的环形缓冲器)写入到缓冲器中的位置.
func updateBufferStates() {
uniformBufferIndex = (uniformBufferIndex + 1) % maxBuffersInFlight
sharedUniformBufferOffset = alignedSharedUniformSize * uniformBufferIndex
anchorUniformBufferOffset = alignedInstanceUniformSize * uniformBufferIndex
sharedUniformBufferAddress = sharedUniformBuffer.contents().advanced(by: sharedUniformBufferOffset)
anchorUniformBufferAddress = anchorUniformBuffer.contents().advanced(by: anchorUniformBufferOffset)
}
下一步,在updateSharedUniforms() 中我们更新该帧的共享的uniforms,并为场景设置光照:
func updateSharedUniforms(frame: ARFrame) {
let uniforms = sharedUniformBufferAddress.assumingMemoryBound(to: SharedUniforms.self)
uniforms.pointee.viewMatrix = simd_inverse(frame.camera.transform)
uniforms.pointee.projectionMatrix = frame.camera.projectionMatrix(withViewportSize: viewportSize, orientation: .landscapeRight, zNear: 0.001, zFar: 1000)
var ambientIntensity: Float = 1.0
if let lightEstimate = frame.lightEstimate {
ambientIntensity = Float(lightEstimate.ambientIntensity) / 1000.0
}
let ambientLightColor: vector_float3 = vector3(0.5, 0.5, 0.5)
uniforms.pointee.ambientLightColor = ambientLightColor * ambientIntensity
var directionalLightDirection : vector_float3 = vector3(0.0, 0.0, -1.0)
directionalLightDirection = simd_normalize(directionalLightDirection)
uniforms.pointee.directionalLightDirection = directionalLightDirection
let directionalLightColor: vector_float3 = vector3(0.6, 0.6, 0.6)
uniforms.pointee.directionalLightColor = directionalLightColor * ambientIntensity
uniforms.pointee.materialShininess = 30
}
下一步,在updateAnchors() 中我们用当前帧的锚点的变换来更新锚点uniform缓冲器:
func updateAnchors(frame: ARFrame) {
anchorInstanceCount = min(frame.anchors.count, maxAnchorInstanceCount)
var anchorOffset: Int = 0
if anchorInstanceCount == maxAnchorInstanceCount {
anchorOffset = max(frame.anchors.count - maxAnchorInstanceCount, 0)
}
for index in 0..<anchorInstanceCount {
let anchor = frame.anchors[index + anchorOffset]
var coordinateSpaceTransform = matrix_identity_float4x4
coordinateSpaceTransform.columns.2.z = -1.0
let modelMatrix = simd_mul(anchor.transform, coordinateSpaceTransform)
let anchorUniforms = anchorUniformBufferAddress.assumingMemoryBound(to: InstanceUniforms.self).advanced(by: index)
anchorUniforms.pointee.modelMatrix = modelMatrix
}
}
下一步,在updateCapturedImageTextures() 我们从提供的帧的捕捉图像里,创建两个纹理:
func updateCapturedImageTextures(frame: ARFrame) {
let pixelBuffer = frame.capturedImage
if (CVPixelBufferGetPlaneCount(pixelBuffer) < 2) { return }
capturedImageTextureY = createTexture(fromPixelBuffer: pixelBuffer, pixelFormat:.r8Unorm, planeIndex:0)!
capturedImageTextureCbCr = createTexture(fromPixelBuffer: pixelBuffer, pixelFormat:.rg8Unorm, planeIndex:1)!
}
下一步,在updateImagePlane() 中,我们更新图像平面的纹理坐标来适应视口:
func updateImagePlane(frame: ARFrame) {
let displayToCameraTransform = frame.displayTransform(withViewportSize: viewportSize, orientation: .landscapeRight).inverted()
let vertexData = imagePlaneVertexBuffer.contents().assumingMemoryBound(to: Float.self)
for index in 0...3 {
let textureCoordIndex = 4 * index + 2
let textureCoord = CGPoint(x: CGFloat(planeVertexData[textureCoordIndex]), y: CGFloat(planeVertexData[textureCoordIndex + 1]))
let transformedCoord = textureCoord.applying(displayToCameraTransform)
vertexData[textureCoordIndex] = Float(transformedCoord.x)
vertexData[textureCoordIndex + 1] = Float(transformedCoord.y)
}
}
下一步,在drawCapturedImage() 中我们在场景中绘制来自相机的画面:
func drawCapturedImage(renderEncoder: MTLRenderCommandEncoder) {
guard capturedImageTextureY != nil && capturedImageTextureCbCr != nil else { return }
renderEncoder.pushDebugGroup("DrawCapturedImage")
renderEncoder.setCullMode(.none)
renderEncoder.setRenderPipelineState(capturedImagePipelineState)
renderEncoder.setDepthStencilState(capturedImageDepthState)
renderEncoder.setVertexBuffer(imagePlaneVertexBuffer, offset: 0, index: 0)
renderEncoder.setFragmentTexture(capturedImageTextureY, index: 1)
renderEncoder.setFragmentTexture(capturedImageTextureCbCr, index: 2)
renderEncoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4)
renderEncoder.popDebugGroup()
}
最终,在drawAnchorGeometry() 中,我们为创建的虚拟内容绘制锚点:
func drawAnchorGeometry(renderEncoder: MTLRenderCommandEncoder) {
guard anchorInstanceCount > 0 else { return }
renderEncoder.pushDebugGroup("DrawAnchors")
renderEncoder.setCullMode(.back)
renderEncoder.setRenderPipelineState(anchorPipelineState)
renderEncoder.setDepthStencilState(anchorDepthState)
renderEncoder.setVertexBuffer(anchorUniformBuffer, offset: anchorUniformBufferOffset, index: 2)
renderEncoder.setVertexBuffer(sharedUniformBuffer, offset: sharedUniformBufferOffset, index: 3)
renderEncoder.setFragmentBuffer(sharedUniformBuffer, offset: sharedUniformBufferOffset, index: 3)
for bufferIndex in 0..<mesh.vertexBuffers.count {
let vertexBuffer = mesh.vertexBuffers[bufferIndex]
renderEncoder.setVertexBuffer(vertexBuffer.buffer, offset: vertexBuffer.offset, index:bufferIndex)
}
for submesh in mesh.submeshes {
renderEncoder.drawIndexedPrimitives(type: submesh.primitiveType, indexCount: submesh.indexCount, indexType: submesh.indexType, indexBuffer: submesh.indexBuffer.buffer, indexBufferOffset: submesh.indexBuffer.offset, instanceCount: anchorInstanceCount)
}
renderEncoder.popDebugGroup()
}
回到前面提到的setupPipeline() 函数.创建两个渲染管线状态对象,一个用于捕捉图像(从相机接收),一个用于我们在场景中放置虚拟物体时创建的锚点.正如所料,每一个状态对象都将有一对自己的顶点函数和片段函数 - 让我们转到最后一个需要关注的文件 - 在Shaders.metal文件中.用来捕捉图像的第一对着色器中,我们在顶点着色器中传递图像的顶点位置和纹理坐标:
vertex ImageColorInOut capturedImageVertexTransform(ImageVertex in [[stage_in]]) {
ImageColorInOut out;
out.position = float4(in.position, 0.0, 1.0);
out.texCoord = in.texCoord;
return out;
}
在片段着色器中,我们采样两个纹理来得到给定纹理坐标处的颜色,然后返回修改过的RGB颜色:
fragment float4 capturedImageFragmentShader(ImageColorInOut in [[stage_in]],
texture2d<float, access::sample> textureY [[ texture(1) ]],
texture2d<float, access::sample> textureCbCr [[ texture(2) ]]) {
constexpr sampler colorSampler(mip_filter::linear, mag_filter::linear, min_filter::linear);
const float4x4 ycbcrToRGBTransform = float4x4(float4(+1.0000f, +1.0000f, +1.0000f, +0.0000f),
float4(+0.0000f, -0.3441f, +1.7720f, +0.0000f),
float4(+1.4020f, -0.7141f, +0.0000f, +0.0000f),
float4(-0.7010f, +0.5291f, -0.8860f, +1.0000f));
float4 ycbcr = float4(textureY.sample(colorSampler, in.texCoord).r, textureCbCr.sample(colorSampler, in.texCoord).rg, 1.0);
return ycbcrToRGBTransform * ycbcr;
}
给锚点几何体的第二对着色器中,顶点着色器中我们在裁剪空间里计算顶点的位置并输出,以供裁剪和光栅化,然后给每个面着上不同颜色,然后计算我们顶点在观察空间的位置,并最终旋转法线到世界坐标系:
vertex ColorInOut anchorGeometryVertexTransform(Vertex in [[stage_in]],
constant SharedUniforms &sharedUniforms [[ buffer(3) ]],
constant InstanceUniforms *instanceUniforms [[ buffer(2) ]],
ushort vid [[vertex_id]],
ushort iid [[instance_id]]) {
ColorInOut out;
float4 position = float4(in.position, 1.0);
float4x4 modelMatrix = instanceUniforms[iid].modelMatrix;
float4x4 modelViewMatrix = sharedUniforms.viewMatrix * modelMatrix;
out.position = sharedUniforms.projectionMatrix * modelViewMatrix * position;
ushort colorID = vid / 4 % 6;
out.color = colorID == 0 ? float4(0.0, 1.0, 0.0, 1.0) // Right face
: colorID == 1 ? float4(1.0, 0.0, 0.0, 1.0) // Left face
: colorID == 2 ? float4(0.0, 0.0, 1.0, 1.0) // Top face
: colorID == 3 ? float4(1.0, 0.5, 0.0, 1.0) // Bottom face
: colorID == 4 ? float4(1.0, 1.0, 0.0, 1.0) // Back face
: float4(1.0, 1.0, 1.0, 1.0); // Front face
out.eyePosition = half3((modelViewMatrix * position).xyz);
float4 normal = modelMatrix * float4(in.normal.x, in.normal.y, in.normal.z, 0.0f);
out.normal = normalize(half3(normal.xyz));
return out;
}
在片段着色器中,我们计算方向光的贡献值,使用漫反射和高光项目的总和,然后通过将从颜色地图的采样与片段的光照值相乘来计算最终颜色,最后,用刚计算出来的颜色和颜色地图的透明通道给片段的透明度值:
fragment float4 anchorGeometryFragmentLighting(ColorInOut in [[stage_in]],
constant SharedUniforms &uniforms [[ buffer(3) ]]) {
float3 normal = float3(in.normal);
float3 directionalContribution = float3(0);
{
float nDotL = saturate(dot(normal, -uniforms.directionalLightDirection));
float3 diffuseTerm = uniforms.directionalLightColor * nDotL;
float3 halfwayVector = normalize(-uniforms.directionalLightDirection - float3(in.eyePosition));
float reflectionAngle = saturate(dot(normal, halfwayVector));
float specularIntensity = saturate(powr(reflectionAngle, uniforms.materialShininess));
float3 specularTerm = uniforms.directionalLightColor * specularIntensity;
directionalContribution = diffuseTerm + specularTerm;
}
float3 ambientContribution = uniforms.ambientLightColor;
float3 lightContributions = ambientContribution + directionalContribution;
float3 color = in.color.rgb * lightContributions;
return float4(color, in.color.w);
}
如果你运行应用,将能够通过点击屏幕来添加立方体到相机视图上,到处移动或凑近或环绕立方体来观察每个面的不同颜色,比如这样:
在本系列的下一章节,我们将更深入学习Tracking追踪和Scene Understanding场景理解,并看看平面检测,点击测试,碰撞和物理效果是如何让我们的经历更美好的.
源代码source code已发布在Github上.
下次见!
