[2024-1] 김경훈 - Point Transformer

 

 

https://arxiv.org/abs/2012.09164

Point Transformer

 

 

---

1. Introduction
기존의 3D point cloud에 대한 접근 방식은 일반적으로 다음과 같습니다:

Voxelize 3D space
Sparse convolution networks
Directly on points
Transformer은 NLP분야와 image analysis 분야에서 유용하게 사용되고 있는데, Self-attention operator은 input에 대해 permutation-invariant하기 때문에, point cloud에서 사용하기 적합하다는 특징을 가지고 있다.

이번 논문에서는 transformer을 3D point의 상황에 맞게 고려하여 적용한다:

Self-attention operator
각각의 점들의 local neighborhood에 대해 self-attention 적용
Positional information encoding

 

 

2. Related Work
Projection-based networks

Multi-view projection을 통해 다양한 image plane을 사용
Geometric information이 projection과정에서 많이 손실된다.
TangentConv: tangent image를 2D convolution에 사용.
Voxel-based networks

Voxelization과정을 거쳐 3D convolution을 이용해 feature map을 얻는다.
Quantization 과정에서 geometric detail 손실이 발생하고, computation / memory cost가 높다.
OctNet: octree를 이용해 sparse convolution효과를 얻는다.
Point-based networks

Point cloud 데이터 자체를 input으로 사용한다.
PointNet / PointNet++
Point set을 graph으로 연결하여 사용: DGCNN / ECC / SPG / KCNet / GACNet / HPEIN / DeepGCNs
continuous convolution을 point set에 바로 적용 : PCCN / SpiderCNN / Spherical CNN / PointConv / KPConv / InterpCNN / PointCNN
Transformer and self-attention

Translation / NLP 분야에서 시작하여 2D image recognition분야에서도 사용되었다.
self-attention은 set operator이기 때문에, input의 순서에 상관없이 사용할 수 있어 3D point cloud에 적합하다.
이번 연구 이전에도 transformer을 이용한 연구가 있었지만, 전체 point cloud에 대해 사용하면 연산량이 너무 많게 되어 large-scale 3D point cloud에서 사용이 어렵고, Scalar dot-product attention을 사용해서 서로 다른 channel이 같은 weight을 공유하게 된다는 단점을 보여주었다.
이번 연구에서는 self-attention을 local끼리만 적용했고, vector attention을 적용하여 더 높은 정확도를 얻을 수 있었다. 또한, 적절한 position encoding을 적용하여 좋은 결과를 얻었다.

 

3. Point Transformer

3.1. Backgrounds

scalar dot product attention

ψ,ϕ,α: pointwise feature transformations
δ: position encoding function
ρ: normalization function (ex – Softmax)
vector attention

β: relation function (ex – subtraction)
γ: mapping function (ex – MLP)
3.2. point transformer layer
vector self-attention을 이용하여 다음과 같이 transformer layer을 구성한다:

 

 

3.3. Position Encoding

 : point i, j의 3D coordinate
𝜃: 2 linear layer + ReLU로 구성된 MLP

 

 

 

3.4. Point Transformer Block

3.5. Network Architecture
Convolution을 사용하지 않고 point transformer layer, pointwise transformation, pooling 으로 구성되어 있다.

Backbone structure

Transition down – feature encoding
Transition up – feature decoding
Transition down

Point set의 cardinality를 farthest point sampling (FPS)을 이용해 줄여준다.
그 후, kNN을 이용해 feature vector을 pooling한다.
Transition up

Transition down에서 줄어든 point set의 크기를 이전의 크기로 키워준다.
Trilinear interpolation을 통해 고해상도 point set을 얻는다.
Interpolation block과 skip-connection으로 구성되어 있다.
Output head

마지막 decoder layer의 output으로 각각의 점들에 대한 feature vector을 얻는다.
MLP와 global average pooling을 통해 결과를 얻을 수 있다.

 

 

4. Experiments

4.1. Semantic Segmentation
Data: S3DIS dataset – 271 rooms in 6 areas from 3 different buildings / 13 categories

Evaluation metric

Mean classwise intersection over union (mIoU)
Mean of classwise accuracy (mAcc)
Overall pointwise accuracy (OA)

Number of parameter

PointTransformer – 4.9M
KPConv - 14.9M
SparseConv – 30.1M

4.2. Shape Classification
Data: ModelNet40 – 12311 CAD models / 40 object categories

Evaluation metric

Mean accuracy within each category (mAcc)
Overall accuracy (OA)

4.3. Object Part Segmentation
Data: ShapeNetPart

16,880 models from 16 shape categories
number of part for each category: 2 ~ 6
Metric

category mIoU
instance mIoU


4.4. Ablation Study
Number of neighbors


Position encoding


Attention type

 

---

 

 

논문 리뷰 정리 PPT  : 

https://blog.naver.com/kgh9080/223439222284

[논문리뷰] Point Transformer