【资源】万字长文资源之 3D-Machine-Learning

写在前面:一直以来,极市得到了许许多多开发者的关注和支持,极市的成长离不开各位开发者的见证,为此我们特开设极市开发者祝愿帖,期待听到您真实的心声和建议~φ (> ω<*) :

极市开发者祝愿活动 (有奖回馈)

前言

之前极市曾分享了几个GitHub上的awesome系列项目,反响都很好。



今天分享一个3D-Machine-Learning相关领域的相关课程、数据。各个细分方向的论文等资源汇总列表,近年来,3D-Machine-Learning领域取得了巨大的进步,这是一个融合计算机视觉,计算机图形学和机器学习的跨学科领域。今年的CVPR2019论文中也出现了不少3D的工作,详情可前往:CVPR 2019 论文汇总(按方向划分,更新中)

作者:timzhang642
来源:timzhang642/3D-Machine-Learning

本列表中作者还用了以下不同图标来表示不同的内容,可以说很可爱了|ू・ω・ )

  • :camera: Multi-view Images
  • :space_invader: Volumetric
  • :game_die: Point Cloud
  • :gem: Polygonal Mesh
  • :pill: Primitive-based


Table of Contents(本文较长,建议收藏阅读)

Available Courses

Stanford CS231A: Computer Vision-From 3D Reconstruction to Recognition (Winter 2018)

UCSD CSE291-I00: Machine Learning for 3D Data (Winter 2018)

Stanford CS468: Machine Learning for 3D Data (Spring 2017)

MIT 6.838: Shape Analysis (Spring 2017)

Princeton COS 526: Advanced Computer Graphics (Fall 2010)

Princeton CS597: Geometric Modeling and Analysis (Fall 2003)

Geometric Deep Learning

Paper Collection for 3D Understanding

CreativeAI: Deep Learning for Graphics

Datasets

To see a survey of RGBD datasets, check out Michael Firman's collection as well as the associated paper, RGBD Datasets: Past, Present and Future. Point Cloud Library also has a good dataset catalogue.

3D Models

Princeton Shape Benchmark (2003) [Link]

1,814 models collected from the web in .OFF format. Used to evaluating shape-based retrieval and analysis algorithms.

Dataset for IKEA 3D models and aligned images (2013) [Link]

759 images and 219 models including Sketchup (skp) and Wavefront (obj) files, good for pose estimation.

ikea_object.png

Open Surfaces: A Richly Annotated Catalog of Surface Appearance (SIGGRAPH 2013) [Link]

OpenSurfaces is a large database of annotated surfaces created from real-world consumer photographs. Our annotation framework draws on crowdsourcing to segment surfaces from photos, and then annotate them with rich surface properties, including material, texture and contextual information.

teaser4-web.jpg

PASCAL3D+ (2014) [Link]

12 categories, on average 3k+ objects per category, for 3D object detection and pose estimation.

pascal3d.png

ModelNet (2015) [Link]

127915 3D CAD models from 662 categories

ModelNet10: 4899 models from 10 categories

ModelNet40: 12311 models from 40 categories, all are uniformly orientated

thumbnail.jpg

ShapeNet (2015) [Link]

3Million+ models and 4K+ categories. A dataset that is large in scale, well organized and richly annotated.

ShapeNetCore [Link]: 51300 models for 55 categories.

shapenet.png

A Large Dataset of Object Scans (2016) [Link]

10K scans in RGBD + reconstructed 3D models in .PLY format.

teaser.jpg

ObjectNet3D: A Large Scale Database for 3D Object Recognition (2016) [Link]

100 categories, 90,127 images, 201,888 objects in these images and 44,147 3D shapes.

Tasks: region proposal generation, 2D object detection, joint 2D detection and 3D object pose estimation, and image-based 3D shape retrieval

ObjectNet3D.png

Thingi10K: A Dataset of 10,000 3D-Printing Models (2016) [Link]

10,000 models from featured “things” on thingiverse.com, suitable for testing 3D printing techniques such as structural analysis , shape optimization, or solid geometry operations.

DRbxWnqXkAEEH0g.jpg:large

ABC: A Big CAD Model Dataset For Geometric Deep Learning [Link][Paper]

This work introduce a dataset for geometric deep learning consisting of over 1 million individual (and high quality) geometric models, each associated with accurate ground truth information on the decomposition into patches, explicit sharp feature annotations, and analytic differential properties.

abc-dataset.png

3D Scenes

NYU Depth Dataset V2 (2012) [Link]

1449 densely labeled pairs of aligned RGB and depth images from Kinect video sequences for a variety of indoor scenes.

nyu_depth_v2_labeled.jpg

SUNRGB-D 3D Object Detection Challenge [Link]

19 object categories for predicting a 3D bounding box in real world dimension

Training set: 10,355 RGB-D scene images, Testing set: 2860 RGB-D images

3dbox.png

SceneNN (2016) [Link]

100+ indoor scene meshes with per-vertex and per-pixel annotation.

20170611155625.png

ScanNet (2017) [Link]

An RGB-D video dataset containing 2.5 million views in more than 1500 scans, annotated with 3D camera poses, surface reconstructions, and instance-level semantic segmentations.

annotations.png

Matterport3D: Learning from RGB-D Data in Indoor Environments (2017) [Link]

10,800 panoramic views (in both RGB and depth) from 194,400 RGB-D images of 90 building-scale scenes of private rooms. Instance-level semantic segmentations are provided for region (living room, kitchen) and object (sofa, TV) categories.

teaser.png

SUNCG: A Large 3D Model Repository for Indoor Scenes (2017) [Link]

The dataset contains over 45K different scenes with manually created realistic room and furniture layouts. All of the scenes are semantically annotated at the object level.

data_full.png

MINOS: Multimodal Indoor Simulator (2017) [Link]

MINOS is a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments. MINOS leverages large datasets of complex 3D environments and supports flexible configuration of multimodal sensor suites. MINOS supports SUNCG and Matterport3D scenes.

MINOS.jpg

Facebook House3D: A Rich and Realistic 3D Environment (2017) [Link]

House3D is a virtual 3D environment which consists of 45K indoor scenes equipped with a diverse set of scene types, layouts and objects sourced from the SUNCG dataset. All 3D objects are fully annotated with category labels. Agents in the environment have access to observations of multiple modalities, including RGB images, depth, segmentation masks and top-down 2D map views.

33509559-87c4e470-d6b7-11e7-8266-27c940d5729a.jpg

HoME: a Household Multimodal Environment (2017) [Link]

HoME integrates over 45,000 diverse 3D house layouts based on the SUNCG dataset, a scale which may facilitate learning, generalization, and transfer. HoME is an open-source, OpenAI Gym-compatible platform extensible to tasks in reinforcement learning, language grounding, sound-based navigation, robotics, multi-agent learning.

overview.png

AI2-THOR: Photorealistic Interactive Environments for AI Agents [Link]

AI2-THOR is a photo-realistic interactable framework for AI agents. There are a total 120 scenes in version 1.0 of the THOR environment covering four different room categories: kitchens, living rooms, bedrooms, and bathrooms. Each room has a number of actionable objects.

AI2-Thor.jpeg

UnrealCV: Virtual Worlds for Computer Vision (2017) [Link][Paper]

An open source project to help computer vision researchers build virtual worlds using Unreal Engine 4.

homepage_teaser.png

Gibson Environment: Real-World Perception for Embodied Agents (2018 CVPR) [Link]

This platform provides RGB from 1000 point clouds, as well as multimodal sensor data: surface normal, depth, and for a fraction of the spaces, semantics object annotations. The environment is also RL ready with physics integrated. Using such datasets can further narrow down the discrepency between virtual environment and real world.

Gibson%20Environment-%20Real-World%20Perception%20for%20Embodied%20Agents%20(2018%20CVPR)%20.jpeg

InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset [Link]

System Overview: an end-to-end pipeline to render an RGB-D-inertial benchmark for large scale interior scene understanding and mapping. Our dataset contains 20M images created by pipeline: (A) We collect around 1 million CAD models provided by world-leading furniture manufacturers. These models have been used in the real-world production. (B) Based on those models, around 1,100 professional designers create around 22 million interior layouts. Most of such layouts have been used in real-world decorations. (C) For each layout, we generate a number of configurations to represent different random lightings and simulation of scene change over time in daily life. (D) We provide an interactive simulator (ViSim) to help for creating ground truth IMU, events, as well as monocular or stereo camera trajectories including hand-drawn, random walking and neural network based realistic trajectory. (E) All supported image sequences and ground truth.

InteriorNet.jpg

Semantic3D[Link]

Large-Scale Point Cloud Classification Benchmark, which provides a large labelled 3D point cloud data set of natural scenes with over 4 billion points in total, and also covers a range of diverse urban scenes.

sg27_8.jpg

3D Pose Estimation

Category-Specific Object Reconstruction from a Single Image (2014) [Paper]

basisshapes_highres.png

Viewpoints and Keypoints (2015) [Paper]

Viewpoints%20and%20Keypoints.jpeg

Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views (2015 ICCV) [Paper]

teaser.jpg

PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization (2015) [Paper]

map.png

Modeling Uncertainty in Deep Learning for Camera Relocalization (2016) [Paper]

Modeling%20Uncertainty%20in%20Deep%20Learning%20for%20Camera%20Relocalization.jpeg

Robust camera pose estimation by viewpoint classification using deep learning (2016) [Paper]

Robust%20camera%20pose%20estimation%20by%20viewpoint%20classification%20using%20deep%20learning.jpeg

Geometric loss functions for camera pose regression with deep learning (2017 CVPR) [Paper]

pose-net.png

Generic 3D Representation via Pose Estimation and Matching (2017) [Paper]

Generic%203D%20Representation%20via%20Pose%20Estimation%20and%20Matching.jpeg

3D Bounding Box Estimation Using Deep Learning and Geometry (2017) [Paper]

3D%20Bounding%20Box%20Estimation%20Using%20Deep%20Learning%20and%20Geometry.png

6-DoF Object Pose from Semantic Keypoints (2017) [Paper]

object3d-teaser.png

Relative Camera Pose Estimation Using Convolutional Neural Networks (2017) [Paper]

Relative%20Camera%20Pose%20Estimation%20Using%20Convolutional%20Neural%20Networks.png

3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions (2017) [Paper]

overview.jpg

Single Image 3D Interpreter Network (2016) [Paper] [Code]

spotlight_3dinn_large.jpg

Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction (2018 CVPR) [Paper]

teaser.png

PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes (2018) [Paper]

PoseCNN.png

Feature Mapping for Learning Fast and Accurate 3D Pose Inference from Synthetic Images (2018 CVPR) [Paper]

images?q=tbn:ANd9GcTnpyajEhbhrPMc0YpEQzqE8N9E7CW_EVWYA3Bxg46oUEYFf9XvkA

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018 CVPR) [Paper]

spotlight_pix3d.jpg

3D Pose Estimation and 3D Model Retrieval for Objects in the Wild (2018 CVPR) [Paper]

pose_retrieval_overview.png

Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects (2018) [Paper]

forwebsite1_0.png

Single Object Classification

:space_invader: 3D ShapeNets: A Deep Representation for Volumetric Shapes (2015) [Paper]

1-Figure1-1.png

:space_invader: VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition (2015) [Paper] [Code]

car_voxnet_side.png

:camera: Multi-view Convolutional Neural Networks for 3D Shape Recognition (2015) [Paper]

mvcnn.png

:camera: DeepPano: Deep Panoramic Representation for 3-D Shape Recognition (2015) [Paper]

1-Figure3-1.png

:space_invader::camera: FusionNet: 3D Object Classification Using Multiple Data Representations (2016) [Paper]

6-Figure5-1.png

:space_invader::camera: Volumetric and Multi-View CNNs for Object Classification on 3D Data (2016) [Paper] [Code]

teaser.jpg

:space_invader: Generative and Discriminative Voxel Modeling with Convolutional Neural Networks (2016) [Paper] [Code]

brock_vae.png

:gem: Geometric deep learning on graphs and manifolds using mixture model CNNs (2016) [Link]

monet.png?resize=581%2C155&ssl=1

:space_invader: 3D GAN: Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016) [Paper] [Code]

model.jpg

:space_invader: Generative and Discriminative Voxel Modeling with Convolutional Neural Networks (2017) [Paper]

GUI3.png

:space_invader: FPNN: Field Probing Neural Networks for 3D Data (2016) [Paper] [Code]

1-Figure2-1.png

:space_invader: OctNet: Learning Deep 3D Representations at High Resolutions (2017) [Paper] [Code]

img03.png

:space_invader: O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis (2017) [Paper] [Code]

teaser.png

:space_invader: Orientation-boosted voxel nets for 3D object recognition (2017) [Paper] [Code]

teaser_w.png

:game_die: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation (2017) [Paper] [Code]

pointnet.png

:game_die: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017) [Paper] [Code]

PointNet%2B%2B-%20Deep%20Hierarchical%20Feature%20Learning%20on%20Point%20Sets%20in%20a%20Metric%20Space.png

:camera: Feedback Networks (2017) [Paper] [Code]

Feedback%20Networks.png

:game_die: Escape from Cells: Deep Kd-Networks for The Recognition of 3D Point Cloud Models (2017) [Paper]

Escape%20From%20Cells.png

:game_die: Dynamic Graph CNN for Learning on Point Clouds (2018) [Paper]

dynamicgcnn_logo.png

:game_die: PointCNN (2018) [Paper]

pointcnn.png

:game_die::camera: A Network Architecture for Point Cloud Classification via Automatic Depth Images Generation (2018 CVPR) [Paper]

A-Network-Architecture-for-Point-Cloud-Classification-via-Automatic-Depth-Images-Generation-Image-600x317.jpg

:game_die::space_invader: PointGrid: A Deep Network for 3D Shape Understanding (CVPR 2018) [Paper] [Code]

PointGrid-%20A%20Deep%20Network%20for%203D%20Shape%20Understanding%20(2018).jpeg

:gem: MeshNet: Mesh Neural Network for 3D Shape Representation (AAAI 2019) [Paper] [Code]

meshnet.jpg

Multiple Objects Detection

Sliding Shapes for 3D Object Detection in Depth Images (2014) [Paper]

teaser.jpg

Object Detection in 3D Scenes Using CNNs in Multi-view Images (2016) [Paper]

Object%20Detection%20in%203D%20Scenes%20Using%20CNNs%20in%20Multi-view%20Images.png

Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images (2016) [Paper] [Code]

DSS.jpg

DeepContext: Context-Encoding Neural Pathways for 3D Holistic Scene Understanding (2016) [Paper]

teaser.png

SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite (2017) [Paper]

teaser.jpg

VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection (2017) [Paper]

DPMtLhHXUAcQUj2.jpg

Frustum PointNets for 3D Object Detection from RGB-D Data (CVPR2018) [Paper]

teaser.jpg

A^2-Net: Molecular Structure Estimation from Cryo-EM Density Volumes (AAAI2019) [Paper]

a-square-net-min.jpg

Stereo R-CNN based 3D Object Detection for Autonomous Driving (CVPR2019) [Paper]

system_newnew.png

Scene/Object Semantic Segmentation

Learning 3D Mesh Segmentation and Labeling (2010) [Paper]

7-Figure7-1.png

Unsupervised Co-Segmentation of a Set of Shapes via Descriptor-Space Spectral Clustering (2011) [Paper]

results6.png

Single-View Reconstruction via Joint Analysis of Image and Shape Collections (2015) [Paper] [Code]

single-view.png

3D Shape Segmentation with Projective Convolutional Networks (2017) [Paper] [Code]

teaser.jpg

Learning Hierarchical Shape Segmentation and Labeling from Online Repositories (2017) [Paper]

teaser.jpg

:space_invader: ScanNet (2017) [Paper] [Code]

voxel-predictions.jpg

:game_die: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation (2017) [Paper] [Code]

pointnet.png

:game_die: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017) [Paper] [Code]

PointNet%2B%2B-%20Deep%20Hierarchical%20Feature%20Learning%20on%20Point%20Sets%20in%20a%20Metric%20Space.png

:game_die: 3D Graph Neural Networks for RGBD Semantic Segmentation (2017) [Paper]

66372-20171018115809740-2125227250.jpg

:game_die: 3DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic
Parsing of Large-scale 3D Point Clouds (2017)
[Paper]

3DCNN-DQN-RNN.png

:game_die::space_invader: Semantic Segmentation of Indoor Point Clouds using Convolutional Neural Networks (2017) [Paper]

Semantic%20Segmentation%20of%20Indoor%20Point%20Clouds%20using%20Convolutional%20Neural%20Networks.png

:game_die::space_invader: SEGCloud: Semantic Segmentation of 3D Point Clouds (2017) [Paper]

SEGCloud.png

:game_die::space_invader: Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55 (2017) [Paper]

Core55.png

:game_die: Dynamic Graph CNN for Learning on Point Clouds (2018) [Paper]

dynamicgcnn_logo.png

:game_die: PointCNN (2018) [Paper]

pointcnn.png

:camera::space_invader: 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018) [Paper]

teaser.jpg

:space_invader: ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans (2018) [Paper]

teaser.jpg

:game_die::camera: SPLATNet: Sparse Lattice Networks for Point Cloud Processing (2018) [Paper]

SPLATNet-%20Sparse%20Lattice%20Networks%20for%20Point%20Cloud%20Processing.jpeg

:game_die::space_invader: PointGrid: A Deep Network for 3D Shape Understanding (CVPR 2018) [Paper] [Code]

PointGrid-%20A%20Deep%20Network%20for%203D%20Shape%20Understanding%20(2018).jpeg

3D Model Synthesis/Reconstruction

Parametric Morphable Model-based methods

A Morphable Model For The Synthesis Of 3D Faces (1999) [Paper][Code]

031717_0222_DataDrivenS4.png?type=w420

The Space of Human Body Shapes: Reconstruction and Parameterization from Range Scans (2003) [Paper]

7-Figure10-1.png

Category-Specific Object Reconstruction from a Single Image (2014) [Paper]

teaser.png

:game_die: DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image (2017) [Paper]

model.png

:gem: Mesh-based Autoencoders for Localized Deformation Component Analysis (2017) [Paper]

point_conv.jpg

:gem: Exploring Generative 3D Shapes Using Autoencoder Networks (Autodesk 2017) [Paper]

Exploring%20Generative%203D%20Shapes%20Using%20Autoencoder%20Networks.jpeg

:gem: Using Locally Corresponding CAD Models for
Dense 3D Reconstructions from a Single Image (2017)
[Paper]

r02.png

:gem: Compact Model Representation for 3D Reconstruction (2017) [Paper]

overview.png

:gem: Image2Mesh: A Learning Framework for Single Image 3D Reconstruction (2017) [Paper]

DW5VhjpW4AAESHO.jpg

:gem: Learning free-form deformations for 3D object reconstruction (2018) [Paper]

learning_ffd_overview.png

:gem: Variational Autoencoders for Deforming 3D Mesh Models(2018 CVPR) [Paper]

TeaserImage.jpg

:gem: Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape from Images (2018 CVPR) [Paper]

realisticava.jpg

Part-based Template Learning methods

Modeling by Example (2004) [Paper]

chair.jpg

Model Composition from Interchangeable Components (2007) [Paper]

teaser.jpg

Data-Driven Suggestions for Creativity Support in 3D Modeling (2010) [Paper]

creativity.png

Photo-Inspired Model-Driven 3D Object Modeling (2011) [Paper]

overview.PNG

Probabilistic Reasoning for Assembly-Based 3D Modeling (2011) [Paper]

highlight9.png

A Probabilistic Model for Component-Based Shape Synthesis (2012) [Paper]

A%20Probabilistic%20Model%20for%20Component-Based%20Shape%20Synthesis.png

Structure Recovery by Part Assembly (2012) [Paper]

Structure%20Recovery%20by%20Part%20Assembly.png

Fit and Diverse: Set Evolution for Inspiring 3D Shape Galleries (2012) [Paper]

teaser.png

AttribIt: Content Creation with Semantic Attributes (2013) [Paper]

teaser.jpg

Learning Part-based Templates from Large Collections of 3D Shapes (2013) [Paper]

Learning%20Part-based%20Templates%20from%20Large%20Collections%20of%203D%20Shapes.png

Topology-Varying 3D Shape Creation via Structural Blending (2014) [Paper]

maxresdefault.jpg

Estimating Image Depth using Shape Collections (2014) [Paper]

pipeline.jpg

Single-View Reconstruction via Joint Analysis of Image and Shape Collections (2015) [Paper]

single-view.png

Interchangeable Components for Hands-On Assembly Based Modeling (2016) [Paper]

Interchangeable%20Components%20for%20Hands-On%20Assembly%20Based%20Modeling.png

Shape Completion from a Single RGBD Image (2016) [Paper]

completion.jpg

Deep Learning Methods

camera: Learning to Generate Chairs, Tables and Cars with Convolutional Networks (2014) [Paper]

chairs-model.png

camera: Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis (2015, NIPS) [Paper]

demo_img.png

game_die: Analysis and synthesis of 3D shape families via deep-learned generative models of surfaces (2015) [Paper]

bsm_teaser.jpg

camera: Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis (2015) [Paper] [Code]

2-Figure1-1.png

camera: Multi-view 3D Models from Single Images with a Convolutional Network (2016) [Paper] [Code]

4-Figure2-1.png

camera: View Synthesis by Appearance Flow (2016) [Paper] [Code]

6-Figure2-1.png

space_invader: Voxlets: Structured Prediction of Unobserved Voxels From a Single Depth Image (2016) [Paper] [Code]

maxresdefault.jpg

space_invader: 3D-R2N2: 3D Recurrent Reconstruction Neural Network (2016) [Paper] [Code]

overview.png

space_invader: Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision (2016) [Paper]

network_arch.png

space_invader: TL-Embedding Network: Learning a Predictable and Generative Vector Representation for Objects (2016) [Paper]

webteaser.jpg

space_invader: 3D GAN: Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016) [Paper]

model.jpg

space_invader: 3D Shape Induction from 2D Views of Multiple Objects (2016) [Paper]

2-Figure2-1.png

camera: Unsupervised Learning of 3D Structure from Images (2016) [Paper]

unsupervised-3d-fig-10.jpeg?w=600

space_invader: Generative and Discriminative Voxel Modeling with Convolutional Neural Networks (2016) [Paper] [Code]

brock_vae.png

camera: Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency (2017) [Paper]

teaserChair.png

camera: Synthesizing 3D Shapes via Modeling Multi-View Depth Maps and Silhouettes with Deep Generative Networks (2017) [Paper] [Code]

spotlight_3dvae.jpg

space_invader: Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis (2017) [Paper] [Code]

teaser.jpg

space_invader: Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs (2017) [Paper] [Code]

2-Figure2-1.png

space_invader: Hierarchical Surface Prediction for 3D Object Reconstruction (2017) [Paper]

image_2.png

space_invader: OctNetFusion: Learning Depth Fusion from Data (2017) [Paper] [Code]

OctNetFusion-%20Learning%20Depth%20Fusion%20from%20Data.jpeg

game_die: A Point Set Generation Network for 3D Object Reconstruction from a Single Image (2017) [Paper] [Code]

A%20Point%20Set%20Generation%20Network%20for%203D%20Object%20Reconstruction%20from%20a%20Single%20Image%20(2017).jpeg

game_die: Learning Representations and Generative Models for 3D Point Clouds (2017) [Paper] [Code]

teaser.jpg

game_die: Shape Generation using Spatially Partitioned Point Clouds (2017) [Paper]

abstract.png

game_die: PCPNET Learning Local Shape Properties from Raw Point Clouds (2017) [Paper]

PCPNET%20Learning%20Local%20Shape%20Properties%20from%20Raw%20Point%20Clouds%20(2017).jpeg

camera: Transformation-Grounded Image Generation Network for Novel 3D View Synthesis (2017) [Paper] [Code]

view_synthesis.gif

camera: Tag Disentangled Generative Adversarial Networks for Object Image Re-rendering (2017) [Paper]

Tag%20Disentangled%20Generative%20Adversarial%20Networks%20for%20Object%20Image%20Re-rendering.jpeg

camera: 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks (2017) [Paper] [Code]

SketchModeling_teaser.png

space_invader: Interactive 3D Modeling with a Generative Adversarial Network (2017) [Paper]

DCsPKLqXoAEBd-V.jpg

camera::space_invader: Weakly supervised 3D Reconstruction with Adversarial Constraint (2017) [Paper] [Code]

Weakly%20supervised%203D%20Reconstruction%20with%20Adversarial%20Constraint%20(2017).jpeg

camera: SurfNet: Generating 3D shape surfaces using deep residual networks (2017) [Paper]

Screenshot-from-2017-07-26-145521-e1501077539723.png

pill: GRASS: Generative Recursive Autoencoders for Shape Structures (SIGGRAPH 2017) [Paper] [Code] [code]

teaser.jpg

pill: 3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks (2017) [Paper][code]

teasor.jpg

gem: Neural 3D Mesh Renderer (2017) [Paper] [Code]

DPSm-4HWkAApEZd.jpg

game_die::space_invader: Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55 (2017) [Paper]

Core55.png

space_invader: Pix2vox: Sketch-Based 3D Exploration with Stacked Generative Adversarial Networks (2017) [Code]

sample.gif

camera::space_invader: What You Sketch Is What You Get: 3D Sketching using Multi-View Deep Volumetric Prediction (2017) [Paper]

x1.png

camera::space_invader: MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017) [Paper]

model.jpg

camera::space_invader::game_die: Learning a Multi-View Stereo Machine (2017 NIPS) [Paper]

Network.png

space_invader: 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions (2017) [Paper]

overview.jpg

space_invader: Scaling CNNs for High Resolution Volumetric Reconstruction from a Single Image (2017) [Paper]

Scaling%20CNN%20Reconstruction.png

pill: ComplementMe: Weakly-Supervised Component Suggestions for 3D Modeling (2017) [Paper]

figure_2.png

game_die: PU-Net: Point Cloud Upsampling Network (2018) [Paper] [Code]

Pu-Net.png

camera::space_invader: Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction (2018 CVPR) [Paper]

teaser.png

camera::game_die: Object-Centric Photometric Bundle Adjustment with Deep Shape Prior (2018) [Paper]

r06.png

camera::game_die: Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction (2018 AAAI) [Paper]

r05.png

gem: Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018) [Paper]

x2.png.750x0_q75_crop.png

gem: AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation (2018 CVPR) [Paper] [Code]

teaser.small.png

space_invader::gem: Deep Marching Cubes: Learning Explicit Surface Representations (2018 CVPR) [Paper]

Deep%20Marching%20Cubes.png

space_invader: Im2Avatar: Colorful 3D Reconstruction from a Single Image (2018) [Paper]

teaser.png

gem: Learning Category-Specific Mesh Reconstruction from Image Collections (2018) [Paper]

teaser.png

pill: CSGNet: Neural Shape Parser for Constructive Solid Geometry (2018) [Paper]

DR-RgbaU8AEyjeW.jpg

space_invader: Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings (2018) [Paper]

pull.png

space_invader::gem::camera: Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018) [Paper] [Code]

decomposition_new.pngMulti-View%20Silhouette%20and%20Depth%20Decomposition%20for%20High%20Resolution%203D%20Object%20Representation.png

space_invader::gem::camera: Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction (2018 CVPR) [Paper]

pixels-voxels-views-rgb2mesh.png

camera::game_die: Neural scene representation and rendering (2018) [Paper]

gqn_image.png

pill: Im2Struct: Recovering 3D Shape Structure from a Single RGB Image (2018 CVPR) [Paper]

niu_cvpr18.jpg

game_die: FoldingNet: Point Cloud Auto-encoder via Deep Grid Deformation (2018 CVPR) [Paper]

FoldingNet.jpg

camera::space_invader: Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018 CVPR) [Paper]

Pix3D%20-%20Dataset%20and%20Methods%20for%20Single-Image%203D%20Shape%20Modeling%20(2018%20CVPR).png

gem: 3D-RCNN: Instance-level 3D Object Reconstruction via Render-and-Compare (2018 CVPR) [Paper]

3D-RCNN-%20Instance-level%203D%20Object%20Reconstruction%20via%20Render-and-Compare%20(2018%20CVPR).jpeg

space_invader: Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers (2018 CVPR) [Paper]

Matryoshka%20Networks-%20Predicting%203D%20Geometry%20via%20Nested%20Shape%20Layers%20(2018%20CVPR).jpeg

space_invader: Global-to-Local Generative Model for 3D Shapes (SIGGRAPH Asia 2018) [Paper]

Global-to-Local%20Generative%20Model%20for%203D%20Shapes.jpg

gem::game_die::space_invader: ALIGNet: Partial-Shape Agnostic Alignment via Unsupervised Learning (TOG 2018) [Paper] [Code]

rep.png

game_die::space_invader: PointGrid: A Deep Network for 3D Shape Understanding (CVPR 2018) [Paper] [Code]

PointGrid-%20A%20Deep%20Network%20for%203D%20Shape%20Understanding%20(2018).jpeg

game_die: GAL: Geometric Adversarial Loss for Single-View 3D-Object Reconstruction (2018) [Paper]

474213_1_En_49_Fig2_HTML.gif

game_die: Visual Object Networks: Image Generation with Disentangled 3D Representation (2018) [Paper]

Visual%20Object%20Networks-%20Image%20Generation%20with%20Disentangled%203D%20Representation%20(2018).jpeg

space_invader: Learning to Infer and Execute 3D Shape Programs (2019)) [Paper]

teaser.jpg

space_invader: Learning to Infer and Execute 3D Shape Programs (2019)) [Paper]

DxFaW-mU8AEo9wc.jpg

gem: Learning View Priors for Single-view 3D Reconstruction (CVPR 2019) [Paper]

Texture/Material Analysis and Synthesis

Texture Synthesis Using Convolutional Neural Networks (2015) [Paper]

Texture%20Synthesis%20Using%20Convolutional%20Neural%20Networks.jpeg

Two-Shot SVBRDF Capture for Stationary Materials (SIGGRAPH 2015) [Paper]

teaser.png

Reflectance Modeling by Neural Texture Synthesis (2016) [Paper]

teaser.png

Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks (2017) [Paper]

teaser.jpg

High-Resolution Multi-Scale Neural Texture Synthesis (2017) [Paper]

multiscale-gram-marble.jpg

Reflectance and Natural Illumination from Single Material Specular Objects Using Deep Learning (2017) [Paper]

tpami17_teaser2.png

Joint Material and Illumination Estimation from Photo Sets in the Wild (2017) [Paper]

Joint%20Material%20and%20Illumination%20Estimation%20from%20Photo%20Sets%20in%20the%20Wild.jpeg

JWhat Is Around The Camera? (2017) [Paper]

arxiv16b_teaser.jpg

TextureGAN: Controlling Deep Image Synthesis with Texture Patches (2018 CVPR) [Paper]

paper_figure.png

Gaussian Material Synthesis (2018 SIGGRAPH) [Paper]

maxresdefault.jpg

Non-stationary Texture Synthesis by Adversarial Expansion (2018 SIGGRAPH) [Paper]

20180424130538_128.jpg

Synthesized Texture Quality Assessment via Multi-scale Spatial and Statistical Texture Attributes of Image and Gradient Magnitude Coefficients (2018 CVPR) [Paper]

39275366-e18c7c1c-4899-11e8-8e61-05072618bbce.PNG

LIME: Live Intrinsic Material Estimation (2018 CVPR) [Paper]

teaser.png

Single-Image SVBRDF Capture with a Rendering-Aware Deep Network (2018) [Paper]

teaser_v0.png

PhotoShape: Photorealistic Materials for Large-Scale Shape Collections (2018) [Paper]

teaser.jpg

Learning Material-Aware Local Descriptors for 3D Shapes (2018) [Paper]

Learning%20Material-Aware%20Local%20Descriptors%20for%203D%20Shapes%20(2018).jpeg

FrankenGAN: Guided Detail Synthesis for Building Mass Models
using Style-Synchonized GANs (2018 SIGGRAPH Asia)
[Paper]

teaser.jpg

Style Learning and Transfer

Style-Content Separation by Anisotropic Part Scales (2010) [Paper]

style_b.jpg?height=145&width=400

Design Preserving Garment Transfer (2012) [Paper]

02_WomanToAll.jpg

Analogy-Driven 3D Style Transfer (2014) [Paper]

2014_st_teaser.png

Elements of Style: Learning Perceptual Shape Style Similarity (2015) [Paper] [Code]

StyleSimilarity_teaser.jpg

Functionality Preserving Shape Style Transfer (2016) [Paper] [Code]

StyleTransfer_teaser.jpg

Unsupervised Texture Transfer from Images to Model Collections (2016) [Paper]

teaser.png

Learning Detail Transfer based on Geometric Features (2017) [Paper]

teaser.png

Co-Locating Style-Defining Elements on 3D Shapes (2017) [Paper]

c118-e100-publicimage_0-itok=yO8OegQO.png

Neural 3D Mesh Renderer (2017) [Paper] [Code]

DPSm-4HWkAApEZd.jpg

Appearance Modeling via Proxy-to-Image Alignment (2018) [Paper]

20171026180502_864.jpg

:gem: Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images (2018) [Paper]

pipeline_01.jpg

Scene Synthesis/Reconstruction

Make It Home: Automatic Optimization of Furniture Arrangement (2011, SIGGRAPH) [Paper]

furniture.gif

Interactive Furniture Layout Using Interior Design Guidelines (2011) [Paper]

furnitureBig.jpg

Synthesizing Open Worlds with Constraints using Locally Annealed Reversible Jump MCMC (2012) [Paper]

Synthesizing%20Open%20Worlds%20with%20Constraints%20using%20Locally%20Annealed%20Reversible%20Jump%20MCMC%20(2012).jpeg

Example-based Synthesis of 3D Object Arrangements (2012 SIGGRAPH Asia) [Paper]

teaser.jpg

Sketch2Scene: Sketch-based Co-retrieval and Co-placement of 3D Models (2013) [Paper]

sketch2scene_thumb.jpg

Action-Driven 3D Indoor Scene Evolution (2016) [Paper]

teaser.jpg

The Clutterpalette: An Interactive Tool for Detailing Indoor Scenes (2015) [Paper]

The%20Clutterpalette-%20An%20Interactive%20Tool%20for%20Detailing%20Indoor%20Scenes.png

Relationship Templates for Creating Scene Variations (2016) [Paper]

teaser.png

IM2CAD (2017) [Paper]

KhtOeuB.jpg

Predicting Complete 3D Models of Indoor Scenes (2017) [Paper]

Predicting%20Complete%203D%20Models%20of%20Indoor%20Scenes.png

Complete 3D Scene Parsing from Single RGBD Image (2017) [Paper]

Complete%203D%20Scene%20Parsing%20from%20Single%20RGBD%20Image.jpeg

Raster-to-Vector: Revisiting Floorplan Transformation (2017, ICCV) [Paper] [Code]

teaser.png

Fully Convolutional Refined Auto-Encoding Generative Adversarial Networks for 3D Multi Object Scenes (2017) [Blog]

1*NckW2hfgbHhEP3P8Z5ZLjQ.png

Adaptive Synthesis of Indoor Scenes via Activity-Associated Object Relation Graphs (2017 SIGGRAPH Asia) [Paper]

c121-e45-publicimage.jpg

Automated Interior Design Using a Genetic Algorithm (2017) [Paper]

teaserq.jpg

SceneSuggest: Context-driven 3D Scene Design (2017) [Paper]

SceneSuggest%20-Context-driven%203D%20Scene%20Design%20(2017).png

A fully end-to-end deep learning approach for real-time simultaneous 3D reconstruction and material recognition (2017) [Paper]

A%20fully%20end-to-end%20deep%20learning%20approach%20for%20real-time%20simultaneous%203D%20reconstruction%20and%20material%20recognition%20(2017).png

Human-centric Indoor Scene Synthesis Using Stochastic Grammar (2018, CVPR)[Paper] [Supplementary] [Code]

cvpr2018synthesis.gif

:camera::game_die: FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans (2018) [Paper] [Code]

teaser.png

:space_invader: ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans (2018) [Paper]

teaser.jpg

Deep Convolutional Priors for Indoor Scene Synthesis (2018) [Paper]

deepsynth.png

Configurable 3D Scene Synthesis and 2D Image Rendering
with Per-Pixel Ground Truth using Stochastic Grammars (2018)
[Paper]

11263_2018_1103_Fig5_HTML.jpg

Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image (ECCV 2018) [Paper]

eccv2018scene.png

Language-Driven Synthesis of 3D Scenes from Scene Databases (SIGGRAPH Asia 2018) [Paper]

teaser.png

Deep Generative Modeling for Scene Synthesis via Hybrid Representations (2018) [Paper]

Deep%20Generative%20Modeling%20for%20Scene%20Synthesis%20via%20Hybrid%20Representations%20(2018).jpeg

GRAINS: Generative Recursive Autoencoders for INdoor Scenes (2018) [Paper]

teaserfig.jpg.750x0_q75_crop.jpg

SEETHROUGH: Finding Objects in Heavily Occluded Indoor Scene Images (2018) [Paper]

result_plate.png

Scene Understanding

Recovering the Spatial Layout of Cluttered Rooms (2009) [Paper]

Recovering%20the%20Spatial%20Layout%20of%20Cluttered%20Rooms.png

Characterizing Structural Relationships in Scenes Using Graph Kernels (2011 SIGGRAPH) [Paper]

graphKernelTeaser.png

Understanding Indoor Scenes Using 3D Geometric Phrases (2013) [Paper]

title.png

Organizing Heterogeneous Scene Collections through Contextual Focal Points (2014 SIGGRAPH) [Paper]

overlapping_clusters.jpg

SceneGrok: Inferring Action Maps in 3D Environments (2014, SIGGRAPH) [Paper]

scenegrok.png

PanoContext: A Whole-room 3D Context Model for Panoramic Scene Understanding (2014) [Paper]

teaser.jpg

Learning Informative Edge Maps for Indoor Scene Layout Prediction (2015) [Paper]

Learning%20Informative%20Edge%20Maps%20for%20Indoor%20Scene%20Layout%20Prediction.png

Rent3D: Floor-Plan Priors for Monocular Layout Estimation (2015) [Paper]

layout-res.jpg

A Coarse-to-Fine Indoor Layout Estimation (CFILE) Method (2016) [Paper]

A%20Coarse-to-Fine%20Indoor%20Layout%20Estimation%20(CFILE)%20Method%20(2016).png

DeLay: Robust Spatial Layout Estimation for Cluttered Indoor Scenes (2016) [Paper]

DeLay-Robust%20Spatial%20Layout%20Estimation%20for%20Cluttered%20Indoor%20Scenes.png

3D Semantic Parsing of Large-Scale Indoor Spaces (2016) [Paper] [Code]

teaser.png

Single Image 3D Interpreter Network (2016) [Paper] [Code]

spotlight_3dinn_large.jpg

Deep Multi-Modal Image Correspondence Learning (2016) [Paper]

2016-floorplan-wide.png

Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks (2017) [Paper] [Code] [Code] [Code] [Code]

teaser.jpg

RoomNet: End-to-End Room Layout Estimation (2017) [Paper]

C7Z29GsV0AASEvR.jpg

SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite (2017) [Paper]

teaser.jpg

Semantic Scene Completion from a Single Depth Image (2017) [Paper] [Code]

teaser.jpg

Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene (2018 CVPR) [Paper] [Code]

teaser.png

LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image (2018 CVPR) [Paper] [Code]

A1D0CAE48130C918FE624FA60495F237C67172F6_size63_w797_h755.jpeg

PlaneNet: Piece-wise Planar Reconstruction from a Single RGB Image (2018 CVPR) [Paper] [Code]

planenet.png

Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery (2018 CVPR) [Paper]

concept_pic.png

Pano2CAD: Room Layout From A Single Panorama Image (2018 CVPR) [Paper]

Compare_2b.png

Automatic 3D Indoor Scene Modeling from Single Panorama (2018 CVPR) [Paper]

Automatic%203D%20Indoor%20Scene%20Modeling%20from%20Single%20Panorama%20(2018%20CVPR).jpeg

Single-Image Piece-wise Planar 3D Reconstruction via Associative Embedding (2019 CVPR) [Paper] [Code]

pipeline.jpg

3D-Aware Scene Manipulation via Inverse Graphics (NeurIPS 2018) [Paper] [Code]

teaser.png