Neural Shape Parsers for
Constructive Solid Geometry
Abstract
Constructive Solid Geometry (CSG) is a geometric modeling technique that defines complex shapes by recursively applying boolean operations on primitives such as spheres and cylinders. We present CSGNet, a deep network architecture that takes as input a 2D or 3D shape and outputs a CSG program that models it. Parsing shapes into CSG programs is desirable as it yields a compact and interpretable generative model. However, the task is challenging since the space of primitives and their combinations can be prohibitively large. CSGNet uses a convolutional encoder and recurrent decoder based on deep networks to map shapes to modeling instructions in a feedforward manner and is significantly faster than bottomup approaches. We investigate two architectures for this task — a vanilla encoder (CNN)  decoder (RNN) and another architecture that augments the encoder with an explicit memory module based on the program execution stack. The stack augmentation improves the reconstruction quality of the generated shape and learning efficiency. Our approach is also more effective as a shape primitive detector compared to a stateoftheart object detector. Finally, we demonstrate CSGNet can be trained on novel datasets without program annotations through policy gradient techniques.
1 Introduction
In recent years, there has been a growing interest in generative models of 2D or 3D shapes, especially through the use of deep neural networks as image or shape priors. However, current methods are limited to the generation of lowlevel shape representations consisting of pixels, voxels, or points. Human designers, on the other hand, rarely model shapes as a collection of these individual elements. For example, in vector graphics modeling packages (e.g., Inkscape, Illustrator, and so on), shapes are often created through higherlevel primitives, such as parametric curves (e.g., Bezier curves) or basic shapes (e.g., circles, polygons), as well as operations acting on these primitives, such as boolean operations, deformations, extrusions, and so on. Describing shapes with higherlevel primitives and operations is highly desirable for designers since it is compact and makes subsequent editing easier. It may also better capture certain aspects of human shape perception such as view invariance, compositionality, and symmetry [biederman1987recognition].
The goal of our work is to develop an algorithm that parses shapes into their constituent modeling primitives and operations within the framework of Constructive Solid Geometry (CSG) [Laidlaw1986CSG]. CSG is a popular geometric modeling framework where shapes are generated by recursively applying boolean operations, such as union or intersection, on simple geometric primitives, such as spheres or cylinders. Figure 1 illustrates an example where a 2D shape (top) and a 3D shape (bottom) are generated as a sequence of operations over primitives or a visual program. Yet, parsing a shape into its CSG program poses a number of challenges. First, the number of primitives and operations is not the same for all shapes i.e., our output does not have constant dimensionality, as in the case of pixel arrays, voxel grids, or fixed point sets. Second, the order of these instructions matter — small changes in the order of operations can significantly change the generated shape. Third, the number of possible programs grows exponentially with program length.
Existing approaches for CSG parsing are predominantly searchbased. A significant portion of related literature has focused on approaches to efficiently estimate primitives in a bottomup manner, and to search for their combinations using heuristic optimization. While these techniques can generate complex shapes, they are prone to noise in the input and are generally slow. Our contribution is a neural network architecture called CSGNet that generates the program in a feedforward manner. The approach is inspired by the ability of deep networks for generative sequence modeling such as for speech and language. As a result CSGNet is efficient at test time, as it can be viewed as an amortized search [Gershman:2014wt] procedure. Furthermore, it be used as an initialization for searchbased approaches leading to improvements in accuracy at the cost of computation.
At a highlevel, CSGNet is an encoderdecoder architecture that encodes the input shape using a convolutional network and decodes it into a sequence of instructions using a recurrent network (Figure 2). It is trained on a large synthetic dataset of automatically generated 2D and 3D programs (Table II). However, this leads to poor generalization when applied to new domains. To adapt models to new domains without program annotations, we employ policy gradient techniques from the reinforcement learning literature [Williams92simplestatistical]. Combining the parser with a CSG rendering engine allows the networks to receive feedback based on the visual difference between the input and generated shape, and the parser is trained to minimize this difference (Figure 2). Furthermore, we investigate two network architectures: a vanilla recurrent network (CSGNet), and a new variant (CSGNetStack) that incorporates explicit memory in a form of a stack, as seen in Figure 3. The memory is based on the execution stack of the CSG engine and enables explicit modeling of the intermediate program state. Our experiments demonstrate that this improves the overall accuracy of the generated programs while using less training data.
We evaluate the CSGNet and CSGNetStack architectures on a number of shape parsing tasks. Both offer consistently better performance than a nearestneighbor baseline and are significantly more efficient than an optimization based approach. Reinforcement learning improves their performance when applying them to new domains without requiring groundtruth program annotations making the approach more practical (Table IV). We also investigate the role of size of training data and reward shaping on the performance of the parser. Finally, we evaluate the performance on the task of primitive detection and compare it with a Faster RCNN detector [NIPS20155638] trained on the same dataset. CSGNet offers higher Mean Average Precision (MAP) and is times faster compared to the Faster RCNN detector, suggesting that joint reasoning about the presence and ordering of objects leads to better performance for object detection (Table VI).
This paper extends our work that first appeared in [Sharma2018], adding analysis on reward shaping and the dependence on training set size, as well as the stackaugmented network architecture. Our PyTorch [pytorch] implementation is publicly available at: https://hippogriff.github.io/CSGNet/.
2 Related Work
CSG parsing has a long history and a number of approaches have been proposed in the literature over the past 20 years. Much of the earlier work can be categorized as “bottomup” and focuses on the problem of converting a boundary representation (bRep) of the shape to a CSG program. Our work is more related to program generation approaches using neural networks which have recently seen a revival in the context of natural language, graphics, and visual reasoning tasks. We briefly summarize prior work below.
2.1 Bottomup shape parsing
An early example of a grammarbased shape parsing approach is the “pictorial structure” model [fischler1973representation]. It uses a treestructured grammar to represent articulated objects and has been applied to parsing and detecting humans and other categories [felzenszwalb2005pictorial, yang2011articulated, bourdev2010detecting]. However, the parse trees are often shallow and these methods rely on accurate bottomup proposals to guide parsing (e.g., face and upperbody detection for humans). In contrast, primitive detection for CSG parsing is challenging as shapes change significantly when boolean operations are applied to them. Approaches, such as [Shapiro:1991:COC, Buchele:2003:THC, Shapiro:1993:SBC], assume an exact boundary representation of primitives which is challenging to estimate from noisy or lowresolution shapes. This combined with the fact that parse trees for CSG can be significantly deeper makes bottomup parsing error prone. Evolutionary approaches have also been investigated for optimizing CSG trees [Hamza, Weiss, Fayolle], however, they are computationally expensive.
Thus, recent work has focused on reducing the complexity of search. Tao et al. [Tao] directly operates on input meshes, and converts the mixed domain of CSG trees (discrete operations and continuous primitive locations) to a discrete domain that is suitable for boolean satisfiability (SAT) based program synthesizers. This is different from our approach which uses a neural network to generate programs without relying on an external optimizer.
2.2 Inverse procedural modeling
A popular approach to generate 3D shapes and scenes is to infer contextfree, often probabilistic “shape grammars” from a small set of exemplars, then sample grammar derivations to create new shapes[vanegas2012inverse, stava2014inverse, ritchie2015controlling, Talton:2012:LDP]. This approach called Inverse Procedural Modeling (IPM) has also been used in analysisbysynthesis image parsing frameworks [yuillek06, TeboulKSKP11, Martinovic:2013:BGL].
Recent approaches employ CNNs to infer parameters of objects [kulkarni2015deep] or whole scenes [Romaszko2017inverse] to aid procedural modeling. A similar trend is observed in graphics applications where CNNs are used to map input images or partial shapes to procedural model parameters [huang2017shape, RitchieTHG16, Nishida:2016:ISU]. Wu et al. [JiajunWu] detect objects in scenes by employing a network for producing object proposals and a network that predicts whether there is an object in a proposed segment, along with various object attributes. Eslami et al. [Eslami16] use a recurrent neural network to attend to one object at a time in a scene, and learn to use an appropriate number of inference steps to recover object counts, identities and poses.
Our goal is fundamentally different: given a generic grammar describing 2D or 3D modeling instructions and a target image or shape, our method infers a derivation, or more specifically a modeling program, that describes it. The underlying grammar for CSG is quite generic compared to specialized shape grammars. It can model shapes in several different classes and domains (e.g., furniture, logos, etc.).
2.3 Neural program induction
Our approach is inspired by recent work in using neural networks to infer programs expressed in some highlevel language, e.g., to answer question involving complex arithmetic, logical, or semantic parsing operations [Neelakantan, Reed2015, Denil, Balog, Joulin2015a, Zaremba2014, Zarembaa, Kaiser2015, Liang2016]. Approaches, such as [Johnson, hu2017learning], produce programs composed of functions that perform compositional reasoning on an image using an execution engine consisting of neural modules [Andreas2016NeuralMN]. Similarly, our method produces a program consisting of shape modeling instructions to match a target image by incorporating a shape renderer.
Other related work include the recent work by Tian et al. [tian2018learning], which proposes a program induction architecture for 3D shape modeling. Here programs contain a variety of primitives and symmetries are incorporated with loops. While this is effective for categories such as chairs, the lack of boolean operations is limiting. A more complex approach is that of Ellis et al. [Ellis], who synthesize handdrawn shapes by combining (lines, circles, rectangles) into Latex programs. Program synthesis is posed as a constraint satisfaction problem which is computationally expensive and can take hours to solve. In contrast, our feedforward model that takes a fraction of a second to generate a program.
2.4 Primitive fitting
Deep networks have recently been applied to a wide range of primitive fitting problems for 2D and 3D shapes. Tulsiani et al. [abstractionTulsiani17] proposed a volumetric CNN that predicts a fixed number of cuboidal primitives to describe an input 3D shape. Zou et al. [Zou] proposed an LSTMbased architecture to predict a variable number of boxes given input depth images. Li et al. [Lingxiao] introduced a point cloud based primitive fitting network where shapes are represented as an union of primitives. Paschalidou et al. [Paschalidou] uses superquadrics instead of traditional cuboids. Huang et al. [Huang2018] decompose an image by detecting primitives and arranging them into layers. Gao et al. [Gao2019] train deep network to produce control points for splines using input images and point cloud. The above approaches are trained to minimize reconstruction error like our method. On the other hand, they are limited to primitive fitting, while our method also learns modeling operating on them.
3 Designing a Neural Shape Parser
In this section, we first present a neural shape parser, called CSGNet, that induces
programs based on a CSG grammar given only 2D/3D shapes as input. We also present another shape parser variant, called CSGNetStack, which incorporates a stack as a form of explicit memory and results in improved accuracy and faster training. We show that both variants can be
trained to produce CSG programs in a supervised learning setting when groundtruth
programs are available. When these are not available, we show that reinforcement learning can be used based on policy gradient and reward shaping
techniques. Finally, we describe ways to improve the shape parsing at test time through a postprocessing stage.
CSGNet. The goal of a shape parser is to produce a
sequence of instructions given an input shape. The parser can be implemented as
an encoderdecoder using neural network modules as shown in
Figure 2. The encoder takes as input an image and
produces an encoding using a CNN. The decoder takes
as input and produces a probability distribution over programs
represented as a sequence of instructions. Decoders can be implemented using
Recurrent Neural Networks (RNNs). We employ Gated Recurrent Units
(GRUs) [chung2014empirical] that have been widely used for sequence
prediction tasks such as generating natural language and speech. The overall
network can be written as . We call this basic
architecture as CSGNet (see also Figure 3, left).
CSGNetStack. The above architecture can further be improved by incorporating feedback from the renderer back to the network. More specifically, the encoder can be augmented with an execution stack that stores the result of the renderer at every time step along with the input shape. This enables the network to adapt to the current rendered result. To accomplish this, our CSG rendering engine executes the program instructions produced by the decoder with the help of stack at each time step . The stack is updated after every instruction is executed and contains intermediate shapes produced by previous boolean operations or simply an initially drawn shape primitive. This stack of shapes is concatenated with the target shape, all stored as binary maps, along the channel dimension. The concatenated map is processed by the network at the next time step. Instead of taking all elements of the stack, which vary in number depending on the generated program, we only take the top maps of the stack. Empty entries in the stack are represented as allzero maps (see also Figure 3, right). At the first time step, the stack is empty, so all maps are zero.
In our implementation, the parser takes as input of size for networks and for networks, where is the input shape, is the execution stack of the renderer, and is the size of the stack. The number of channels is since the target shape, also represented as (or in 3D), is concatenated with the stack. Details of the architecture are described in Section 4. Similarly to the basic CSGNet architecture, the encoder takes as input and yields a fixed length encoding , which is passed as input to the decoder to produce a probability distribution over programs . The stackbased network can be written as . We call this stack based architecture CSGNetStack. The difference between the two architectures is illustrated in Figure 3.
Grammar. The space of programs can be efficiently described according to a contextfree grammar [hopcroft]. For example, in constructive solid geometry the instructions consist of drawing primitives (eg, spheres, cubes, cylinders, etc) and performing boolean operations described as a grammar with the following production rules:
Each rule indicates possible derivations of a nonterminal symbol separated by the symbol. Here is the start symbol, is chosen from a set of defined modeling operations and the is a primitive chosen from a set of basic shapes at different positions, scales, orientations, etc. Instructions can be written in a standard postfix notation, e.g., . Table 4 shows an example of a program predicted by the network and corresponding rendering process.
Instruction  Execution  Stack  

circle(32,32,28)  push circle(32,32,28)  [P1]  
square(32,40,24)  push square(32,40,24)  [P2 P1]  
circle(48,32,12)  push circle(48,32,12)  [P3 P2 P1]  
circle(24,32,16)  push circle(24,32,16)  [P4 P3 P2 P1]  
union  A=pop; B=pop; push(BA)  [E1 P2 P1] // E1=P3P4  
intersect  A=pop; B=pop; push(BA)  [E2 P1] // E2=P2E1  
subtract  A=pop; B=pop; push(BA)  [Out] // Out=P1E2 
3.1 Learning
Given the input shape and execution stack of the renderer, the parser network generates a program that minimizes a reconstruction error between the shape produced by executing the program and a target shape. Note that not all programs are valid hence the network must also learn to generate grammatical programs.
3.1.1 Supervised learning
When target programs are available both CSGNet and CSGNetStack variants can be trained with standard supervised learning techniques. Training data consists of shapes, corresponding programs, and also in the case of CSGNetStack stacks, program triplets , . The groundtruth program can be written as a sequence of instructions , .. , where is the length of the program . Similarly, in the case of CSGNetStack, the can be written as sequence of states of stack , .. used by the rendering engine while executing the instructions in program . Note that while training in supervised setting, the stack is generated by the renderer while executing ground truth instructions , but during inference time, the stack is generated by the renderer while executing the predicted instructions. For both network variants, the RNN produces a categorical distribution for both variants.
The parameters for either variant can be learned to maximize the loglikelihood of the ground truth instructions:
(1) 
3.1.2 Learning with policy gradients
Without target programs one can minimize a reconstruction error between the shape obtained by executing the program and the target. However, directly minimizing this error using gradientbased techniques is not possible since the output space is discrete and execution engines are typically not differentiable. Policy gradient techniques [Williams92simplestatistical] from the reinforcement learning (RL) literature can instead be used in this case.
Concretely, the parser , that represents a policy network, can be used to sample a program = (, .. ) conditioned on the input shape and in the case of CSGNetStack, also on the stack = (, .. ). Note that while training using policy gradient and during inference time, the stack is generated by the renderer while executing predicted instructions by the parser since groundtruth programs are unavailable. Then a reward can be estimated by measuring the similarity between the generated image obtained by executing the program and the target shape . With this setup, we want to learn the network parameters that maximize the expected rewards over programs sampled under the predicted distribution across images sampled from a distribution :
The outer expectation can be replaced by a sample estimate on the training data. The gradient of the inner expectation can be obtained by rearranging the equation as^{1}^{1}1conditioning on stack and input image is removed for the sake of brevity.:
It is often intractable to compute the expectation since the space of programs is very large. Hence the expectation must be approximated. The popular REINFORCE [Williams92simplestatistical] algorithm computes a MonteCarlo estimate as:
by sampling programs from the policy . Each program is obtained by sampling instructions from the distribution at every time step until the stop symbol (EOS) is sampled. The reward is calculated by executing the program . Samplingbased estimates typically have high variance that can be reduced by subtracting a baseline without changing the bias as:
(2) 
A good choice of the baseline is the expected value of returns starting from
[Sutton:1999:PGM:3009657.3009806, Williams92simplestatistical]. We compute the
baseline as the running average of past rewards.
Reward. The rewards should be primarily designed to encourage visual similarity of the generated program with the target. Visual similarity between two shapes is measured using the Chamfer distance (CD) between points on the edges of each shape. The CD is between two point sets, and , is defined as follows:
The points are scaled by the image diagonal, thus . The distance can be efficiently computed using distance transforms. In our implementation, we also set a maximum length for the induced programs to avoid having too long or redundant programs (e.g., repeating the same modeling instructions over and over again). We then define the reward as:
where is a reward shaping function and is the CSG rendering engine that renders the program into a binary image. Since invalid programs get zero reward, the maximum length constraint on the programs encourages the network to produce shorter programs with high rewards. We use maximum length in all of our RL experiments. The function shapes the CD as with an exponent . Higher values of makes the reward closer to zero, thereby making the network to produce programs with smaller CD. Table I (left) shows the dynamics of reward shaping function with different value and (right) shows that increasing values decreases the average CD calculated over the test set. We choose in our experiments.
3.2 Inference
Greedy decoding and beam search. Estimating the most likely
program given an input is intractable using RNNs. Instead one usually employs a
greedy decoder that picks the most likely instruction at each time step. An
alternate is to use a beam search procedure that maintains the kbest likely
sequences at each time step. In our experiments we report results with varying
beam sizes.
Visuallyguided refinement. Both parser variants produce a program with a discrete set of primitives. However, further refinement can be done by directly optimizing the position and size of the primitives to maximize the reward. The refinement step keeps the program structure of the program and primitive type fixed but uses a heuristic algorithm [Powell1964] to optimize the parameters using feedback from the rendering engine. In our experiments, we observed that the algorithm converges to a local minima in about iterations and consistently improves the results.
4 Experiments
We describe our experiments on different datasets exploring the generalization capabilities of our network variants (CSGNet and CSGNetStack). We first describe our datasets: (i) an automatically generated dataset of 2D and 3D shapes based on synthetic generation of CSG programs, (ii) 2D CAD shapes mined from the web where groundtruth programs are not available, and (iii) logo images mined also from the web where groundtruth programs are also not available. Below we discuss our qualitative and quantitative results on the above dataset.
4.1 Datasets
To train our network in the supervised learning setting, we automatically
created a large set of 2D and 3D CSGbased synthetic programs according to the
grammars described below.
Synthetic 2D shapes. We sampled derivations of the following CSG grammar to create our synthetic dataset in the 2D case:
Primitives are specified by their type: square, circle, or
triangle, locations and circumscribing circle of radius on a
canvas of size . There are three boolean operations: ,
, and . L is discretized to lie on a square grid with spacing
of units and R is discretized with spacing of units. The
triangles are assumed to be upright and equilateral. The synthetic
dataset is created by sampling random programs containing different number of
primitives from the above grammar, constraining the distribution of various
primitive types and operation types to be uniform. We also ensure that no
duplicate programs exist in our dataset. The primitives are rendered as binary
images and the programs are executed on a canvas of pixels. Samples from
our dataset are shown in Figure 5. Table II
provides details about the size and splits of our dataset.

2D  3D  

Train  Val  Test  Train  Val  Test  
3  25k  5k  5k  100k  10k  20k  
5  100k  10k  50k  200k  20k  40k  
7  150k  20k  50k  400k  40k  80k  
9  250k  20k  50k        
11  350k  20k  100k        
13  350k  20k  100k       
Synthetic 3D shapes. We sampled derivations of the following grammar in the case of 3D CSG:
The operations are same as in the 2D case.
Three basic solids are denoted by ‘’: Sphere, ‘’: Cube, ‘’: Cylinder. represents
the center of primitive in a 3D voxel grid.
specifies radius of sphere and cylinder, or the size of cube.
is the height of cylinder. The
primitives are rendered as voxels and the programs are executed on a 3D
volumetric grid of size . We used the same
random sampling method as used for the synthetic 2D dataset, resulting in
3D CSG programs. 3D shape samples are shown in Figure
5.
2D CAD shapes.
We collected CAD shapes from the Trimble 3DWarehouse
dataset [TrimbleWarehouse] in three categories: chair, desk and lamps. We
rendered the CAD shapes into binary masks from their front and
side views. In Section 4, we show that the rendered shapes can be
parsed effectively through our visual program induction method. We split this
dataset into shapes for training, validation and for testing.
Web logos. We mined binary logos from the web that can be modeled using the primitives in our output shapes. We test our approach on these logos without further training or finetuning our net on this data.
4.2 Implementation details
2D shape parsing.
Our encoder is based on an imagebased convnet in the case of 2D inputs. In the case of CSGNetStack, the input to the network is a fixed size
stack along with target image concatenated along the channel dimension, resulting in an
the input tensor of size , where is the number of used maps in the stack (stack size). In the architecture without stack (CSGNet), is simply set to 0.
The output of the encoder is passed as input to our GRUbased decoder
at every program step. The hidden state of our GRU units is passed through two
fullyconnected layers, which are then converted into a probability distribution
over program instructions through a classification layer. For the 2D CSG there
are unique instructions corresponding to different primitive types,
discrete locations and sizes, the boolean operations and the stop symbol.
3D shape parsing. In the case of 3D shapes, the encoder is based on an volumetric, voxelbased convnet. 3DCSGNetStack concatenates the stack with the target shape along the channel dimension, resulting in an input tensor of size , where is the number of used maps in the stack (stack size). In the architecture without stack (3DCSGNet), is simply set to 0. The encoder comprises of multiple layers of convolutions yielding a fixed size encoding vector. Similarly to the case, the GRUbased decoder takes the output of the encoder and sequentially produces the program instructions. In this case, there are unique instructions with different types of primitives with different sizes and locations, plus boolean modeling operations and a stop symbol.
During training, on synthetic dataset, we sample images/3D shapes rendered from programs of variable length (up to for 2D and up to for 3D dataset) from training dataset from Table II. More details about the architecture of our encoder and decoder (number and type of layers) are provided in the supplementary material.
For supervised learning, we use the Adam optimizer [KingmaB14] with learning rate and dropout of in nonrecurrent network connections. For reinforcement learning, we use stochastic gradient descent with momentum, learning rate, and with the same dropout as above.
4.3 Results
We evaluate our network variants in two different ways: (i) as models for inferring the entire program, and (ii) as models for inferring primitives, i.e., as object detectors.
4.3.1 Inferring programs
Method  IOU (k=1)  IOU (k=10)  CD (k=1)  CD (k=10) 

NN  73.9    1.93   
CSGNet  86.77  88.74  0.70  0.32 
CSGNetStack  91.33  93.45  0.60  0.12 

Evaluation on the synthetic 2D shapes. We perform supervised learning to train our stackbased network CSGNetStack and the nonstackbased network CSGNet on the training split of this synthetic dataset, and evaluate performance on its test split under different beam sizes. We compare with a baseline that retrieves a program in the training split using a Nearest Neighbor (NN) approach. In NN setting, the program for a test image is retrieved by taking the program of the train image that is most similar to the test image.
Table III compares CSGNetStack, CSGNet, and a NN baseline using the Chamfer distance between the test target and predicted shapes using the complete synthetic dataset. Our parser is able to outperform the NN method. One would expect that NN would perform well here because the size of the training set is large. However, our results indicate that our compositional parser is better at capturing shape variability, which is still significant in this dataset. Results are also shown with increasing beam sizes (k) during decoding, which consistently improves performance. Figure 7 also shows the programs retrieved through NN and our generated program for a number of characteristic examples in our test split of our synthetic dataset.
We also examine the learning capability of CSGNetStack with significantly less
synthetic training dataset in comparison to CSGNet in the Figure
6. With just of the total dataset, CSGNetStack performs IOU ( CD) in comparison to IOU ( CD) using
CSGNet. The CSGNetStack continues to perform better compared to CSGNet in
the case of more training data. This shows that incorporating the extra
knowledge in the form of an execution stack based on the proposed architecture
makes it easier to learn to parse shapes.
Evaluation on 2D CAD shapes. For this dataset, we report results on its test split under two conditions: (i) when training our network only on synthetic data, and (ii) when training our network on synthetic data and also finetuning it on the training split of rendered CAD dataset using policy gradients.
Table IV shows quantitative results on this dataset. We first compare with the NN baseline. For any shape in this dataset, where ground truth program is not available, NN retrieves a shape from synthetic dataset and we use the ground truth program of the retrieved synthetic shape for comparison.
We then list the performance of CSGNetStack and CSGNet trained in a supervised manner only on our synthetic dataset. Further training with Reinforcement Learning (RL) on the training split of the 2D CAD dataset improves the results significantly and outperforms the NN approach by a considerable margin. This also shows the advantage of using RL, which trains the shape parser without groundtruth programs. The stack based network CSGNetStack performs better than CSGNet showing better generalization on the new dataset. We note that directly training the network using RL alone does not yield good results which suggests that the twostage learning (supervised learning and RL) is important. Finally, optimizing the best beam search program with visually guided refinement yielded results with the smallest Chamfer Distance. Figure 8 shows a comparison of the rendered programs for various examples in the test split of the 2D CAD dataset for variants of our network. Visually guided refinement on top of beam search of our two stagelearned network qualitatively produces results that best match the input image.
We also show an ablation study indicating how much pretraining on the synthetic
dataset is required to perform well on the CAD dataset in Figure
9. With just of the synthetic dataset based pretraining,
CSGNetStack gives IOU (and CD) in comparison to IOU (and 1.9
CD), which shows the faster learning capability of our stack based
architecture. Increasing the synthetic training size used in pretraining shows
slight decrease in performance for the CSGNetStack network after , which
hints at the overfitting of the network on the synthetic dataset domain.
Method  NN  3DCSGNet  3DCSGNetStack  
k=  k=  k=  k=1  k=5  k=10  
IOU (%)  90.5 
Evaluation on Logos.
We experiment with the logo dataset described in Section
4.1 (none of these logos participate in training).
Outputs of the induced programs parsing the input logos are shown in Figure
10. In general, our method is able to parse logos into primitives
well, yet performance can degrade when long programs are required to generate
them, or when they contain shapes that are very different from our used
primitives.
Evaluation on Synthetic 3D CSG. Finally, we show that our approach can be extended to 3D shapes. In the 3D CSG setting we use 3DCSG dataset as described in the Section 4.1. We train a stack based 3DCSGNetStack network that takes voxel representation of input shape concatenated with voxel representation of stack. The input to our 3DCSGNet are voxelized shapes in a grid. Our output is a 3D CSG program, which can be rendered as a highresolution polygon mesh (we emphasize that our output is not voxels, but CSG primitives and operations that can be computed and rendered accurately). Figure 11 show pairs of input voxel grids and our output shapes from the test split of the 3D dataset. The quantitative results are shown in the Table V, where we compare our 3DCSGNetStack and 3DCSGNet networks at different beam search decodings with NN method. The stackbased network also improves the performance over the nonstack variant. The results indicate that our method is promising in inducing correct programs for 3D shapes, which also has the advantage of accurately reconstructing the voxelized surfaces into highresolution surfaces.
4.3.2 Primitive detection
Successful program induction for a shape requires not only predicting correct primitives but also correct sequences of operations to combine these primitives. Here we evaluate the shape parser as a primitive detector (i.e., we evaluate the output primitives of our program, not the operations themselves). This allows us to directly compare our approach with bottomup object detection techniques.
In particular we compare against Faster RCNNs [NIPS20155638], a stateoftheart object detector. The Faster RCNN is based on the VGGM network [Chatfield14] and is trained using boundingbox and primitive annotations based on our 2D synthetic training dataset. At test time the detector produces a set of bounding boxes with associated class scores. The models are trained and evaluated on 640640 pixel images. We also experimented with bottomup approaches for primitive detection based on Hough transform [Duda] and other rulebased approaches. However, our experiments indicated that the Faster RCNN was considerably better.
For a fair comparison, we obtain primitive detections from CSGNet trained on the 2D synthetic dataset only (same as the Faster RCNN). To obtain detection scores, we sample programs with beamsearch decoding. The primitive score is the fraction of times it appears across all beam programs. This is a Monte Carlo estimate of our detection score. The accuracy can be measured through standard evaluation protocols for object detection (similar to those in the PASCAL VOC benchmark). We report the Mean Average Precision (MAP) for each primitive type using an overlap threshold between the predicted and the true bounding box of intersectionoverunion. Table VI compares the parser network to the Faster RCNN approach.
Our parser clearly outperforms the Faster RCNN detector on the squares and triangles category. With larger beam search, we also produce slighly better results for circle detection. Interestingly, our parser is considerably faster than Faster RCNN tested on the same GPU.
Method  Circle  Square  Triangle  Mean  Speed (im/s) 

Faster RCNN  87.4  71.0  81.8  80.1  5 
CSGNet,  86.7  79.3  83.1  83.0  80 
CSGNet,  88.1  80.7  84.1  84.3  20 
proposalcomparison
5 Conclusion
We believe that our work represents a step towards neural generation of modeling programs given target visual content, which we believe is quite ambitious and hard problem. We demonstrated that the model generalizes across domains, including logos, 2D silhouettes, and 3D CAD shapes. It also is an effective primitive detector in the context of 2D shape primitive detection.
One might argue that the 2D images and 3D shapes considered in this work are relatively simple in structure or geometry. However, we would like to point out that even in this ostensibly simple application scenario (i) our method demonstrates competitive or even better results than stateoftheart object detectors, and most importantly (ii) the problem of generating programs using neural networks was far from trivial to solve: based on our experiments, a combination of memoryenabled networks, supervised and RL strategies, along with beam and local exploration of the state space all seemed necessary to produce good results.
As future work, we would like to generalize our approach to longer programs with much larger spaces of parameters in the modeling operations and more sophisticated reward functions balancing perceptual similarity to the input image and program length. Other promising direction is alternate strategies for combining bottomup proposals and topdown approaches for parsing shapes, in particular, approaches based on constraint satisfaction and generic optimization.
Acknowledgments. The project is supported in part by grants from the National Science Foundation (NSF) CHS1422441, CHS1617333, IIS1617917. We also acknowledge the MassTech collaborative grant for funding the UMass GPU cluster.