Abstract: We present a novel and compact architecture for deep Convolutional Neural Networks (CNNs) in this paper, termed 3D-FilterMap Convolutional Neural Networks (3D-FM-CNNs). The convolution layer of 3D-FM-CNN learns a compact representation of the filters, named 3D-FilterMap, instead of a set of independent filters in the conventional convolution layer. The filters are extracted from the 3DFilterMap as overlapping 3D submatrics with weight sharing among nearby filters, and these filters are convolved with the input to generate the output of the convolution layer for 3D-FM-CNN. Due to the weight sharing scheme, the parameter size of the 3D-FilterMap is much smaller than that of the filters to be learned in . the conventional convolution layer when 3D-FilterMap generates the same number of filters. Our work is fundamentally different from the network compression literature that reduces the size of a learned large network in the sense that a small network is directly learned from scratch. Experimental results demonstrate that 3D-FM-CNN enjoys a small parameter space by learning compact 3D-FilterMaps, while achieving performance compared to that of the baseline CNNs which learn the same number of filters as that generated by the corresponding 3D-FilterMap.