Flower Image Classification is a Fine-Grained Classification problem.The main difficulty of Fine-Grained Classification is the large inter-class similarity and the inner-class difference.In this paper,we propose a new algorithm based on Saliency Map and PCANet to overcome the difficulty.This algorithm mainly consists of two parts:flower region selection,flower feature learning.In first part,we combine saliency map with gray-scale map to select flower region.In second part,we use the flower region as input to train the PCANet which is a simple deep learning network for learning flower feature automatically,then a 102-way softmax layer that follow the PCANet achieve classification.Our approach achieves 84.12%accuracy on Oxford 17 Flowers dataset.The results show that a combination of Saliency Map and simple deep learning network PCANet can applies to flower image classification problem.