This paper proposes a matrix-based feature selection and classification method that takes the advantage of L2,1-norm regularization. Current studies show that feature extraction and selection have been important steps in classification. However, the existing methods consider feature extraction and selection to be separated phases, which generates suboptimal features for the recognition task. Aiming at making up for this deficiency, we designed a novel classification framework that performs Unsupervised Optimal Feature Selection (UOFS) to simultaneously integrate dimensionality reduction, sparse representation, jointly sparse feature extraction and feature selection as well as classification into a unified optimization objective. Specifically, an L2,1-norm-based sparse representation model is constructed as an initial prototype of the proposed method. Then a projection matrix with L2,1-norm regularization is introduced into the model for subspace learning and jointly sparse feature extraction and selection. Finally, we impose a scatter matrix-like constraint on the proposed model in pursuit of the features with less redundancy for recognition. We also provide an alternative iteration optimization with convergence analysis for solving UOFS. Experiments on public gesture and human action datasets validate the superiority of UOFS over other state-of-the-art methods.