Memristors are promising components for applications in nonvolatile memory, logic circuits, and neuromorphic computing. In this paper, a novel circuit for memristor-based multilayer neural networks is presented, which can use a single memristor array to realize both the plus and minus weight of the neural synapses. In addition, memristor-based switches are utilized during the learning process to update the weight of the memristor-based synapses. Moreover, an adaptive back propagation algorithm suitable for the proposed memristor-based multilayer neural network is applied to train the neural networks and perform the XOR function and character recognition. Another highlight of this paper is that the robustness of the proposed memristor-based multilayer neural network exhibits higher recognition rates and fewer cycles as compared with other multilayer neural networks.