With the development of aeronautics and astronautics, the spatial resolution of remote sensing images is increasing, and the difficulty in obtaining data is reduced. Today, with the growing popularity of remote sensing techniques, the automatic information extraction from high-resolution remote sensing images has become research focus. Buildings, which play an important role in human daily life, mark the urban development. Currently, the automatic building extraction from high-resolution remote sensing images has become an important means in urban studies, such as urban sprawl, urban planning, urban heat island effect, population estimation and damage evaluation. With the improvement of spatial resolution, there are more details of ground objects in images and the scenes become more complex, and it’s more difficult for building extraction by traditional methods. How to extract buildings automatically, accurately and efficiently is one of the difficulties in remote sensing image processing field. For the complex scenes in high-resolution remote sensing images, we will introduce the visual attention mechanism to building extraction. Based on visual attention mechanism and object-oriented image analysis method, combining the bottom-up scenario-driven primary feature extraction with top-down task-driven empirical knowledge guide, through studying the visual salient features of buildings, we will propose a multi-scale automatic extraction method from built-up areas to buildings.