Here we identify the background object such as tree, sky, road based on background recognition and perceptual organization. The main contribution of this paper is a developed perceptual organization model (POM) for boundary detection. The POM quantitatively incorporates a list of Gestalt laws and therefore is able to capture the nonaccidental structural relationships among the constituent parts of a structured object. Bottom up segmentation method also tries to capture nonlocal image characteristics. Background classifiers are used to identify the background patches. Compared to the large number of structured object classes, there are only a few common background objects in outdoor scenes. These background objects have low visual variety and hence can be reliably recognized. After background objects are identified, we roughly know where the structured objects are and delimit perceptual organization in certain areas of an image. For many objects with polygonal shapes, such as the major object classes appearing in the streets (e.g., buildings, vehicles, signs, people, etc.) and many other objects, POM is used to group the remaining patches (parts) to larger regions that correspond to structured objects or semantically meaningful parts of structured objects. Our experimental results show that our proposed method outperformed two state-of-the-art image segmentation approaches on outdoor image.