A complete semantics-based behavior recognition approach that depends on object tracking has been introduced and extensively investigated. The proposed framework obtains detecting and tracking anaomoly people and in the scene using a Dynamic Patch Grouping (DGP) and KNN classifier. Based on the temporal properties of these behaviors and events are semantically recognized by employing object and inter object motion features. A number of types of behavior that are relevant to security in public transport areas have been selected to demonstrate the capabilities of this approach. Examples of these are monitoring the survaylines to identify and track the unwanted if all peoples are waliking and the car is cross the road. It is the abnormal of that scenario that is tracking accuractly in our process.