Abstract: An efficient scheme for object detection and tracking under a fixed camera environment with SURF (Speeded Up Robust Feature) algorithm and the difference image method is proposed in this study. Since, SURF provides local feature descriptors robust on scale, rotation and illumination changes, SURF algorithm is adopted for its advantages over SIFT (Scale Invariant Feature Transform) algorithm. In order to improve the speed and accuracy in detecting and tracking objects, the proposed algorithm combines SURF algorithm and the difference image method. Experiments on two video data sets imply that the proposed method can be an alternative to conventional methods because of the improvements in terms of the speed of operation and the tracking accuracy over conventional methods.