Abstract: In today’s digital age, law enforcement officials and even employers may find it easier than ever to take advantage of camera and wiretap surveillance. Surveillance cameras now line many public streets and workplace locations in an attempt to monitor activity and law enforcement agencies continue to use wiretapping to aid in investigations. Even with the advancement of technology, we have always resorted to manned surveillance techniques which will require impeccable human attention to the video feed received from the surveillance cameras. The orthodoxical way of surveillance system can be automated by spatially identifying the human body and the vital body parts namely head, torso, etc. from live video frames such as CCTV camera footage. With this spatial information from the video frames, we estimate the poses held with the temporal association between the successive and the previous video frames. The system works well with unconstrained backgrounds and without any premeditation of the clothing, brightness of the video frame. We also do not impose any constraints on the position of a person in the video frame. The only constraint that is imposed by our system is that people should be in a head-over-torso position with either near-frontal or near-rear viewpoint for greater accuracy of the estimation However, the system responded with a considerable accuracy for side poses as well, during testing. The poses gleaned are used for detecting punching activity.