A framework for crowd modeling using a combination of multiple kernel learning (MKL)-based fast head detection and shape-aware matching is proposed. First, the MKL technique is used to train a classifier for head detection using a combination of the histogram of oriented gradient and local binary patterns feature sets. Further, the head detection process is accelerated by implementing the classification procedure only at those spatial locations in the image where the gradient points overlap with moving objects. Such moving objects are determined using an adaptive background subtraction technique. Finally, the crowd is modeled as a deformable shape through connected boundary points (head detection) and matched with the subsequent detection from the next frame in a shape-aware manner. Experimental results obtained from crowded videos show that the proposed framework, while being characterized by a low computation load, performs better than other state-of-art techniques and results in reliable crowd modeling.