A vehicle detection algorithm based on the multilevel knowledge base is proposed to overcome the problem of poor robustness as well as the difficulty of identifying weak vehicle targets. The multilevel task-driven method is adopted in the algorithm by building three different classes of knowledge bases to achieve the accuracy recognition of vehicle targets. First, a simple knowledge base is constructed via choosing Haar-like features to detect the vehicle region of interest in the traffic scene image; second, the optimal structure symmetry decision function is obtained by establishing the structure characteristics knowledge base, which is used to determine the region of the potential vehicle; finally, the property feature knowledge base is built to precisely identify vehicle targets via calculating maximum similarity. Then the relevant knowledge base will be continuously updated to achieve the method adaptive adjustment when satisfying the criterion. Experimental results illustrate that the recognition rate is more than 95% in different traffic scenarios, while the recognition rate for weak contrast vehicle targets is in excess of 71% and the false-alarm rate is simultaneously under 5%.