The Charpy impact test technique assesses the toughness of an engineering material. The test measures the amount of energy a specimen can resist before it is broken by the impact of a heavy pendulum. Estimation of toughness is carried out manually by a skilled operator; they assess the percentage of light-reflective brittle regions on the fracture area. Because this assessment is performed manually, there is some subjectivity in the results. This study proposes a machine-based-learning algorithm to estimate this measure automatically. The method consists of capturing a digital image of the fracture surface after impact, preprocessing it, dividing it up into segments, and extracting from each segment features associated with its texture. Feature vectors feed a classifier whose purpose is to distinguish between brittle and ductile images (binary output). To estimate toughness, the classifier’s outputs are used to construct a binary image, which is postprocessed to determine the percentage of the brittle region. To assess the accuracy of the algorithm, automatically and manually classified images are compared. Results show that the algorithm proposed was able to distinguish between brittle and ductile regions successfully and could be used instead of the manually performed technique.