Brain tumors have existed for a long time, but their rate is increasing among all age groups. Early detection is essential for timely prognosis as it is a dangerous and life-threatening disease. Since the advent of artificial intelligence and machine learning, different algorithms have been proposed to classify and segment brain tumors. However, all have their limitations when it comes to local deployment. The main drawback is designing multiple architectures for multiple tasks, which increases the time and computational complexity and affects performance. This paper addresses this drawback by proposing a multi-task learning (MTL) model that can take 2D-magnetic resonance imaging (MRI) images as input and gives predictions for multiple outputs such as detection and segmentation. It gives two outputs from one architecture: the system’s efficiency. Brain Tumor Segmentation (BRaTs) 2019 and BRaTs 2020 dataset has been used to evaluate the proposed architecture. Experimental results show that the best model shows 98% accuracy for detecting MRI images as either normal/abnormal, whereas an overall Dice score of 92% for multi-class segmentation of high-grade glioma gliomas into the whole tumor, enhancing tumor and core tumor. The overall performance of the proposed architecture proves that it can be the best-suited framework for clinical setup.