Roofing material classification is critical for urban sustainability, energy efficiency, public health and environmental protection, and regulatory compliance. Despite the need for scalable solutions, existing approaches are hindered by reliance on costly, specialised, multispectral/hyperspectral imagery, and spatial biases, and they overlook the potential of deep learning and multimodal data fusion. This paper addresses these research gaps by introducing RoofSense, a multimodal semantic segmentation dataset for roofing material classification in diverse urban contexts, leveraging~\qty{8}{\cm} aerial~\acs{rgb} imagery and airborne lidar data. Representing eight diverse classes and spanning more than~\qty{138}{ha} and 480 buildings across five Dutch cities, RoofSense is the largest publicly available dataset of its kind. By fusing spectral and geometric information at the pixel level and employing a novel weighting scheme to address class imbalance, RoofSense was used to achieve competitive classification and segmentation performance with a tuned, off-the-shelf model based on ResNet-18-D and DeepLabv3+. Lidar-derived features were added to improve performance in difficult classes and materials commonly used in pitched roofs, but results were ultimately sensitive to material and building context, clutter, and modality alignment. The implementation is publicly accessible at \url{https://github.com/ANONYMOUS}.