Livable cities enhance urban economic development, improve physical and mental health, foster well-being, and foster urban sustainability. Evaluating urban livability is therefore important for policymakers to develop urban planning and development strategies aimed at improving livability. Mainstream methods of evaluating urban livability assign different weights to diverse indicators extracted from survey data, statistical data, and geospatial data. To relieve such time-consuming and labor- intensive data collection, this study proposes a transformer-based multi-task multimodal regression (TMTMR) model for the simultaneous evaluation of urban livability focusing on five domain-specific scores. Pretrained state-of-the-art computer vision and natural language processing models serve as backbones to extract features from high spatial resolution remote sensing (RS) images, digital surface models (DSM), night light remote sensing (NLRS) images and point of interest (POI) data. An attention mechanism helps the TMTMR model to assign varying significance levels to features from different modalities, thus capturing both intrinsic information and interrelationships among modalities for livability evaluation. Experiments show the proposed TMTMR model is capable of effectively evaluating urban livability directly from multimodal geospatial data.