Current standard methods for voxel-level dosimetry in radionuclide therapy suffers from a tradeoff between accuracy and computational efficiency. Monte Carlo (MC) radiation transport algorithms are considered as the gold standard, but are associated with long computation time, while fast voxel dose kernel (VDK) based methods can be inaccurate in the presence of tissue density heterogeneities. This paper investigates a deep residual Convolutional Neural Networks (CNN) approach that learns the difference between the MC and the VDK dose-rate maps to address the speed-accuracy trade-off issue. As with MC and VDK-based dosimetry, the input to the CNN was the patient's SPECT activity map and CT-based density map. MC dosimetry was used only during the training process to generate ground truth training labels. Furthermore, to potentially account for the degradation of dose-rate maps due to poor SPECT spatial resolution, we trained the CNN using dose-rate maps directly corresponding to phantom activity/density maps that were generated from patient's PET scans. The test data consisted of phantom simulations and one patient who underwent 177Lu DOTATATE therapy for neuroendocrine tumors. In phantom cases, the lesion/organ mean dose-rates from ground truth (GT) agreed better with the CNN dose-rates compared to VDK with density scaling, with an average of 60% improvement for lesions and 55%, 63% improvement for left/right kidney, respectively. For all regions, the normalized root mean square error (NRMSE) relative to GT was substantially lower with CNN than with VDK and MC, i.e., an average of 23%, 22% improvement for lesion, respectively. Using a GPU, the CNN took only about 2.0 seconds to generate a patient's 512×512×130 absorbed dose-rate map while the same calculation took about 40 minutes using our fast in-house Dose Planning Method (DPM) MC algorithm that runs on a CPU. In conclusion, the proposed CNN approach demonstrated consistently higher accuracy than VDK-density scaling and comparable accuracy versus MC and is fast enough to be used clinically.