Abstract: COVID-19 pandemic has been ravaging the world we know since it's insurgence. Computer-Aided Diagnosis (CAD) systems with high precision and reliability can play a vital role in the battle against COVID-19. Most of the existing works in the literature focus on developing sophisticated methods yielding high detection performance yet not addressing the issue of predictive uncertainty. Uncertainty estimation has been explored heavily in the literature for Deep Neural Networks; however, not much work focused on this issue on COVID-19 detection. In this work, we explore the efficacy of state-of-the-art (SOTA) uncertainty estimation methods on COVID-19 detection. We propose to augment the best performing method by using feature denoising algorithm to gain higher Positive Predictive Value (PPV) on COVID positive cases. Through extensive experimentation, we identify the most lightweight and easy-to-deploy uncertainty estimation framework that can effectively identify the confusing COVID-19 cases for expert analysis while performing comparatively with the existing resource heavy uncertainty estimation methods. In collaboration with medical professionals, we further validate the results to ensure the viability of the framework in clinical practice.