At the temple, the prime minister offered prayers and performed rituals amid Vedic chants