Artificial Intelligence (AI) is reshaping legal practice, from research to drafting, but its growing use in courts raises concerns about accuracy, ethics, and accountability. Courts in India now face a key question: can AI be trusted in legal proceedings? AI tools like ChatGPT, Manupatra, and SCC Online have evolved from research assistants to tools capable of generating arguments and citations. However, “AI hallucination” where false cases or reasoning are produced poses serious risks. Instances of fake citations, flagged by judges like Justice B.V. Nagarathna, highlight misuse and professional negligence. Globally, cases like Mata v. Avianca, Inc. show courts penalising lawyers for relying on unverified AI outputs. The core principle remains: lawyers bear absolute responsibility for accuracy and cannot shift blame to AI. While AI offers benefits like faster research and improved access to justice, it also raises issues of bias, confidentiality, and misinterpretation of complex judgments. The Supreme Court of India has termed such risks an “institutional concern.” Ultimately, AI is a powerful assistant, not a substitute for legal reasoning courts will judge the lawyer, not the machine.