What is meant by transgression in supplication? Is it permissible to say: O Allah! Destroy all the Jews and Christians without any exception? ما هو الاعتداء في الدعاء

Praise be to Allah, and may Allah’s peace and blessings be upon the Messenger of Allah, his family and his companions.

To proceed:

Transgression is  in supplication the supplication itself or its characteristic like saying, : O Allah! Forgive me if You wish, or making an assonant supplication or supplicating in a slow manner usually prolonging vowels. Violation can also be found in asking what is not permissible to ask like asking  to be granted the status of the messengers, requesting Allah to grant you knowledge or power of everything, asking Allah to make you ever living or asking for something against a Muslim unjustly. Part of violation in supplication is asking for something which is prohibited by textual evidence as well as supplicating against the Jews and Christians without specifying their descriptions such as the aggressors among them, like what is done by other people who when supplicating say: "O Allah! Destroy them by burning all of them in their numbers and do not leave anyone amongst them." However, a general supplication which does not contradict religious text like cursing them or asking victory or political dominance over them was established by the Quran and the Prophetic Sunnah. Some of the verses supporting this are: “The Curse of Allah is on the Zalimun (polytheists, wrong-doers, oppressors, etc.).” [Hud: 18].
“Those among the Children of Israel who disbelieved were cursed by the tongue of Dawûd (David) and 'Iesa (Jesus), son of Maryam (Mary).” [al-Maaidah: 78].
Another evidence to this is a hadith transmitted by Bukhari and Muslim on the authority of Aishah and Ibn Abbas (may Allah be pleased with them) who said, “When Allah's Messenger (peace be on him) became ill seriously, he started covering his face with his woollen sheet, and when he felt short of breath, he removed it from his face and said, 'That is so! Allah's curse be on the Jews and the Christians, as they took the graves of their prophets as (places of worship),' intending to warn (the Muslims) of what they had done.” Another proof is that the companions used to make similar supplications in qunut. In al-Mufhim, Qurtubi says: “There is no scholarly difference on the permissibility of cursing the disbelievers and supplicating against them”.
And Allah knows best.