No, self-retaliation is not legal. I am not a lawyer or entitled to give law-related advice in any jurisdiction whatsoever; but I bet such punitive action is explicitly forbidden by the laws of the country you live in (at least). Generically, if you refer to the usual typology of societies as defined by Elman Service, in societies which have reach the “chiefdom” or “state” levels, violence (and thus its corollary, justice) becomes a monopoly of the society’s leader and his delegates.
One way to deter people from putting up bought PDF on free download platforms is to tag the PDF with the user name. That’s what ANSI does (I have bought a copy of the X9.62 standard, and the PDF I got bears my name in the margin of each page). A motivated attacker may edit the PDF files to remove this kind of tag, but that would suspiciously look like work, and attackers, being also humans, are inherently lazy. This ought to make your problem less frequent.
Acting on one illegal act with another illegal act does not make any of the two legal. In fact, you might effectively prevent any legal action against the perpetrator of the former, if your latter illegal act in vengeance (whatever it might be) rendered any evidence gathered against the former useless, or even illegal, depending on how it was obtained and if it’s possible for the perpetrator of the former to dispute their legality. His act would still not be legal, but it might be impossible to prosecute. I’d be extremely wary of how you thread, and what action you deem ethical in the legal corridors.
Disclaimer: I am not a lawyer
No. Just like it’s illegal to hunt down and kill a thief, it’s illegal to retaliate in most countries, and you would be in just as much trouble as in the case where you hack some random website that you have no grudge against.
Instead, in the case of the pdfs being posted on a file hosting service (which are usually DMCA safe harbors), file a DMCA takedown request for that file. If the host isn’t classified as a safe harbor, you can sue.
In the future, try using DRM.