I want to close the back doors into my system.
I believe that is your answer. You need to perform some incident response and find out what the cause of your breach was and close the holes that you have found.
Security is a mulch-layered approach, you will want some software like firewalls, antivirus, intrusion detection, etc. at the system level and likely at the network level. You may want to proxy connections to your server as well at both the network and the application layer. For example, on a web server you may want to include a web application firewall.
One of the most likely causes if you are using off the shelf software is that your software is out of date and needs to be patched, software should be updated regularly. In addition, you also need to ensure you configure all optional settings in the appropriate and most secure manner, which will vary by application. You can google for guides, or you may consider asking a more specific question to the community about how to best secure a specific application.
If you are writing your own software, you need to ensure that you follow secure coding practices. You must ensure there is input validation, proper application of protocols/algorithms, and that you are using updated/secured versions of third party libraries. If you are doing web apps, check out OWASP.
In terms of the incident response, you need to look for evidence of how they got on to your system and what they did. It’s possible they were only able to get to certain files or compromise certain applications, however, from your original questions its not quite sure what happened or what they actually did.
Evidence can be found by comparing files with known values (file integrity monitoring) and by reviewing access logs (assuming they were no also compromised). You should search through some existing questions here on this site on incident response that may help you on the right path.
Restoring from a backup will get you back online, but it will not solve your problem, the backup likely holds the same vulnerabilities.