I see a lot of hype regarding cyber/network security. And also find an unending list of books and courses for the same.

As I have been working on system software projects for the last couple of months, such as OSes and compilers, I want to delve deep into system security.

I can find some material online like malware analysis and cryptanalysis etc. But where do I start if I want to go deeper into System Security?

P.S. I am a CS Undergrad.

  • 1
    $\begingroup$ I just wanted to comment that as far as I know, cryptanalysis (which you mentioned above) has pretty much nothing to do with system security, other than that it's sometimes possible to exploit the use of very weak ciphers (without usually needing to know much about them). So, generally speaking, I would exclude cryptanalysis from your list of topics. $\endgroup$
    – yoniLavi
    Feb 22, 2023 at 18:22

1 Answer 1


It's a great field to explore!

  • Operating Systems, Three Easy Pieces is a way to continue your exploration of OS, and is free online.

  • The Shellcoder's Handbook has been recommended to me before, though it is not free.

  • Hacking: The Art of Exploitation is great, and it goes into fuzzing and manual attacks nicely. I use this textbook with my students. (Note that the contents are rather out of date, but also get at difficult ideas that underlie more modern attacks)

  • Reverse Engineering for Beginners has also been recommended to me, and is free. I have not read it myself.

And, of course, you should join CTFs. There are simply hundreds of them, and there are contests you can join all year long. PicoCTF is nominally aimed at high school, but it is a particularly good one to get started with because it starts the level quite low, and the challenges get progressively harder until they are quite advanced, even for undergrads. You would have to identify yourself as an adult, which would make you ineligible for prizes, but the contest is otherwise free, and unlike many CTFs, it is explicitly designed to teach.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.