# How to figure out what I need to search for in order to create a program?

I'm a student in computer science and got my bachelor's degree 2 weeks ago. I had the opportunity to practice a lot of C/C++ (more C than C++) during the past 2 years.

I frequently don't know what I should know in order to realize what I want to do, or in order to fix a bug, because in order to do what I want to do, I need to know what the steps are to reach my goal, or at least in which direction I should look in order to figure out the intermediate step.

For example, let's say I ask you to create a shell from scratch. A friend told me that:

• There is a function which allows you to execute some builtins command (execv() function in C)
• There is also a function which allows you to execute a function without leaving the current process (fork() function in C)

How are you supposed to know that you need a function which allows you to execute a function without leaving the current process? How are you supposed to came to the conclusion that you need a function to execute builtins, but inside a child process in order to keep the program running normally?

How can I figure out what steps I need to take to realize what I want to do, or how to figure out what I need to search for in order to create a program?

I ask the question because I may have a bachelor in computer science but I'm not capable of keeping a job as a developer.

• Why don't you join me in the Guidance office (chatroom)? – ItamarG3 Aug 6 '17 at 18:53
• "I'm a student in computer science and got my bachelor's degree 2 weeks ago."- Are you still in school, e.g. grad school? – Nat Aug 6 '17 at 22:17
• Apprenticeship. No one should just be cast adrift on a sea of problems, alone, with only their "square girlfriend" as company. But, if you are, take advantage, you are unsupervisable, and not easy to replace. You are being paid to learn whatever you need to, or want to. – user737 Aug 8 '17 at 1:10
• Great question - both in terms of asking about an early career point, and identifying something which maybe isn't taught all that well by goal based education. – Sean Houlihane Aug 8 '17 at 7:29
• Note you were told wrong about fork and exec: fork create a new process, exec replaces program with external program (not builtin). – ctrl-alt-delor Aug 8 '17 at 10:43

To a certain degree, it comes down to experience. But if you don't have experience, I feel you can get a lot of mileage of just trying to do the simplest thing possible to see what'll happen and googling your questions as you go.

At some point, you'll run into an obstacle you didn't anticipate and realize you need to pivot and adjust or even entirely re-write your code to accommodate. (Having to rewrite code can seem painful at first, but it's basically something everybody needs to get used to. It's helpful to have a sort of zen mindset, where you let go of your attachment to your code).

Once you do this enough times, you start to develop a sort of gut sense for what sorts of obstacles you might end up.

For example, take your shell example. If I had absolutely no idea what I was doing but just moved forward instead, I would probably...

1. Start by writing the code to do the user input/output stuff so I can get commands from the user. This code would likely be extremely minimal/something you'd write in a CS1 class.
2. Next implement some basic shell navigation functionality like cd and ls directly within the shell. Now, my shell is a little more complicated: I need to remember things like where I'm located in the file system; I'd probably have to do some research on how you'd implement things like ls within C...
3. Next, try I'd try and implement the ability to have my shell call other programs. Since I'm a beginner, I have no idea how to do this, so I'd pretty much immediately just google "C run another program". This will immediately bring up results that mention the "fork" and "exec" keywords.

I might not know what those mean, but it gives me just enough context that I can form more specific google queries. ("C exec example", "C exec vs excv", etc...)

Now, depending on how I previously wrote my shell, I might immediately be able to adapt these examples to my own code, or I might need to rewrite and rearrange things significantly. This is mildly painful, but such is the life of a beginner -- no biggie, I do what I need to.

Then you rinse and repeat...

Another strategy you can try is to try planning ahead: specifically, figure out what precisely you want to do, then decompose that plan into smaller and smaller subproblems. The nice thing about this approach is that it forces you to think about potential obstacles you might run into and can help make it easier for you to form precise google queries.

For example, we want to write a shell. Well, what exactly does this shell need to be able to do? Let's say as a minimum prototype, I want to be able to:

1. Get user input
2. Run basic navigational commands like cd and ls
3. Define basic environment variables
4. Be able to run other programs

Ok, how do we do all three of these things? Well, if we drill down, we might end up coming up with something like this:

1. Get user input
1. Recognize and expand environment variables first?
2. Run basic navigational commands like cd and ls
1. Make some kind of system or library call to get the data I need? (Note to self: google this)
3. Define basic environment variables
1. Process the user input to get the variable name and the value somehow?
2. Store the variables in a map of some sort?
4. Be able to run other programs
1. ??? (Note to self: google this)

Notice that if we had kept drilling down, we'd end up learning about the fork and exec thing here as well.

• The bullet point 2 is exactly what the OP is about. Why ls could be an external program, but cd can not? To answer this question one need to understand an entire Unix process model, and I am afraid it takes a bit more than a Google query. BTW, in other models, like MS-DOS, cd could happily be an external utility. – user58697 Aug 7 '17 at 3:33
• @user58697 -- I'm not sure if that distinction is really important, or relevant? If you were trying to make a simple shell, the easiest thing to do would be to implement as much functionality as possible within a single codebase, including handling both 'ls' and 'cd. That would suffice until OP runs into whatever limitations emerge from taking that approach -- that that point, OP would have a better sense of the context needed to make effective queries. – Michael0x2a Aug 7 '17 at 4:41
• I guess my broader point is that when you're not sure what to do, just trying the first thing you can think of even if you have no idea how it'll turn out can be a surprisingly effective way of learning. It's of course not as efficient as forming a complete plan from the start or knowing exactly what to do, but I do think exploratory programming is a valid strategy for acquiring experience and domain knowledge. – Michael0x2a Aug 7 '17 at 4:49
• @Michael0x2a Sounds a bit like BusyBox. – user Aug 7 '17 at 14:48
• I wrote a DOS TSR to share modems on a Novell network (over 25 years ago) this way. One or two crucial books made the difference, but my manager just let me run with the project when I had never done anything this complex before. Got it to work! – user737 Aug 8 '17 at 1:21

You ask an excellent question. That means it's a hard to answer one! The example happens to be an especially difficult little program to write because its operation is so entwined with the way Linux approaches process management. On Windows, for example, one uses the CreateProcess function, which avoids a lot of the oddness of the steps you refer to. However, I recognize you intended your example to just be an example, so I'll try to use it to point out how other software might be written.

I'm a big fan of serendipity. Last night I was watching a bunch of Russian physicists discussing consciousness with the Dalai Lama. In it, one of the physicists brought up a model Einstein used to depict his notion of modern physics. In a letter to Maurice Solovine, Einstein depicted how modern physics works like this:

As he described it, the process for modern physics operated as a loop. On the bottom, we have the experiences we have in life. The arrow sweeping off to the left is intuition. From these experiences in life we distill highly abstract concepts of what the "laws of physics" might be, which are the axioms of science. From those, we deduce very specific testable statements which then encourage us to go out and actually run the tests and make the results part of our experience. The cycle then continues.

I think this is very similar to how a programmer builds up a program. They start from the sum of their experience as a developer, and use that to fuel an intuitive leap to a set of axioms that one thinks can solve the problem at hand. One then builds more concrete statements which suggest ways you can test these axioms to see if they're right. When we actually test it, we experience what really happens, and that fuels the next intuitive leap.

Of course, the challenge as a young developer such as yourself is that you haven't developed that intuition to make the first leap. This is not unusual. Most students get through college without developing this intuition because it's not what school teaches. Instead, you develop that intuition very rapidly once you get out in the field. Don't worry, you're not alone!

To jump start the process, we in the field of computer programming rely on something called "requirements." These are axioms which someone else gives you and says "your project must fit to these axioms." If you're lucky, these requirements were developed by someone who knows something about the problem at hand, so they're pretty close to the correct ones. You then simply have to develop the deduced statements and test them out, developing your own experience.

If you have a very good team lead, they'll give you requirements that are just right enough to keep you productive, but have just enough flaws that you can start developing your own intuition. Often there is something wrong with the requirements, and when you realize this, you are often the only person with the experience required to make the intuitive leap to the new axioms. To give a concrete example, if your team lead has only ever worked on Windows, they may have the Windows axioms in their head, which assume there is a CreateProcess function. If your product needs to work on Linux, their requirements may end up being wrong.

So what happens at this point? Well, you have requirements to find the CreateProcess function on Linux. You deduce that such a function should appear in the Linux API documentation, so you go test that theory. You look through the documentation, and don't find it. Now you have an experience that your team lead didn't have -- your team lead hasn't gone through the documentation like you just did!

Now you know you can reject one of the axioms as "not correct." So what was it's purpose? This is where your intuition comes in. You read the documentation on CreateProcess on Windows, you study what your product (the shell) is supposed to do, and make the intuitive leap that what your team lead really wanted was process management. Now you have an axiom that you've invented. Nobody told you to do "process management." You were told to use CreateProcess. You intuitively theorized the reasons for why they wanted you to do that.

Next comes the development of deduced statements. You have an axiom stating that your job as a developer is to figure out how to do process management. You need to develop a few specific statements which can test this axiom. This might mean making a few statements which you may ask your team lead about. You might tell your team lead that there is no CreateProcess function on Linux, and ask for clarification to see if what he's really looking for is general-purpose process management. Or you might deduce "this sounds like something people have done before, I bet there's an answer on StackExchange." (If it's not on stack exchange, it's probably not worth doing!). Or you might have to take the long route. You might have to say "I understand that Linux has functions to create processes. Which functions are capable of creating new processes?" This would then lead you to go look through the documentation again, and find fork.

At this point, you can fork your process, but obviously that isn't exactly what you want. You want to execute a new program. Another intuitive leap: "given that people execute programs on Linux, there's probably a way to do it." Combining "fork is the way we spawn new processes" with "there's a way to spawn other programs on Linux," you can deduce the idea that there's probably a function which turns one program into another. This will lead you to find exec.

Now all of this is dependent on a balance. Your team lead has to give you enough requirements that your fresh-out-of-school intuition is capable of filling in the gaps, but they also need to be able to listen to you when you come up with new axioms which might change the requirements. Team leads generally have experience. They have the skill to do this (most of them, at lest).

And what if you're on your own? What if you're just sitting at home honing your skills without a team lead to direct you? Well then, you are your own team lead! I recommend starting small. As a general principle, we learn the most when we pick a program that is within our reach, but just outside of our grasp. I wouldn't pick a shell as my first test program, because they're rather hard. However, if you just finished a course on how processes are managed by the operating system, that task may be right up your alley!

And remember to always have fun! It's so much harder to develop intuition when you're not having fun!

• Doco is the way. Whenever you are not busy, raid the doco. – user737 Aug 8 '17 at 1:16

Michael's answer is already pretty spot-on, but I wanted to offer another angle: you need to break your problem down into smaller pieces and then take those pieces on one at a time. Those steps should be in English, not in code. If you aren't sure what code you need, then you haven't broken down your problem into small enough (English) pieces yet.

That process of breaking things down into smaller pieces is one of the most important skills in programming, and it's very hard to teach (and learn!). The only way to learn it is by going through it, over and over again. Think about it this way: every time you had a homework assignment that you didn't know how to start, the real lesson wasn't about syntax or regurgitating code. The real lesson was the process of taking a larger goal and breaking it down into smaller pieces!

For example, let's say I ask you to create a shell from scratch.

The first thing you need to do is break that down in a set of smaller steps, in English. I might start that like this:

1. Create a GUI that resembles a console. Note that this should be further broken down into individual pieces, like create a text area box that shows the output of anything you run, and create a text field box that allows users to input commands.

2. Allow users to run commands using the input field. This will involve taking commands as string values and converting them into system instructions.

3. Take the output from any commands that are run and display it back in the text area we build in step one.

Each of these steps can be further broken down, but hopefully you can see that these bite-sized chunks are a little more manageable. It's easier to google "C++ create text area" than it is to google "C++ create shell from scratch".

A friend told me that:

One of the amazing, and frustrating, things about programming is that there are a million ways to do everything. I've outlined my approach. Your friend has mentioned things they would try. None of that matters. The only thing that matters is how you break the problem down into smaller pieces, and how you think each of those pieces should be approached.

You're worrying about the fact that you didn't "know" the steps that your friend knew. But the thing is: your friend won't "know" the steps that you come up with either!

How are you supposed to know that you need a function which allows you to execute a function without leaving the current process? How are you supposed to came to the conclusion that you need a function to execute builtins, but inside a child process in order to keep the program running normally?

You don't. You first break your problem down into English steps (aka an algorithm), and then you start researching how to implement those English steps in code.

The key is to just give whatever it is a whirl. Try, and see what happens. Quite a lot of code might be written, and most of it will break, but that's how you learn - you gain an experience of what methods work and what methods don't and fill out your figurative toolbox.

First, figure out what the problem is. What are your goals? Is it just to build a working shell? Or must it be done in a certain language? Often, in school, the problems are defined for you, but in the real world, they often aren't. Figure out your parameters and what exactly a solution needs to be like to be a solution.

This is much more important than you'd think. Unfortunately true anecdote: I was writing a program to solve a simple problem. But then I wanted to enhance it. And I wanted to do this and that, and oh, maybe I'll add that - and the code got super unwieldy, with more and more bugs that I needed to work out - and finally I thought about it and realized that most all the code in there was unnecessary for my purposes. I wasted multiple weeks on that project.

Second, start breaking down the problem. With your shell, maybe you think, okay, I need a GUI, and I need a way for user input to 'act' as commands, and those commands to produce some sort of output. Here you should be of course check your requirements to make sure the "chunks" of your program can handle everything that needs to be done.

Third, the exploratory google searches. I kid you not. When I first wanted to write a GUI, I googled "python GUI". And started reading. I found out tkinter was a common package used. So I googled "tkinter GUI". I learned what things that entailed, and I checked whether the method could do what I needed it to do. I explored some of the difficulties I might have. Make sure, for example, that the package you're using to create the GUI doesn't interfere with whatever things you might be using to get user input, or make it into commands for the computer, or show output, or whatever. Make sure it can handle all the things it needs to handle (check the requirements for this).

Fourth, take the plunge and start writing code. Or solving the problem. Whatever. Follow your googlings and your goals and your requirements (those things are important) and your broken-down sections of the problem. When you run into errors, google. When you by chance hit on a better solution, rewrite. Debug and code and sketch out code and google and rethink your design and then tremble in awe at the finished product.

Painful? Yes. Hard? Yes. Do you learn a lot? Oh, man, yes.

• Yes. Google is so much better than a big shelf of manuals and the few books at the local Borders. – user737 Aug 8 '17 at 1:18

## Experience

Gain experience with simpler programs: start with writing ls, what happens when you try to write cd?

## Test-First / Test Driven Development

Practice something like test-first development, but at first with auto automation.

1. Add a smallest possible test (that will fail).
2. Run all tests and see if the new test fails.
3. Write the smallest possible code, to get test to pass.
4. Run tests, to see that all tests pass.
5. Refactor code.
6. Repeat.

## Mentor

Get a mentor. They can guide you. For example tell you which routines you will need. fork (create a new process), exec replace process with external program from file-system, pipe create inter-process pipe, dup copy file descriptor, close close file descriptor, wait wait for a process. Plus other that you should have used before.

## Real project

Find a real project to work on. There are many Free-Software / Open-Source projects that you could work on. These will give real experience.

Free-Software / Open-Source projects are a good source of code for you to read. You could read the source code for bash.

• Reading OS code is just the most amazing way to learn. Nowadays it is far easier to get your hands on it. I used to peruse the Turbo C library code back in the early 90's. – user737 Aug 8 '17 at 15:43

As a developper you rarely reinvent the wheel. It's more like a craftsman's job, doing well known things in a slightly different way.

Let's say you're a carpenter and want to craft a table. You can think of the abstract notion of a table and pore over the infinite number of possible ways to build one, or you can simply look at existing ones and see how they were made.

You can wonder what kind of pegs or screws are available, or you can look at a few tables and see how their pieces are held together. Maybe you'll realize glue was the best solution in some cases.

In my opinion programming is very similar to writing. If you want to tell a good story, you can read a few good books and see how they are made, their style, their purpose. That sure will help you write your own.

So instead of starting with a blank page or raw planks, you could try to tinker with an existing story or table and see how you can change or improve it. First imitating the designs that suit you best, and eventually finding your own style.

If you don't know where to search, it may already be too late.

Part of learning at a university is not just completing the coursework, but in learning how to learn, learning where to look. One gets there by having spent significant time reading/studying related areas/technologies/problems just outside the assigned projects. The farther outside (but hopefully turning out to be still related), the better problems solver one might become. The more one might know which paths possibly lead to wherever.

And if you don't find the function or conclude the existing solution, but know how to keep hunting for ideas, that's how you end up being an inventor (patents, etc.) of perhaps a better way to solve the holistic problem. You might find out that your friend's "solution" wasn't the best.