I was teaching an intro to Programming class, full of social scientists. After explaining that the basic building blocks of a program are called tokens, I took them through Keywords, Identifiers, Literals etc.

I was asked a question. And have been reading to find answers in books and online. They asked if we group the basic elements of a program under the word "token", then what word groups the other program elements. e.g. expressions, statements and blocks, that are derived from tokens?

Help with a reference or a term that classifies these constructs as a group.

  • $\begingroup$ I wonder whether I just managed now to parse your question as you intended. Are you asking, what is the name for non-token language element? In a grammar the literals (eg tokens) are "terminals" & everything else (rule or its associated set of expressions/strings/trees) is a "non-terminal". But I had thought from your last line you that were asking about acategory containing just expression/statement/block. (Whatever you think of those as.) Can you clarify? $\endgroup$
    – philipxy
    Oct 11 '19 at 19:16
  • $\begingroup$ Well, what about "program elements" ? $\endgroup$ Oct 18 '19 at 11:52
  • $\begingroup$ Program elements is the main topic. The issue is some program elements are called tokens and these are the atomic pieces, what will those pieces aggregated into expressions, statements and blocks be designated by collectively? $\endgroup$ Oct 26 '19 at 1:14
  • 1
    $\begingroup$ Perhaps "Token groups" covers it. Or, perhaps "Compound tokens" would fit. My preference is for "Expressions", "Statements", "Blocks". Each is different, yet composed of tokens. In social science speak, you have "taboos", "mores" and "norms". Similar functions, culturally, without being in the same "classification" without creating one artificially. $\endgroup$ Oct 27 '19 at 2:16
  • $\begingroup$ I like the compound tokens term. Comes closest to what I'm looking for. $\endgroup$ Oct 29 '19 at 0:20

People create categories as needed to manage complexity. A language designer has defined syntactic & semantic things--including categories of things--relevant to their purposes & so can anyone--specific or fuzzy. It's unhelpful & misconceived to expect that somehow categories are out there to be uncovered or that everything must be in a category or that any pattern one gets a glimpse of is evidence of a category out there to be uncovered. This is a case of the fallacy of Reification. Of mathematics, Kronecker said, "God made the integers; all else is the work of man."

In programming languages, expressions, statements and blocks are categories of language strings or trees with associated grammar production rules. Relevant helpful larger grouping for these are things like definitions, programs & compilation units. But those are ad hoc per the language.

Generally, what helpful syntactic categories there are is driven by what helpful semantic categories there are. Sadly, programming languages' semantics are typically poorly taught. Even though its semantics are the raison d'etre of a language.


This is a surprisingly complex question. In many languages one can distinguish between commands and expressions. A command is something that changes the state of the computation. An expression is something that returns a value, but doesn't change the state. This is actually a good mental model even in languages that can confuse the two ideas in small or large ways.

Some languages consider statements to be commands, but in C, for example, an assignment statement is also an expression that has a value equal to the value assigned to the variable on the left. That may or may not be the value of the expression on the right, actually, as when a short is assigned to a long.

Likewise blocks can either be commands or expressions in most languages. A function block is an expression, returning what the function returns (more or less) but it can also have command-like properties if it changes the state generally, say by modifying a non-local variable. But the if-clause in an if statement is usually just a command.

So, I don't think there is a single unifying category for all three of your items, but the distinction between commands and expressions is useful.

It is useful for another reason as well. In C a "function" can return a value or not (void functions). But programs are easier to understand if a function that returns a value does not also change anything in the non-local state. Then functions when invoked will be either expressions (the value returned) or commands (the state changes) but never both.

But C also has a distinction that even command-like functions often return an error state indicator, zero if there are no errors. So common practice gets in the way of the useful rule in the paragraph above. C++ can use exceptions instead of error codes, of course.

But in more "modern" languages, the distinction between expressions and commands is maintained with a bit more clarity. Java, for example.


I just learned quite a bit from Buffy's wonderful answer! I would only add that a token is simple the smallest unit of text that carries semantic meaning to the compiler. Thus,

int func(int input){ has 7 tokens (four word tokens and three symbol tokens). An escape character in a string such as \", then, is a note to the tokenizer.

How they are later grouped and interpreted are decisions for later in the compilation process, based on the contents of each token.

I don't think that I would try to relate token to either of your other terms, as the relationship is in no way straightforward. Expressions can be tokens (such as 42), or can be groups of tokens (x + 5). That depends on the contents of the tokens themselves.

  • $\begingroup$ Ah yes, tokens are syntactic elements (the scanner handles these). The other three are semantic elements. The parser handles them structurally and the semantic analyzer imputes meaning to them. $\endgroup$
    – Buffy
    Oct 7 '19 at 20:50
  • $\begingroup$ I know expressions, statements & blocks end up first decomposed to tokens then after subsequent analysis meaning is imputted to. But that's not the question. My students are asking if tokens are the small elements from which these more "sophisticated" combinations are built, shouldn't they conceptually be grouped in a category of their own even if for learning purposes? Hence, as social scientists used to arguing over concepts, they demanded a category to embrace these "higher" programming constructs. Do we have one or can someone attempt to coin one for me? $\endgroup$ Oct 8 '19 at 12:16
  • 1
    $\begingroup$ Characters & strings of characters (used in tokenizing) are smaller syntactic & semantic units than tokens. You yourself have just talked about characters that mean something to the compiler. We normally talk about a "pass" of a compiler for which we posit new atomic parts, non-atomic groupings, input & output. $\endgroup$
    – philipxy
    Oct 10 '19 at 6:55
  • $\begingroup$ Yea, philipxy, the atomic/non-atomic characterization is a close match. However, non-atomic applies to a much larger set than expressions, statements & blocks. That's why I hesitated to use that. However, as you said earlier catgories are. made as needed even if imperfect. Perhaps I'll settle for that for now then keep searching and thinking of a tighter term. Thanks $\endgroup$ Oct 11 '19 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.