Lexical Analysis Python

Tokens

A token is the smallest element of a Python program that is meaningful to the compiler.

The Python parser recognizes these kinds of tokens:

Tokens are usually separated by white space.

identifiers

An identifier or variable is anything that the user gives a name to. For example method names, variable names, dictionary names, class names, etc are all identifiers.

In Python, keywords are reserved identifiers which cannot be used as names for the variables in a program.

Creating variables and assigning values

To create a variable in Python, all you need to do is specify the variable name, and then assign a value to it.

<variable name> = <value>

Variable assignment works from left to right. So the following will give you an syntax error :

Python
0 = x
=> Output: SyntaxError: can't assign to literal

Tip

You can not use python’s keywords as a valid variable name. You can see the list of keyword by:

Python
import keyword
print(keyword.kwlist)

Rules for variable naming

1. Variables names must start with a letter or an underscore.

2. The remainder of your variable name may consist of letters, numbers and underscores.

Python
has_0_in_it = "Still Valid"

3. Names are case sensitive.

Python
x = 9
y = X*5
=>NameError: name 'X' is not defined

Literals

Literal is the value assigned to a variable :

# 20 is the Integer literal here
int a=20

Literals categories :

  • Integer literals
  • String literals
  • Boolean literals
  • Character literals
  • Float literals
  • Null literals

punctuators

Used to implement the grammatical and structure of a Syntax.Following are the python punctuators:

( ) { } [ ] ; , . \ # @ : = ‘ “