C++ Parser

C++ Parser
Prev	Chapter 3. The Parsers	Next

The C++ parser reads C++ source code and generates an AST. It is the most advanced Synopsis parser, using the following components:

A modified version of UCPP by Thomas Pornin. The modifications are to improve C++ standards conformance, and add features needed by Synopsis such as storing information about #includes and #defines, and recording the position of every macro expansion.
A modified version of OpenC++ (OCC) by Shigeru Chiba and others. OCC is used to generate a "parse-tree" from the preprocessed source code. The parse-tree is a data structure that directly represents the syntax of the source code. OCC uses Hans Boehm's garbage collector, but this can be disabled by passing --disable-gc to configure on platforms like Cygwin where it causes problems.
Code to "walk" the parse-tree and build an Abstract Syntax Tree. A symbol table is also maintained which is used to correctly resolve all types according to the C++ standard.
Code to record the positions of all macro expansions, keywords, comments, type and name references if SXR or XREF (see below) is enabled.
Miscellenous code to convert the C++ AST to a Python data structure, maintain the list of files, implement a Python module etc.

Like a compiler, the C++ parser has to parse the whole translation unit for each file to make sure it has the correct type references. Unfortunately this means it has to wade through all the standard library headers for each input file which, like with a compiler without precompiled headers, can take a long time. Luckily the C++ parser is written in C++ for speed and has been optimised in critical areas. If you are not using SXR or XREF then it skips all method bodies which speeds things up, and even if you are it only has to resolve type and function call information - there is no code generation or optimisation going on.

Fortunately you can parse multiple headers in one go if you have a main header file which includes everything else (such as how python2.2/Python.h includes the other 41 Python header files), and if you are using a Project config file. See the multiple_files option below.

SXR and XREF

So what are SXR and XREF?

SXR stands for Synopsis Cross Reference. It is a combined feature of the C++ parser and the HTML formatter which displays the source code but with hyperlinks for every type reference and every name reference.

This means that if you click on a method signature in the source code, you will be taken to the documentation. Each declaration in the documentation will also have a link to the source code. Variable and type references in the source code also have "titles", which show as tooltips if you hover the mouse over them, telling you what they are (eg: "parameter Person" or "member variable std::string"). Note that because Synopsis uses a real C++ parser the links are always to the right declarations - unlike other systems which use purely string based references which are often ambiguous!

XREF stands for Cross Reference. This feature creates an index of all references to every declaration, including function and method calls. This allows you to follow a call graph, or see where a method or class is used (useful for refactoring!)

Both of these features need to be coordinated with both the C++ parser and the HTML formatter.

For SXR you need: C++ Parser: syntax_prefix. HTML Formatter: The pages option must include 'FileSource'.

For XREF you need: C++ Parser: xref_prefix. Linker: The operations list must include 'XRefCompiler', and XRefCompiler must be configured. HTML Formatter: The pages option must include 'XRefPages', and XRefPages must be configured.

See the respective sections for details of these configuration options.

Note that in the current release there are some limitations with respect to expressions that are understood, and template instantiations are not supported very well. This means that in the source code some methods will not be hyperlinked and some cross references will be missing.

Options

name (string)

Name of this config object: Must be 'C++'

verbose (boolean)

Verbosity flag. For config files, this attribute is set by the constructor, but only if 'verbose' was passed as a config option.

Can be set to true from the command line with "-Wp,-v"

main_file (boolean)

Flag that selects whether Synopsis should only store the AST generated from the file being processed, and not from any #included files. If you are using the multiple_files option then the AST generated from each of the named files is stored, but not from any other #included files.

The default is 1 (true).

Can be set to true from the command line with "-Wp,-m"

basename (string)

A file path to strip from the start of all filenames before storing. Setting this option for example will remove any redundant parent directories in the HTML FileListing page.

Can be set from the command line with, e.g.: "-Wp,-b,../some/basename/"

include_path (list of strings)

A list of paths to add to the include path. For example: ['/usr/local/corba/', '../include']

Can be set from the command line with, e.g.: "-Wp,-I,/usr/local/corba/,-I,../include"

defines (list of strings)

A list of defines to pass to the preprocessor. For example: ['FOO', 'BAR=true']

Can be set from the command line with, e.g.: "-Wp,-D,FOO,-D,BAR=true"

preprocessor (string)

Which preprocessor to use. Not setting this causes the builtin ucpp to be used, which can track macro expansions when doing SXR stuff and extract macro definitions for the documentation. Setting it to 'gcc' will cause gcc (well, really g++) to be used instead, for use only in cases when ucpp can't parse your standard libraries (usually because of compiler specific syntax). There are no other settings.

Can be set to gcc from the command line with "-Wp,-g"

extract_tails (boolean)

If set to true, then the parser will look for trailing comments before close braces. If it finds them, it will create dummy declarations to hold the comments. If you set this, you should probably also use the 'dummy' or 'prev' comment processors in the Linker options otherwise you will see the Dummy declarations in your documentation.

Can be set to true from the command line with "-Wp,-t"

storage (string)

Deprecated, see syntax_file which is the same thing.

syntax_prefix (string)

If set, must be a string which defines a prefix to store syntax info into. The source filenames are appended to the prefix to create the output filename, so it should be a directory name to avoid clashes (there is no suffix!). For example, if your file is "src/foo.cc" and prefix is "syn/" then the syntax information will be stored in "syn/src/foo.cc".

xref_prefix (string)

If set, must be a string which defines a prefix to store xref info into. See syntax_prefix.

syntax_file (string)

If set, must be a string with the file to store syntax info into. Setting this also causes the parser to look more carefully at the input file, so that it can generate the links from inside blocks of code (otherwise it skips over them). Note that the syntax info can only hold syntax information about one source file, so this option is of limited use in a config file unless you only have one file.

Can be set from the command line with, e.g: "-Wp,-s,syn/". You might use it in your Makefile as "-Wp,-s,$<.links" ($< is the first dependency, usually the source file being parsed).

xref_file (string)

If set, must be a string with the file to store xref info into. Setting this also causes the parser to look more carefully at the input file, so that it can generate the links from inside blocks of code (otherwise it skips over them). Note that the xref info can only hold xref information about one source file, so this option is of limited use in a config file unless you only have one file.

Can be set from the command line with, e.g.: "-Wp,-x,xref/". You might use it in your Makefile as "-Wp,-x,$<.xref" ($< is the first dependency, usually the source file being parsed).

fake_std (boolean)

If set, this causes the parser to construct a fake using directive from the std namespace to the global namespace. In effect, this fixes problems seen with using the stdc++ headers for gcc 2.95.x where most things dont get placed in the std namespace.

Can be set to true from the command line with "-Wp,-f"

multiple_files (boolean)

This option can only be used with the Project file format. If set to true then the parser accepts a list of files, and stores the generated AST for all of them in the one output. Only one file is parsed, so the others must be included by the main file (or by one of the files it includes, etc.). If syntax_prefix and/or xref_prefix is set then the extra files will get syntax and/or xref info recorded into the appropriate files. Only one AST is output, but it is as if the ASTs for the individual files were already linked. That, and the fact that system headers will only be parsed once, can lead to huge time savings!

To use this option, your Project file must have a single SourceAction connected to this ParserAction. The SourceAction should have a Simple rule first which is the main sourcefile, and any number of other rules to select other files to record the AST for.