\documentclass[twoside,a4paper,10pt]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8x]{inputenc}
\usepackage[english]{babel}
\usepackage{enumerate}
\usepackage{amsmath}
\usepackage{hyperref}
\usepackage{minted}
\usepackage{color}
\hypersetup{final=true}
\begin{document}

\newcommand{\pbgen}{\texttt{pbgen} }

\title{\pbgen\\
\small{(\textbf{p}roto\textbf{b}uf \textbf{gen}erator)}}
\maketitle

\tableofcontents

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Introduction}

\pbgen is a program that parses a C native structure decorated with
``special'' comments and generates C code necessary to encode and
decode protobuf representation of the structure, and a pretty-printer
to provide the structure in human-readable form.  The generated code
is kept small, fast, and does not allocate memory.

Main goal of \pbgen is to provide a small (in size), fast, secure and
easy way to work with protobuf-encoded messages in C embedded
programs.  It provides features that are absent in other libraries or
generators which will be discussed later.  The main difference is that
instead of generating code from a \texttt{.proto} file, \pbgen parses
C code where a programmer defined how a protobuf-encoded message
should look like natively, as a C native structure.  This provides a
huge benefit of a really simple API to work with such a structure in
protobuf world --- for a structure named X, \pbgen generates just four
(at most) functions that a programmer wants to use:

\begin{minted}{C}
uint8_t *pb_encode_X(const struct X *in,
                     uint8_t *s, uint8_t *e);

const uint8_t *pb_decode_X(struct X *out,
                           const uint8_t *s, const uint8_t *e);

char *pb_seprint_X(const struct X *value,
                   char *s, char *e,
                   int start_indent, int indent);

void pb_reset_X(struct X *out);
\end{minted}

Programmer doesn't need to look through generated code or make any
wrong assumptions, since the native C structure is the only thing that
matters.  This becomes more apparent in the next section.

\pbgen consists of three files to be copied into project that needs it:
\begin{enumerate}[a)]
	\item \texttt{pb.c} --- implementation.
	\item \texttt{pb.h} --- header used by generated code and (ocasionally) by
		the project itself.
	\item \pbgen ---
		AWK\footnote{\url{http://en.wikipedia.org/wiki/AWK}} script to parse
		and generate code, to be incorporated into the project's build system.

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage
\section{Defining protobuf representation}

The ``special comments'' mentioned in the introduction are of the form
\texttt{//\#}.  This way \pbgen can tell when a comment should
be parsed.

\inputminted[linenos,tabsize=4,mathescape]{C}{def0.c}

On line \ref{enum} \texttt{enum E} is marked as protobuf-enabled by
appending \texttt{//\#}.  This ensures \pbgen will generate a
pretty-printer for this data type.  This is needed since the enum is
used later inside \texttt{struct X}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Decoder, encoder}

Either (or both) \texttt{decoder} or \texttt{encoder} need to be
specified for each structure one wants to use \pbgen with.  Not
specifying \texttt{decoder}, for example, will make \pbgen not
generate a decoder.  This is useful if the structure is only going to
be \textbf{sent} over the wire, but never received.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Field index}

\texttt{\#1 \ldots \#5} specifies protobuf index of a field.

Omitting the number (line \ref{only_pretty}) makes the field used just
for pretty-printing and resetting.  This is useful when the value for
a field is calculated with a specific code, depending on values of
other fields, and the pretty-printed structure needs to have it as
well.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Default values}

The ``default value'' for any field is \mintinline{C}{0} for numbers,
\mintinline{C}{NULL} for pointers, and all zeroes for arrays.  Default
value may be overridden with \texttt{default=\ldots} (line
\ref{default}) and has several effects:
\begin{enumerate}[a)]
	\item a field set to its default value is skipped when encoding
	\item default values aren't printed by pretty-printers \item when
		decoding, fields that aren't set in the protobuf-encoded message
		will be set to their default values
\end{enumerate}

The generated \mintinline{C}{pb_reset_X} function may be used to set
fields of the structure to their default values without touching any
fields that aren't used by \pbgen, like \mintinline{C}{t} and
\mintinline{C}{u}.  This may be used prior to setting fields to their
values and calling \mintinline{C}{pb_encode_X}, so that only
explicitly changed fields are encoded.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Pretty printing}

\texttt{pretty=\ldots} (lines 6 and 7) is optional and is used to
override the enumeration value or field name when pretty-printing.  If
it's not set, the original value or name is used.

The \texttt{print=no} (line \ref{print_no}) means field is hidden when
pretty-printing.

A custom pretty-printing function name may be specified as the value
as well.  The type of the pretty-printer is:
\begin{minted}{C}
char *(*seprint)(const void *in,
                 char *s, char *e,
                 int left, int indent,
                 const struct pb_field *field);
\end{minted}

The arguments are:

\begin{tabular}{ l l }
	in		&	Pointer to the value that needs to be pretty-printed.					\\
	s		&	Pointer to the beginning of the destination buffer.						\\
	e		&	Pointer to past the end of the destination buffer.						\\
	left	&	Left intentation for multi-line values.									\\
	indent	&	Indentation for additional ``levels'' of a tree-like multi-line values.	\\
	field	&	Should be ignored.														\\
\end{tabular}

The returned value must be pointer past the last character written to
the destination buffer.

The header (\texttt{pb.h}) provides a useful function to correctly
print formatted output to the destination buffer:
\begin{minted}{C}
char *seprintf(char *s, char *e, char *fmt, ...);
\end{minted}

\pbgen is able to generate pretty good pretty-printers by default, so
one rarely needs to write their own.

Note: pretty-printing is enabled in compile-time by defining
\mintinline{C}{PB_PRINTERS} macro.  All custom pretty-printers should
be guarded with \mintinline{C}{#ifdef PB_PRINTERS}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Strings}

With \pbgen, strings are either a \mintinline{C}{char} array or a
pointer.

If a string field has a non-\mintinline{C}{const} pointer type, there
needs to be a ``buffer'' field (line \ref{buffer}) added to the
structure that will hold all strings packed tightly.  This helps with
saving crucial memory space on embedded systems and most importantly
allows an easy way to check if a value is set: it's not set when it's
\mintinline{C}{NULL}.

On the other hand, if a string field \textbf{is} \mintinline{C}{const
char *} \textbf{and} has \texttt{num\_at=\ldots} set (required), the
string may be stored outside of the structure.  When decoding, it will
point into the original protobuf message buffer while field specified
with \texttt{num\_at=\ldots} will have its value set to the length of
the string (obviously, it would be wrong for \pbgen to modify the
protobuf message while decoding).

When string doesn't fit into a \mintinline{C}{char} array, or into a
separate ``buffer'' field, the protobuf message parsing will fail with
an error unless \textbf{truncate} is set for the field for that
string.  Automatic truncation of strings isn't enabled by default and
should only be enabled for information that is not required for the
program's correct functioning.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Arrays}

There are two types of arrays in \pbgen: fixed-size and variable-size.
When \texttt{num\_at=\ldots} points to a field where the number of
valid elements is stored (line \ref{variable_size}), the array is said
to be of variable size \textbf{unless} \texttt{size=<number>} is set.
If an array doesn't have \texttt{num\_at=\ldots}, the array has a
constant size (line \ref{fixed_size}).

If a value being parsed has a size different from the one specified
for a fixed-size array, parsing will fail.  This is useful when some
kind of a fixed-size hash needs to be decoded or encoded: \pbgen makes
sure the size is alway correct.

\texttt{num\_at=\^} (line \ref{num_at_above}) is a shortcut to point
\pbgen at the field above.

\pbgen supports packed integers when decoding.

Byte arrays must be of type \mintinline{C}{uint8_t}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Integers and booleans}

The integer types \pbgen expects one to use are standard ones,
including the ones provided by \mintinline{C}{stdint.h}, like
\mintinline{C}{int, uint32_t, int16_t, long, bool}, etc, but
\textbf{not} \mintinline{C}{uint8_t} (only for byte arrays).  Boolean
is just an integer in protobuf encoding scheme, so \pbgen doesn't
differentiate either.  If the type name begins with ``u'' character,
it will be assumed \mintinline{C}{unsigned}.  \pbgen checks if the
value to be stored in a field cannot be represented by the type, in
which case parsing fails.

Because of how protobuf encoding scheme was designed, numbers have two
different ways of being represented on the wire.  By default, in
\pbgen all integers are \textbf{not}
ZigZag\footnote{\url{https://developers.google.com/protocol-buffers/docs/encoding\#signed-integers}}
encoded.  If you need a field representing \texttt{sint32} or
\texttt{sint64} in the \texttt{.proto} file, you have to specify
\texttt{signed} for the field.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Callbacks}

Each time a new value is decoded for a field, a callback function
(line \ref{cb}) pointed by \texttt{callback=\ldots} (line
\ref{callback}) will be called for custom processing.

In general case, callback has type \mintinline{C}{int (*cb)(Parent*,
Child*)}, where the value decoded is a second argument, always as a
pointer.  The first argument points to the structure that holds the
field for which the value is decoded.  In case the value is a byte
array (or a string), additional third argument is added:
\mintinline{C}{int (*cb)(Parent*, uint8_t*, int)}.  It specifies the
size in bytes (length of the string).

If the parsing needs to be stopped with an error, callback may return
a negative number.  To throw away the value parsed, zero should be
returned.  For anything else, a positive number should be returned.
For convenience, \mintinline{C}{"pb.h"} defines the following:

\begin{minted}[tabsize=4]{C}
/* callback return codes */
enum {
	PB_CALLBACK_ERROR = -1, /* parsing should fail */
	PB_CALLBACK_THROW_AWAY, /* throw away the value */
	PB_CALLBACK_OK,
};
\end{minted}

In the example given, \mintinline{C}{each_x} function (line \ref{cb})
calculates the sum of all \mintinline{C}{x->i} numbers and stores it
in the parent structure.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{``Required'' fields}

Specifying \texttt{required} (line \ref{required}) does exactly what
is expected:
\begin{enumerate}[a)]
	\item decoding a structure will fail unless the field is set.
	\item the field is \textbf{always} encoded, no matter the value.
	\item the field is \textbf{always} pretty-printed, no matter the value.
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{``Rest''}

Sometimes there is a need to parse a protobuf message, store it in
memory, and later transfer it back (or somewhere else).  If this needs
to be done in a ``future-proof'' way so that one can add more fields
to the structure on the server, and client is still required to
transfer all the fields back, \texttt{rest=\ldots} (line \ref{rest})
may be used.  The ``rest'' of the protobuf message (which is not
pointed to by any of the fields of the native structure) will be
stored in this buffer.  This information will be stored unchanged to
the destination buffer when encoding the native structure to protobuf
message.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Invoking \pbgen}

The two ways of running \pbgen are:

\begin{minted}{Shell}
$ awk -f pbgen <a.c >a.pb.h
$ awk -f pbgen -v header=a.pb.h <a.c >a_data.pb.h
\end{minted}

When no ``header'' is specified (the first method), all generated
functions are defined as \mintinline{C}{static} and the generated file
\mintinline{C}{"a.pb.h"} is supposed to be included exactly once, in a
single C file which uses these functions.

Second method, on the other hand, makes \pbgen generate two files.
One (\mintinline{C}{"a_data.pb.h"}) containing all the functions and
static ``data'', to be included once in a C file.  And the other one
(\mintinline{C}{"a.pb.h"}) containing the prototypes of the generated
functions.  This method is used when different C files need to work
with protobuf representations of the same data structures.

\end{document}