\documentclass[twoside,a4paper,10pt]{article}\usepackage[T1]{fontenc}\usepackage[utf8x]{inputenc}\usepackage[english]{babel}\usepackage{enumerate}\usepackage{amsmath}\usepackage{hyperref}\usepackage{minted}\usepackage{color}\hypersetup{final=true}\begin{document} \newcommand{\pbgen}{\texttt{pbgen} } \title{\pbgen\\\small{(\textbf{p}roto\textbf{b}uf \textbf{gen}erator)}}\maketitle \tableofcontents %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\newpage\section{Introduction} \pbgen is a program that parses a C native structure decorated with``special'' comments and generates C code necessary to encode anddecode protobuf representation of the structure, and a pretty-printerto provide the structure in human-readable form.  The generated codeis kept small, fast, and does not allocate memory. Main goal of \pbgen is to provide a small (in size), fast, secure andeasy way to work with protobuf-encoded messages in C embeddedprograms.  It provides features that are absent in other libraries orgenerators which will be discussed later.  The main difference is thatinstead of generating code from a \texttt{.proto} file, \pbgen parsesC code where a programmer defined how a protobuf-encoded messageshould look like natively, as a C native structure.  This provides ahuge benefit of a really simple API to work with such a structure inprotobuf world --- for a structure named X, \pbgen generates just four(at most) functions that a programmer wants to use: \begin{minted}{C}uint8_t *pb_encode_X(const struct X *in,                     uint8_t *s, uint8_t *e); const uint8_t *pb_decode_X(struct X *out,                           const uint8_t *s, const uint8_t *e); char *pb_seprint_X(const struct X *value,                   char *s, char *e,                   int start_indent, int indent); void pb_reset_X(struct X *out);\end{minted} Programmer doesn't need to look through generated code or make anywrong assumptions, since the native C structure is the only thing thatmatters.  This becomes more apparent in the next section. \pbgen consists of three files to be copied into project that needs it:\begin{enumerate}[a)]	\item \texttt{pb.c} --- implementation.	\item \texttt{pb.h} --- header used by generated code and (ocasionally) by		the project itself.	\item \pbgen ---		AWK\footnote{\url{http://en.wikipedia.org/wiki/AWK}} script to parse		and generate code, to be incorporated into the project's build system. \end{enumerate} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\newpage\section{Defining protobuf representation} The ``special comments'' mentioned in the introduction are of the form\texttt{//\#}.  This way \pbgen can tell when a comment shouldbe parsed. \inputminted[linenos,tabsize=4,mathescape]{C}{def0.c} On line \ref{enum} \texttt{enum E} is marked as protobuf-enabled byappending \texttt{//\#}.  This ensures \pbgen will generate apretty-printer for this data type.  This is needed since the enum isused later inside \texttt{struct X}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{Decoder, encoder} Either (or both) \texttt{decoder} or \texttt{encoder} need to bespecified for each structure one wants to use \pbgen with.  Notspecifying \texttt{decoder}, for example, will make \pbgen notgenerate a decoder.  This is useful if the structure is only going tobe \textbf{sent} over the wire, but never received. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{Field index} \texttt{\#1 \ldots \#5} specifies protobuf index of a field. Omitting the number (line \ref{only_pretty}) makes the field used justfor pretty-printing and resetting.  This is useful when the value fora field is calculated with a specific code, depending on values ofother fields, and the pretty-printed structure needs to have it aswell. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{Default values} The ``default value'' for any field is \mintinline{C}{0} for numbers,\mintinline{C}{NULL} for pointers, and all zeroes for arrays.  Defaultvalue may be overridden with \texttt{default=\ldots} (line\ref{default}) and has several effects:\begin{enumerate}[a)]	\item a field set to its default value is skipped when encoding	\item default values aren't printed by pretty-printers \item when		decoding, fields that aren't set in the protobuf-encoded message		will be set to their default values\end{enumerate} The generated \mintinline{C}{pb_reset_X} function may be used to setfields of the structure to their default values without touching anyfields that aren't used by \pbgen, like \mintinline{C}{t} and\mintinline{C}{u}.  This may be used prior to setting fields to theirvalues and calling \mintinline{C}{pb_encode_X}, so that onlyexplicitly changed fields are encoded. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{Pretty printing} \texttt{pretty=\ldots} (lines 6 and 7) is optional and is used tooverride the enumeration value or field name when pretty-printing.  Ifit's not set, the original value or name is used. The \texttt{print=no} (line \ref{print_no}) means field is hidden whenpretty-printing. A custom pretty-printing function name may be specified as the valueas well.  The type of the pretty-printer is:\begin{minted}{C}char *(*seprint)(const void *in,                 char *s, char *e,                 int left, int indent,                 const struct pb_field *field);\end{minted} The arguments are: \begin{tabular}{ l l }	in		&	Pointer to the value that needs to be pretty-printed.					\\	s		&	Pointer to the beginning of the destination buffer.						\\	e		&	Pointer to past the end of the destination buffer.						\\	left	&	Left intentation for multi-line values.									\\	indent	&	Indentation for additional ``levels'' of a tree-like multi-line values.	\\	field	&	Should be ignored.														\\\end{tabular} The returned value must be pointer past the last character written tothe destination buffer. The header (\texttt{pb.h}) provides a useful function to correctlyprint formatted output to the destination buffer:\begin{minted}{C}char *seprintf(char *s, char *e, char *fmt, ...);\end{minted} \pbgen is able to generate pretty good pretty-printers by default, soone rarely needs to write their own. Note: pretty-printing is enabled in compile-time by defining\mintinline{C}{PB_PRINTERS} macro.  All custom pretty-printers shouldbe guarded with \mintinline{C}{#ifdef PB_PRINTERS}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{Strings} With \pbgen, strings are either a \mintinline{C}{char} array or apointer. If a string field has a non-\mintinline{C}{const} pointer type, thereneeds to be a ``buffer'' field (line \ref{buffer}) added to thestructure that will hold all strings packed tightly.  This helps withsaving crucial memory space on embedded systems and most importantlyallows an easy way to check if a value is set: it's not set when it's\mintinline{C}{NULL}. On the other hand, if a string field \textbf{is} \mintinline{C}{constchar *} \textbf{and} has \texttt{num\_at=\ldots} set (required), thestring may be stored outside of the structure.  When decoding, it willpoint into the original protobuf message buffer while field specifiedwith \texttt{num\_at=\ldots} will have its value set to the length ofthe string (obviously, it would be wrong for \pbgen to modify theprotobuf message while decoding). When string doesn't fit into a \mintinline{C}{char} array, or into aseparate ``buffer'' field, the protobuf message parsing will fail withan error unless \textbf{truncate} is set for the field for thatstring.  Automatic truncation of strings isn't enabled by default andshould only be enabled for information that is not required for theprogram's correct functioning. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{Arrays} There are two types of arrays in \pbgen: fixed-size and variable-size.When \texttt{num\_at=\ldots} points to a field where the number ofvalid elements is stored (line \ref{variable_size}), the array is saidto be of variable size \textbf{unless} \texttt{size=<number>} is set.If an array doesn't have \texttt{num\_at=\ldots}, the array has aconstant size (line \ref{fixed_size}). If a value being parsed has a size different from the one specifiedfor a fixed-size array, parsing will fail.  This is useful when somekind of a fixed-size hash needs to be decoded or encoded: \pbgen makessure the size is alway correct. \texttt{num\_at=\^} (line \ref{num_at_above}) is a shortcut to point\pbgen at the field above. \pbgen supports packed integers when decoding. Byte arrays must be of type \mintinline{C}{uint8_t}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{Integers and booleans} The integer types \pbgen expects one to use are standard ones,including the ones provided by \mintinline{C}{stdint.h}, like\mintinline{C}{int, uint32_t, int16_t, long, bool}, etc, but\textbf{not} \mintinline{C}{uint8_t} (only for byte arrays).  Booleanis just an integer in protobuf encoding scheme, so \pbgen doesn'tdifferentiate either.  If the type name begins with ``u'' character,it will be assumed \mintinline{C}{unsigned}.  \pbgen checks if thevalue to be stored in a field cannot be represented by the type, inwhich case parsing fails. Because of how protobuf encoding scheme was designed, numbers have twodifferent ways of being represented on the wire.  By default, in\pbgen all integers are \textbf{not}ZigZag\footnote{\url{https://developers.google.com/protocol-buffers/docs/encoding\#signed-integers}}encoded.  If you need a field representing \texttt{sint32} or\texttt{sint64} in the \texttt{.proto} file, you have to specify\texttt{signed} for the field. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{Callbacks} Each time a new value is decoded for a field, a callback function(line \ref{cb}) pointed by \texttt{callback=\ldots} (line\ref{callback}) will be called for custom processing. In general case, callback has type \mintinline{C}{int (*cb)(Parent*,Child*)}, where the value decoded is a second argument, always as apointer.  The first argument points to the structure that holds thefield for which the value is decoded.  In case the value is a bytearray (or a string), additional third argument is added:\mintinline{C}{int (*cb)(Parent*, uint8_t*, int)}.  It specifies thesize in bytes (length of the string). If the parsing needs to be stopped with an error, callback may returna negative number.  To throw away the value parsed, zero should bereturned.  For anything else, a positive number should be returned.For convenience, \mintinline{C}{"pb.h"} defines the following: \begin{minted}[tabsize=4]{C}/* callback return codes */enum {	PB_CALLBACK_ERROR = -1, /* parsing should fail */	PB_CALLBACK_THROW_AWAY, /* throw away the value */	PB_CALLBACK_OK,};\end{minted} In the example given, \mintinline{C}{each_x} function (line \ref{cb})calculates the sum of all \mintinline{C}{x->i} numbers and stores itin the parent structure. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{``Required'' fields} Specifying \texttt{required} (line \ref{required}) does exactly whatis expected:\begin{enumerate}[a)]	\item decoding a structure will fail unless the field is set.	\item the field is \textbf{always} encoded, no matter the value.	\item the field is \textbf{always} pretty-printed, no matter the value.\end{enumerate} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\subsection{``Rest''} Sometimes there is a need to parse a protobuf message, store it inmemory, and later transfer it back (or somewhere else).  If this needsto be done in a ``future-proof'' way so that one can add more fieldsto the structure on the server, and client is still required totransfer all the fields back, \texttt{rest=\ldots} (line \ref{rest})may be used.  The ``rest'' of the protobuf message (which is notpointed to by any of the fields of the native structure) will bestored in this buffer.  This information will be stored unchanged tothe destination buffer when encoding the native structure to protobufmessage. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\section{Invoking \pbgen} The two ways of running \pbgen are: \begin{minted}{Shell}$ awk -f pbgen <a.c >a.pb.h$ awk -f pbgen -v header=a.pb.h <a.c >a_data.pb.h\end{minted} When no ``header'' is specified (the first method), all generatedfunctions are defined as \mintinline{C}{static} and the generated file\mintinline{C}{"a.pb.h"} is supposed to be included exactly once, in asingle C file which uses these functions. Second method, on the other hand, makes \pbgen generate two files.One (\mintinline{C}{"a_data.pb.h"}) containing all the functions andstatic ``data'', to be included once in a C file.  And the other one(\mintinline{C}{"a.pb.h"}) containing the prototypes of the generatedfunctions.  This method is used when different C files need to workwith protobuf representations of the same data structures. \end{document}