27 lines · 864 B · Text
RawDownload
= tokenize: split string into words     include "sys.m";    sys := load Sys Sys->PATH;     tokenize: fn(s, delim: string): (int, list of string); Tokenize  breaks  s  into  words  separated  by characters indelim.  The returned tuple holds the number of words  and  anordered list of those words (whose hd gives the leftmost wordfrom s). Words are delimited by the maximal sequences of any characterfrom  the  delim string.  Tokenize skips delimiter charactersat the beginning and  end  of  s,  so  each  element  in  thereturned list has non-zero length. If s is nil or contains no words, tokenize returns a count ofzero and a nil list. Delim may be nil or the empty string, specifying no delimitercharacters.  The resulting word list will be nil (if s is nilor the empty string) or a single-item list with a copy of s. == see ALSOsys-intro(2)