The document was prepared to help me came up with a parallel build solution, but to no avail.
Showing the changes to misc/notes (1 of 1 file). Show all
Unified · Split
| @@ -1,227 +0,0 @@ | |||
| 1 | There is some unclaimed parallelism inside the build process: | ||
| 2 | |||
| 3 | a) all C compilation can be done in parallel | ||
| 4 | |||
| 5 | b) ditto all _independent_ OCaml compilation (i.e. things having the | ||
| 6 | same indentation (and growing from the same "root") in the | ||
| 7 | following build.bash output) | ||
| 8 | |||
| 9 | The "neat" solution of how to get there is elusive though. Currently: | ||
| 10 | |||
| 11 | - bash build.bash build | ||
| 12 | utils.ml -> utils.cmo | ||
| 13 | keys.ml -> keys.cmo | ||
| 14 | wsi.mli -> wsi.cmi | ||
| 15 | utf8syms.ml -> utf8syms.cmo | ||
| 16 | lablGL/raw.ml -> lablGL/raw.cmo | ||
| 17 | lablGL/gl.ml -> lablGL/gl.cmo | ||
| 18 | lablGL/glPix.ml -> lablGL/glPix.cmo | ||
| 19 | lablGL/glMisc.ml -> lablGL/glMisc.cmo | ||
| 20 | lablGL/glTex.ml -> lablGL/glTex.cmo | ||
| 21 | lablGL/glFunc.ml -> lablGL/glFunc.cmo | ||
| 22 | lablGL/glDraw.ml -> lablGL/glDraw.cmo | ||
| 23 | lablGL/glArray.ml -> lablGL/glArray.cmo | ||
| 24 | parser.ml -> parser.cmo | ||
| 25 | build/confstruct.ml -> confstruct.cmo | ||
| 26 | config.ml -> config.cmo | ||
| 27 | ffi.ml -> ffi.cmo | ||
| 28 | glutils.ml -> glutils.cmo | ||
| 29 | lablGL/glMat.ml -> lablGL/glMat.cmo | ||
| 30 | listview.ml -> listview.cmo | ||
| 31 | help.mli -> help.cmi | ||
| 32 | help.ml -> help.cmo | ||
| 33 | lablGL/glClear.ml -> lablGL/glClear.cmo | ||
| 34 | main.ml -> main.cmo | ||
| 35 | link.c -> link.o | ||
| 36 | cutils.c -> cutils.o | ||
| 37 | version.c -> version.o | ||
| 38 | lablGL/ml_gl.c -> lablGL/ml_gl.o | ||
| 39 | lablGL/ml_glarray.c -> lablGL/ml_glarray.o | ||
| 40 | lablGL/ml_raw.c -> lablGL/ml_raw.o | ||
| 41 | 9 sec | ||
| 42 | |||
| 43 | IOW - whole build (including lablGL, but excluding mupdf/ocaml and its | ||
| 44 | dependencies) takes less than 10 seconds even on this circa 2014 | ||
| 45 | MacMini. | ||
| 46 | |||
| 47 | Making things parallel ought to bring some benefits to the build on | ||
| 48 | the Raspberry Pi3 and it might even benefit single core PPC MacMini | ||
| 49 | due to interleaving IO/CPU bound parts (but this is just that - | ||
| 50 | baseless speculation) | ||
| 51 | |||
| 52 | ============================================================= | ||
| 53 | |||
| 54 | With following applied, build.bash will produce a Makefile that builds | ||
| 55 | llpp. Building things via this Makefile (i.e. `\time -pv make -jN') | ||
| 56 | seems to suggest that there is some (a lot) parallelism that can be | ||
| 57 | harvested even for OCaml code (parallelism for interface deprived | ||
| 58 | OCaml code (as is the case with llpp) is limited) | ||
| 59 | |||
| 60 | diff --git a/build.bash b/build.bash | ||
| 61 | index e9a2919..37382df 100755 | ||
| 62 | --- a/build.bash | ||
| 63 | +++ b/build.bash | ||
| 64 | @@ -1,6 +1,8 @@ | ||
| 65 | #!/bin/bash | ||
| 66 | set -eu | ||
| 67 | |||
| 68 | +:>Makefile | ||
| 69 | + | ||
| 70 | now() { date +%s; } | ||
| 71 | S=$(now) | ||
| 72 | vecho() { ${vecho-:} "$*"; } | ||
| 73 | @@ -159,6 +161,7 @@ bocaml2() { | ||
| 74 | |||
| 75 | cmd="ocamlc $(oflags $o) -c -o $o $s" | ||
| 76 | keycmd="digest $o $s $(< $o.depl)" | ||
| 77 | + printf "$o: $s $(< $o.depl)\n\t$cmd\n" >>Makefile | ||
| 78 | isfresh "$o" "$overs$cmd$(eval $keycmd)" || { | ||
| 79 | printf "%*.s%s -> %s\n" $n '' "${s#$srcd/}" "${o#$outd/}" | ||
| 80 | eval "$cmd || die '$cmd failed'" | ||
| 81 | @@ -191,6 +194,7 @@ bocaml() ( | ||
| 82 | *) false;; | ||
| 83 | esac && { | ||
| 84 | local s1=${s#$srcd/} | ||
| 85 | + echo "$outd/${s1%.ml}.cmo: $o" >>Makefile | ||
| 86 | bocaml1 $n "$s" "$outd/${s1%.ml}.cmo" "${o#$outd/}" | ||
| 87 | } || true | ||
| 88 | cycle=$cycle1 | ||
| 89 | @@ -203,6 +207,7 @@ bocamlc() { | ||
| 90 | local cmd="ocamlc $cc-ccopt \"$(cflags $o) -MMD -MF $o.dep -MT_ -o $o\" $s" | ||
| 91 | test -r $o.dep && read _ d <$o.dep || d= | ||
| 92 | local keycmd='digest $o $d' | ||
| 93 | + printf "$o: $s $d\n\t$cmd\n" >>Makefile | ||
| 94 | isfresh "$o" "$cmd$(eval $keycmd)" || { | ||
| 95 | printf "%s -> %s\n" "${s#$srcd/}" "${o#$outd/}" | ||
| 96 | eval "$cmd || die '$cmd failed'" | ||
| 97 | @@ -229,6 +234,7 @@ ver=$(cd $srcd && git describe --tags --dirty) || ver=unknown | ||
| 98 | |||
| 99 | cmd="(. $srcd/genconfstr.sh >$outd/confstruct.ml)" | ||
| 100 | keycmd="digest $srcd/genconfstr.sh $outd/confstruct.ml" | ||
| 101 | +printf "$outd/confstruct.ml: $srcd/genconfstr.sh\n\t$cmd\n" >>Makefile | ||
| 102 | isfresh "$outd/confstruct.ml" "$cmd$(eval $keycmd)" || { | ||
| 103 | echo "generating $outd/confstruct.ml" | ||
| 104 | eval "$cmd || die genconfstr.sh failed" | ||
| 105 | @@ -295,6 +301,7 @@ fi | ||
| 106 | ord=$(grep -v \.cmi $outd/ordered) | ||
| 107 | cmd="ocamlc -custom $libs -o $outd/llpp $cobjs $(echo $ord) -cclib \"$clibs\"" | ||
| 108 | keycmd="digest $outd/llpp $cobjs $ord $mulibs" | ||
| 109 | +printf "$outd/llpp: $cobjs $(echo $ord)\n\t$cmd\n" >>Makefile | ||
| 110 | isfresh "$outd/llpp" "$cmd$(eval $keycmd)" || { | ||
| 111 | echo linking $outd/llpp | ||
| 112 | eval "$cmd || die '$cmd failed'" | ||
| 113 | |||
| 114 | ======================================== | ||
| 115 | ~/xsrc/llpp | ||
| 116 | - rm build/*.cm* build/lablGL/*.cm* build/wsi/x11/*.cm* build/*.o build/lablGL/*.o build/wsi/x11/*.o | ||
| 117 | ~/xsrc/llpp | ||
| 118 | - \time -pv bash build.bash build | ||
| 119 | utils.ml -> utils.cmo | ||
| 120 | keys.ml -> keys.cmo | ||
| 121 | wsi.mli -> wsi.cmi | ||
| 122 | wsi/x11/wsi.mli -> wsi/x11/wsi.cmi | ||
| 123 | wsi/x11/wsi.ml -> wsi/x11/wsi.cmo | ||
| 124 | utf8syms.ml -> utf8syms.cmo | ||
| 125 | lablGL/raw.ml -> lablGL/raw.cmo | ||
| 126 | lablGL/gl.ml -> lablGL/gl.cmo | ||
| 127 | lablGL/glPix.ml -> lablGL/glPix.cmo | ||
| 128 | lablGL/glMisc.ml -> lablGL/glMisc.cmo | ||
| 129 | lablGL/glTex.ml -> lablGL/glTex.cmo | ||
| 130 | lablGL/glFunc.ml -> lablGL/glFunc.cmo | ||
| 131 | lablGL/glDraw.ml -> lablGL/glDraw.cmo | ||
| 132 | lablGL/glArray.ml -> lablGL/glArray.cmo | ||
| 133 | parser.ml -> parser.cmo | ||
| 134 | build/confstruct.ml -> confstruct.cmo | ||
| 135 | config.ml -> config.cmo | ||
| 136 | ffi.ml -> ffi.cmo | ||
| 137 | glutils.ml -> glutils.cmo | ||
| 138 | lablGL/glMat.ml -> lablGL/glMat.cmo | ||
| 139 | listview.ml -> listview.cmo | ||
| 140 | help.mli -> help.cmi | ||
| 141 | help.ml -> help.cmo | ||
| 142 | lablGL/glClear.ml -> lablGL/glClear.cmo | ||
| 143 | main.ml -> main.cmo | ||
| 144 | link.c -> link.o | ||
| 145 | cutils.c -> cutils.o | ||
| 146 | version.c -> version.o | ||
| 147 | lablGL/ml_gl.c -> lablGL/ml_gl.o | ||
| 148 | lablGL/ml_glarray.c -> lablGL/ml_glarray.o | ||
| 149 | lablGL/ml_raw.c -> lablGL/ml_raw.o | ||
| 150 | wsi/x11/keysym2ucs.c -> wsi/x11/keysym2ucs.o | ||
| 151 | wsi/x11/xlib.c -> wsi/x11/xlib.o | ||
| 152 | 10 sec | ||
| 153 | Command being timed: "bash build.bash build" | ||
| 154 | User time (seconds): 8.75 | ||
| 155 | System time (seconds): 0.93 | ||
| 156 | Percent of CPU this job got: 101% | ||
| 157 | Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.55 | ||
| 158 | Average shared text size (kbytes): 0 | ||
| 159 | Average unshared data size (kbytes): 0 | ||
| 160 | Average stack size (kbytes): 0 | ||
| 161 | Average total size (kbytes): 0 | ||
| 162 | Maximum resident set size (kbytes): 77656 | ||
| 163 | Average resident set size (kbytes): 0 | ||
| 164 | Major (requiring I/O) page faults: 0 | ||
| 165 | Minor (reclaiming a frame) page faults: 348362 | ||
| 166 | Voluntary context switches: 3401 | ||
| 167 | Involuntary context switches: 706 | ||
| 168 | Swaps: 0 | ||
| 169 | File system inputs: 0 | ||
| 170 | File system outputs: 10272 | ||
| 171 | Socket messages sent: 0 | ||
| 172 | Socket messages received: 0 | ||
| 173 | Signals delivered: 0 | ||
| 174 | Page size (bytes): 4096 | ||
| 175 | Exit status: 0 | ||
| 176 | ~/xsrc/llpp | ||
| 177 | - rm build/*.cm* build/lablGL/*.cm* build/wsi/x11/*.cm* build/*.o build/lablGL/*.o build/wsi/x11/*.o | ||
| 178 | ~/xsrc/llpp | ||
| 179 | - \time -pv make -j4 build/llpp -s | ||
| 180 | Command being timed: "make -j4 build/llpp -s" | ||
| 181 | User time (seconds): 5.11 | ||
| 182 | System time (seconds): 0.64 | ||
| 183 | Percent of CPU this job got: 220% | ||
| 184 | Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.60 | ||
| 185 | Average shared text size (kbytes): 0 | ||
| 186 | Average unshared data size (kbytes): 0 | ||
| 187 | Average stack size (kbytes): 0 | ||
| 188 | Average total size (kbytes): 0 | ||
| 189 | Maximum resident set size (kbytes): 77648 | ||
| 190 | Average resident set size (kbytes): 0 | ||
| 191 | Major (requiring I/O) page faults: 0 | ||
| 192 | Minor (reclaiming a frame) page faults: 166642 | ||
| 193 | Voluntary context switches: 303 | ||
| 194 | Involuntary context switches: 369 | ||
| 195 | Swaps: 0 | ||
| 196 | File system inputs: 0 | ||
| 197 | File system outputs: 89008 | ||
| 198 | Socket messages sent: 0 | ||
| 199 | Socket messages received: 0 | ||
| 200 | Signals delivered: 0 | ||
| 201 | Page size (bytes): 4096 | ||
| 202 | Exit status: 0 | ||
| 203 | ~/xsrc/llpp | ||
| 204 | - \time -pv make -j1 build/llpp -s | ||
| 205 | Command being timed: "make -j1 build/llpp -s" | ||
| 206 | User time (seconds): 3.62 | ||
| 207 | System time (seconds): 0.43 | ||
| 208 | Percent of CPU this job got: 99% | ||
| 209 | Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.07 | ||
| 210 | Average shared text size (kbytes): 0 | ||
| 211 | Average unshared data size (kbytes): 0 | ||
| 212 | Average stack size (kbytes): 0 | ||
| 213 | Average total size (kbytes): 0 | ||
| 214 | Maximum resident set size (kbytes): 77592 | ||
| 215 | Average resident set size (kbytes): 0 | ||
| 216 | Major (requiring I/O) page faults: 0 | ||
| 217 | Minor (reclaiming a frame) page faults: 166577 | ||
| 218 | Voluntary context switches: 176 | ||
| 219 | Involuntary context switches: 138 | ||
| 220 | Swaps: 0 | ||
| 221 | File system inputs: 0 | ||
| 222 | File system outputs: 89008 | ||
| 223 | Socket messages sent: 0 | ||
| 224 | Socket messages received: 0 | ||
| 225 | Signals delivered: 0 | ||
| 226 | Page size (bytes): 4096 | ||
| 227 | Exit status: 0 | ||