Three optimizations on top of what stock -O levels do. Two of them are optimizer switches, the third is always on:
| Optimization | Where it acts | On from | Effect |
|---|---|---|---|
-OoMEMINLINE |
any mode | -O2 |
FillChar(), FillByte(), FillWord(), FillDWord(), FillQWord() and Move() with a constant count of at most 64 bytes expand into direct stores instead of an RTL call |
-OoAUTOINLINE |
{$mode unleashed} units |
-O3 |
small routines are marked for inlining without an inline directive |
| Procvar Devirtualization | {$mode unleashed} units |
every level, -O- included |
a call through a procvar that provably holds the address of one routine is rewritten into a direct call to that routine |
-OoMEMINLINE works in every mode. It needs no modeswitch and no source change, only the optimization level.
The other two act on units compiled in {$mode unleashed} and do nothing anywhere else. In an objfpc, delphi or fpc unit no routine is ever auto-marked and no procvar call is ever rewritten. -O3 does not change that, and neither does -OoAUTOINLINE spelled out on the command line.
The mode that counts is the mode of the unit being compiled. The two optimizations read it at different moments, so they behave differently across unit boundaries:
-OoAUTOINLINEmarks a routine. The decision happens when the unit declaring the routine is compiled, and the mark travels in that unit's.ppulike aninlinedirective would. So a routine auto-marked in an unleashed unit is expanded at call sites in other units too, whatever mode those units are written in. Compile the same unit inobjfpcand there is no mark, so there is nothing to expand anywhere.- Devirtualization rewrites a call site. The decision happens when the unit holding the call is compiled. A call written in an
objfpcunit stays indirect even when the target routine comes from an unleashed unit. A call written in an unleashed unit is rewritten even when the target lives in anobjfpcunit.
-OoAUTOINLINE exists in stock Free Pascal too, but no -O level turns it on there. Its stock heuristic prices any control flow as infinitely complex, so a routine holding a single if never qualified, no matter how the threshold was set. The heuristic here judges the shape of the body instead, which is what makes the switch worth enabling by default.
The two switches are ordinary optimizer switches, so all the usual ways to address them work:
fpc -O2 -OoNOMEMINLINE unit.pas
fpc -O1 -OoMEMINLINE unit.pas
fpc -O3 -Ooautoinline- unit.pas
{$optimization NOAUTOINLINE}
{$optimization autoinline-}
{$optimization autoinline+}A trailing + on the switch name sets it, a trailing - clears it, and the NO prefix clears it too. The spellings are interchangeable. The directive is read per routine: a routine declared while the switch is off compiles without the optimization, a routine declared after turning it back on gets it again.
A block fill or a block copy of a handful of bytes spends most of its time in the RTL routine's size dispatch, not in the stores. With a constant count the size is known at compile time, so the call is replaced by the stores it would end up performing.
type TFrame = record kind: byte; len, crc: dword; end; // 12 bytes
var frame: TFrame;
FillChar(frame, sizeof(frame), 0); // two 8-byte stores, no call| Requirement | Detail |
|---|---|
| the routine | FillChar(), FillByte(), FillWord(), FillDWord(), FillQWord() or Move() from the system unit |
| the count | a constant, at least one element and at most 64 bytes in total: 64 for FillChar() / FillByte() / Move(), 32 words, 16 dwords, 8 qwords. Integer conversions around the constant are seen through |
| the operands | destination (and source, for Move()) must have an address |
| the target | the stores are unaligned, so targets that require naturally aligned accesses (ARM, MIPS, SPARC, RISC-V, m68k and the like) keep the call |
A non-constant count, a count over the cap and a zero count all stay a regular call.
One operand has no address in practice: a constant actual spliced into the body by an inline expansion. Such a constant only gets an address when the RTL call materializes it, so that call is kept too.
The bytes are covered with naturally sized stores. A size that is not a multiple of the widest store gets a final store overlapping the previous one, which beats a tail of narrower stores.
| Total bytes | Stores |
|---|---|
| 1, 2, 4, 8 | one store of that width |
| 3 | 2 bytes, then 1 byte |
| 5, 6, 7 | 4 bytes at offset 0, then 4 bytes ending at the last byte |
| 9 and up | 8-byte stores from the front, plus a final 8-byte store ending at the last byte when the size is not a multiple of 8 |
So 12 bytes are two 8-byte stores overlapping in the middle, and 64 bytes are eight of them with no overlap.
The address of the operand is computed once. A plain variable (global, local or parameter, but not a threadvar) has its address folded into every store. Any other operand gets one pointer temp: a field of a dereferenced pointer, an indexed element, a function result. The temp is assigned once, exactly like the call would have evaluated its argument once.
A constant fill value is replicated across 64 bits at compile time, so every store gets an immediate. A runtime value is evaluated once into a temp and spread over the 64 bits with a single multiply; narrower stores truncate that temp:
FillChar(buf, 16, b); // t := b * $0101010101010101, then two 8-byte stores of tFillWord() / FillDWord() use the matching multiplier, FillQWord() needs none.
The RTL Move() handles overlapping source and destination. The expansion has to as well, so it loads every chunk into a temp before the first store, and the two are then equivalent:
for var i := 0 to 15 do buf[i] := i;
Move(buf[0], buf[4], 12); // 00 01 02 03 00 01 02 03 04 05 06 07 08 09 0A 0BAt -O3 a routine in a {$mode unleashed} unit is inlined even though it never says inline, provided its body is small and simple enough. There is no directive and no source change: a getter, a clamp or a two-line wrapper stops costing a call. Routines in any other mode are never auto-marked, even with the switch given explicitly.
The decision is made once per routine, when its code is generated. It is reported as a hint at the routine's declaration, so the compiler log lists which routines stopped being real calls:
demo.pp(12,1) Hint: Auto inlining: clampToByte(LongInt):System.Byte;
Hints are silenced as usual: all of them with -vh-, just this one with -vm6055.
The body must fit a node budget and consist only of node kinds worth duplicating:
| Rule | Value |
|---|---|
| body size | at most 40 nodes |
| body size when it contains a call | at most 20 nodes, and at most one call |
| allowed shapes | expressions, assignments, if, case, exit, raise, calls, temporaries, type conversions |
| allowed intrinsics | inc(), dec(), succ(), pred(), ord(), chr(), length(), assigned(), abs(), sqr(), sizeof(), typeof(), lo(), hi(), the rol / ror family, include(), exclude(), aligned(), unaligned(), volatile() |
A body holding one call is admitted on purpose. Wrappers, overloads that only supply a default argument, and guard helpers that raise are exactly the routines worth folding away. Their budget is halved because each splice duplicates that call's parameter setup as well; without the tighter cap a compiler self-build grows 23% instead of 4%.
| Reason | Example |
|---|---|
| a loop | for, while, repeat in the body - unbounded work behind a small node count |
| an exception frame | try ... except / try ... finally |
an asm block or a goto |
cannot be priced by a node count |
| an intrinsic that becomes an RTL call | write(), str(), new(), SetLength(), ... |
| more than one call | two calls in one body |
| direct recursion | the routine calls itself |
| forwarding an own by-reference parameter | the body passes one of its own parameters on as var / out / constref or to an untyped parameter |
| the body is too big | over the node budget |
Three more rules on top of the table. The routine must not already be inline or noinline. It must not declare nested routines. And the usual suspects never enter: constructors, destructors, class constructors and destructors, unit initialization and finalization, the program body, and virtual, external, exports, interrupt, iocheck and safecall routines.
This is not Forced Inlining. The forced regime is entered by one thing alone: the inline modifier written by hand on the declaration. No heuristic ever promotes a routine into it.
-OoAUTOINLINE marks the routine the way an inline directive in a stock mode would, and the mark stays a stock-grade hint. Every call site still goes through the inliner's own size budget, and that budget shrinks with expansion depth, so a deeply nested chain of auto-inlined calls stops expanding at some point, silently. Auto inlining never bloats a build the way a forced chain can: small bodies enter, budgeted call sites expand. A hand-written inline in {$mode unleashed} is the opposite: every direct call expands, and a failure is a warning.
Three ways to keep something out:
{$optimization autoinline-}/-OoNOAUTOINLINEstops the automatic marking and leaves explicitinlineroutines alone.noinlinekeeps one routine out.{$inline off}turns off auto-inlining together with every other expansion: a routine whose body is parsed while it is off is never auto-marked, and a call parsed while it is off does not expand anything. See Forced Inlining - Turning it off.
A call through a procedure variable that provably holds the address of one specific routine is rewritten into a direct call. There is no switch and no optimization level to set: the rewrite acts on call sites written in {$mode unleashed} units, at every level from -O- up, and reports a hint at the call site:
demo.pp(21,26) Hint: Devirtualized call: doubler(LongInt):System.LongInt;
This is not inlining. The call stays a call and the target keeps its standalone body. Only the indirection goes away: call rax becomes call doubler.
What happens afterwards is the target's own inline regime, exactly as at any other direct call site: a routine in the forced regime expands, a small routine is picked up by -OoAUTOINLINE at -O3, and a routine that is neither stays a real call. So at -O- a wrapper without an inline directive is devirtualized and nothing more.
Two shapes, chased through at most 4 locations:
- The address itself.
@routine, behind any value-preserving casts, reaching the call. Written in place it looks likeTFn(@doubler)(6). It also appears when the inliner splices a wrapper's body: the wrapper's procvar parameter received@routineat the wrapper's own call site, and after the splice that address stands right at the inner call. - A local with a single store. A local variable or a compiler temporary whose only store in the whole routine is such an address. The store itself is removed when nothing else reads the location; see Debugging.
procedure run;
var
p: TFn;
begin
p := @doubler; // the only store: a routine address
writeln(p(5)); // direct call; at -O3 folded away entirely
end;The wrapper case is where it adds up. Once apply() is inlined, forced or auto, its parameter load becomes the address constant. The inner call then devirtualizes, doubler() inlines in turn, and the chain folds to its result:
function apply(f: TFn; x: longint): longint;
begin
result := f(x);
end;
var g := apply(@doubler, 6); // -O3: g is assigned the constant 12, no calls| Case | Why |
|---|---|
a global or unit-level procvar (program-body vars included) |
any routine, any thread may store to it |
| a procvar parameter, when the wrapper is not inlined | the value is only known per call site |
a local with more than one store, its address taken, or volatile() |
not provably one routine |
the enclosing routine declares nested routines or contains an asm block |
they can write locals invisibly |
a method procvar (of object) or a nested procvar |
carries a self/frame value along with the address |
| a target with a different calling convention | the rewrite requires a call-identical signature |
the call site is written in a mode other than {$mode unleashed} |
the rewrite is mode-gated, see the table at the top |
the call site sits in an {$inline off} region |
see below |
The routine and the call both survive the rewrite, so a breakpoint inside the target hits as it did before and the frame shows up in a stack trace.
The procvar itself is a different matter. Once the call reads the address directly, a local whose only job was to carry that address has a store nobody reads, and that store is removed, so a watch on such a variable shows whatever its stack slot happened to hold. Nothing else picks those stores up later: dead store elimination is not part of any -O level.
{$inline off} turns the rewrite off along with the expansions, which is how to step through a region with everything in place: the call goes back to being indirect and the store is emitted again.
{$inline off}
procedure stepping_here;
begin
var p := @doubler; // stored, and a watch on p shows the address
writeln(p(5)); // stays an indirect call
end;
{$inline on}To keep only the target out of inline expansion while its call sites still devirtualize, mark it noinline.
program optimizations_demo;
{$mode unleashed}
uses
{$ifdef WINDOWS}windows{$else}baseunix, unix{$endif}, sysutils;
type
TBuf = array[16] of byte;
TClampFn = function(v: integer): byte;
// the tick counter has a 15.6 ms resolution on Windows, too coarse for a loop
// that runs in tens of milliseconds
function micros: qword;
{$ifdef WINDOWS}
var freq, cnt: int64;
begin
QueryPerformanceFrequency(freq);
QueryPerformanceCounter(cnt);
result := round(cnt/freq*1000000);
end;
{$else}
var tv: TTimeVal;
begin
fpgettimeofday(@tv, nil);
result := qword(tv.tv_sec)*1000000+qword(tv.tv_usec);
end;
{$endif}
// a body of plain expressions and branches, under the node budget: at -O3
// this is picked for automatic inlining without an `inline` directive
function clampToByte(v: integer): byte;
begin
if v < 0 then result := 0 else if v > 255 then result := 255 else result := byte(v);
end;
function hexOf(const buf: TBuf): string;
begin
result := '';
for var i := 0 to high(buf) do result := result+IntToHex(buf[i], 2);
end;
// a local procvar with a single store is devirtualized into a direct call
// at any level (the compiler hints it); at -O3 the target then inlines too
procedure devirtDemo;
var
clamp: TClampFn;
begin
clamp := @clampToByte;
writeln('devirt ', clamp(300));
end;
var
buf: TBuf;
fillValue: byte;
total: int64 = 0;
started: qword;
begin
// 5 bytes: one 4-byte store plus a second one overlapping it
buf := default(TBuf);
FillChar(buf, 5, $AB);
writeln('fill 5 ', hexOf(buf));
// 7 bytes at a non-zero offset, same overlapping pattern
buf := default(TBuf);
FillChar(buf[3], 7, $CD);
writeln('fill 7 at 3 ', hexOf(buf));
// a runtime fill value is spread over 64 bits with a single multiply
fillValue := clampToByte(300);
FillChar(buf, 16, fillValue);
writeln('fill runtime ', hexOf(buf));
// every chunk is loaded before the first store, so an overlapping Move
// gives what the RTL routine gives
for var i := 0 to high(buf) do buf[i] := i;
Move(buf[0], buf[4], 12);
writeln('move overlap ', hexOf(buf));
devirtDemo;
started := micros;
for var i := 1 to 20000000 do total += clampToByte(i-10000000);
writeln('sum ', total, ' in ', micros-started, ' us');
{$ifdef WINDOWS}readln;{$endif}
end.Output (built with -O3; the timing depends on the machine):
fill 5 ABABABABAB0000000000000000000000
fill 7 at 3 000000CDCDCDCDCDCDCD000000000000
fill runtime FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
move overlap 00010203000102030405060708090A0B
devirt 255
sum 2549967615 in 16064 us
The build log carries the two hints:
optimizations_demo.pp(33,1) Hint: Auto inlining: clampToByte(LongInt):System.Byte;
optimizations_demo.pp(50,38) Hint: Devirtualized call: clampToByte(LongInt):System.Byte;
None of this changes what the program prints. Built with -O1 it produces the identical lines, needs about 34000 us for the loop, still shows the devirtualization hint, and drops the auto-inlining one. What changes is the code behind the lines: -OoNOMEMINLINE puts three FillChar() calls and one Move() call back into the assembly, and -OoNOAUTOINLINE puts back the calls to clampToByte(), one of them inside the counting loop.