Jekyll2023-09-14T03:36:18+00:00https://schellingerhout.github.io/feed.xmlJasper Schellingerhout’s BlogThe personal blog of Jasper Schellingerhout.Jasper SchellingerhoutData Transmission with Delphi Redux2023-09-12T00:00:00+00:002023-09-12T00:00:00+00:00https://schellingerhout.github.io/data%20transmission/datatransmission-redux<p>In this blog post I will revisit my series from four years ago and update it with more current technology</p>
<!--more-->
<p>Revisiting the series that deal with the transfer of raw data at high speed with direct access using pointers to structures that can be supported by most programming platforms.</p>
<h2 id="why-am-i-revisiting-a-four-year-old-post">Why am I revisiting a four year old post?</h2>
<p>I was never happy with the solution to get default values based on the generic type and new functionality in Delphi has allowed me to clean up some of the code I wrote for <a href="/data%20transmission/datatransmission3/">Part 3</a> of my series on data transmission.</p>
<p>I want to share what I found, how it can clean up the code from the prior article and potential for other uses in your own projects.</p>
<h3 id="in-case-you-missed-it">In case you missed it</h3>
<p>I finished my original series of three articles exactly four years ago to the date. If you missed it here are some links to the original posts.</p>
<ul>
<li><a href="/data%20transmission/datatransmission1/">Part 1: Pointers and Structures</a></li>
<li><a href="/data%20transmission/datatransmission2/">Part 2: Arrays and Pointer Arithmetic</a></li>
<li><a href="/data%20transmission/datatransmission3/">Part 3: Transmitting and Interpreting Data</a></li>
</ul>
<p>Here is the gist for those that don’t want to read that much:
If two parties share a pointer and some common understanding of the structure of the memory at that location, then we can have very fast communication. Since arrays are contiguous blocks of memory we can rapidly advance through memory and read structures provided that we know the size of each of the structures, their composition and the number of structures we need to read.</p>
<p>The end result was a library example using closures to configure records for transmission. In my example I created a transmitter <code class="language-plaintext highlighter-rouge">TTxer</code> that can send any of a number of geometry entity records defined using generics.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="c1">//Transmit records one by one
</span> <span class="n">TTXer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">TxLineRec</span><span class="p">>(</span>
<span class="k">procedure</span><span class="p">(</span><span class="k">var</span> <span class="n">R</span><span class="p">:</span> <span class="n">TxLineRec</span><span class="p">)</span>
<span class="k">begin</span>
<span class="n">R</span><span class="p">.</span><span class="n">p1</span><span class="p">.</span><span class="n">x</span> <span class="p">:=</span> <span class="m">0.5</span><span class="p">;</span>
<span class="n">R</span><span class="p">.</span><span class="n">p1</span><span class="p">.</span><span class="n">y</span> <span class="p">:=</span> <span class="m">0.25</span><span class="p">;</span>
<span class="n">R</span><span class="p">.</span><span class="n">p2</span><span class="p">.</span><span class="n">x</span> <span class="p">:=</span> <span class="m">1.0</span><span class="p">;</span>
<span class="n">R</span><span class="p">.</span><span class="n">p2</span><span class="p">.</span><span class="n">y</span> <span class="p">:=</span> <span class="m">2.0</span><span class="p">;</span>
<span class="k">end</span>
<span class="p">);</span>
<span class="c1">// Transmit records as an array (pointer and count)
</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">TxPolyLineRec</span><span class="p">>(</span><span class="n">FPolylines</span><span class="p">.</span><span class="n">Count</span><span class="p">,</span>
<span class="k">Procedure</span><span class="p">(</span><span class="k">var</span> <span class="n">R</span><span class="p">:</span> <span class="n">TxPolyLineRec</span><span class="p">;</span> <span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">)</span>
<span class="k">begin</span>
<span class="n">R</span><span class="p">.</span><span class="n">VertexCount</span> <span class="p">:=</span> <span class="n">Length</span><span class="p">(</span><span class="n">FPolylines</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">Vertices</span><span class="p">);</span>
<span class="n">R</span><span class="p">.</span><span class="n">Vertices</span> <span class="p">:=</span> <span class="n">FPolylines</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">Vertices</span><span class="p">;</span>
<span class="k">end</span>
<span class="p">);</span></code></pre></figure>
<h2 id="getting-defaults-from-generics-types-are-messy">Getting defaults from generics types are messy</h2>
<p>In order to transmit my data I had to initialize each of my records before passing them to the anonymous callback closure. Below is the code to send individual and arrays of records. It is rather elegant:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProc</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">Var</span>
<span class="n">L</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">L</span> <span class="p">:=</span> <span class="n">TxRec</span><span class="p">.</span><span class="n">Default</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="n">AConfigureProc</span><span class="p">(</span><span class="n">L</span><span class="p">);</span>
<span class="n">SendRecord</span><span class="p">(@</span><span class="n">L</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">ANumRecords</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span> <span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProcIter</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">Var</span>
<span class="n">LDynArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">LDefault</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">LDynArray</span><span class="p">,</span> <span class="n">ANumRecords</span><span class="p">);</span>
<span class="n">LDefault</span> <span class="p">:=</span> <span class="n">TxRec</span><span class="p">.</span><span class="n">Default</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="n">ANumRecords</span> <span class="p">-</span> <span class="m">1</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="n">LDynArray</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">:=</span> <span class="n">LDefault</span><span class="p">;</span>
<span class="n">AConfigureProc</span><span class="p">(</span><span class="n">LDynArray</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">i</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="n">SendRecords</span><span class="p">(@</span><span class="n">LDynArray</span><span class="p">[</span><span class="m">0</span><span class="p">],</span> <span class="n">ANumRecords</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>However, the nastiness was factored out and hidden in <code class="language-plaintext highlighter-rouge">TxRec.Default<T></code>, which had to figure out type information for type T and then return a default for it. Since generics are generated for each type the code was duplicated for each type that filled <code class="language-plaintext highlighter-rouge">T</code> for <code class="language-plaintext highlighter-rouge">class function TxRec.Default<T>: T;</code>.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">function</span> <span class="n">TxRec</span><span class="p">.</span><span class="n">Default</span><span class="p"><</span><span class="n">T</span><span class="p">>:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">var</span>
<span class="n">PT</span><span class="p">:</span> <span class="p">^</span><span class="n">T</span><span class="p">;</span> <span class="c1">// this will be a pointer to a const, do not modify values via this pointer
</span><span class="k">begin</span>
<span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxPointRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultPointRec</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxLineRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultLineRec</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxArcRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultArcRec</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxPolyLineRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultPolyLineRec</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxGeometryListRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultGeometryListRec</span>
<span class="k">else</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="nb">nil</span><span class="p">;</span> <span class="c1">// raise exception
</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">PT</span><span class="p">^;</span> <span class="c1">// We Copy value, so the constant is not inadvertently modified
</span><span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>I really dislike this code and tried to find alternatives. I investigated the build in function <code class="language-plaintext highlighter-rouge">Default(T)</code> that is used in the Generic Collections to set or return default values, but I could not find a way to override its behavor for my types. The code for <code class="language-plaintext highlighter-rouge">Default(T)</code> is not in system.pas and may be some compiler magic. I finally resolved to leave it as such.</p>
<h3 id="delphi-104-has-some-new-tricks">Delphi 10.4 has some new tricks</h3>
<p>Unbeknownst to most Delphi developers two new operators were snuck into Delphi 10.4, they are not even in the installed help file, but only mentioned in the online the documentation and in a blog post by Marco Cantu. These two operators are <code class="language-plaintext highlighter-rouge">Initialize</code> and <code class="language-plaintext highlighter-rouge">Finalize</code> and the implications of these are huge (more on this later).</p>
<p><code class="language-plaintext highlighter-rouge">Initialize</code> allows us to define code that runs when a record first enters into scope, plus and most astounding to me, the code for <code class="language-plaintext highlighter-rouge">Initialize</code> is also called per element in an array if its allocated.</p>
<p>To make our record a “managed” record we add these two operators</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TMyRecord</span> <span class="p">=</span> <span class="k">record</span>
<span class="c1">// record data not relevant to illustration
</span>
<span class="k">class</span> <span class="k">operator</span> <span class="n">Initialize</span> <span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TMyRecord</span><span class="p">);</span>
<span class="k">class</span> <span class="k">operator</span> <span class="n">Finalize</span><span class="p">(</span><span class="k">var</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TMyRecord</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h3 id="replacing-txrecdefaultt-with-intialize-operator">Replacing <code class="language-plaintext highlighter-rouge">TxRec.Default<T></code> with <code class="language-plaintext highlighter-rouge">Intialize</code> operator</h3>
<p>In our case we don’t box any types that need to be disposed so we don’t need <code class="language-plaintext highlighter-rouge">Finalize</code>, but we can use <code class="language-plaintext highlighter-rouge">Initialize</code> per each of our records to have them load their size and enumerated type.</p>
<p>For instance, our line record can be changed to look like this:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TxLineRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">// UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="c1">// Line Specific
</span> <span class="n">p1</span><span class="p">,</span> <span class="n">p2</span><span class="p">:</span> <span class="n">PointRec</span><span class="p">;</span>
<span class="k">class</span> <span class="k">operator</span> <span class="n">Initialize</span><span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TxLineRec</span><span class="p">);</span>
<span class="k">End</span><span class="p">;</span>
<span class="p">...</span>
<span class="k">class</span> <span class="k">operator</span> <span class="n">TxLineRec</span><span class="p">.</span><span class="n">Initialize</span><span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TxLineRec</span><span class="p">);</span>
<span class="k">begin</span>
<span class="n">Dest</span><span class="p">.</span><span class="n">Size</span> <span class="p">:=</span> <span class="n">SizeOf</span><span class="p">(</span><span class="n">TxLineRec</span><span class="p">);</span>
<span class="n">Dest</span><span class="p">.</span><span class="n">RecType</span> <span class="p">:=</span> <span class="n">TxRectType_Line</span><span class="p">;</span>
<span class="c1">// p1 and p2 are not initialized
</span><span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>The rest of the records can similarly be adjusted. This means we no longer have one central function that checks the type information to determine the default value to return, but rather each type controls their own initialization.</p>
<p>Our send code is now even cleaner</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProc</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">Var</span>
<span class="n">L</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">AConfigureProc</span><span class="p">(</span><span class="n">L</span><span class="p">);</span>
<span class="n">SendRecord</span><span class="p">(@</span><span class="n">L</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">ANumRecords</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span> <span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProcIter</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">Var</span>
<span class="n">LDynArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">LDefault</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">LDynArray</span><span class="p">,</span> <span class="n">ANumRecords</span><span class="p">);</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="n">ANumRecords</span> <span class="p">-</span> <span class="m">1</span> <span class="k">do</span>
<span class="n">AConfigureProc</span><span class="p">(</span><span class="n">LDynArray</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">i</span><span class="p">);</span>
<span class="n">SendRecords</span><span class="p">(@</span><span class="n">LDynArray</span><span class="p">[</span><span class="m">0</span><span class="p">],</span> <span class="n">ANumRecords</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h2 id="cost-and-benefits-of-managed-records">Cost and Benefits of Managed records</h2>
<p>When we call SetLength and new records are allocated for the array there is one call to <code class="language-plaintext highlighter-rouge">Initialize</code> per element. In my older code it was a faster local memory copy in the loop before configuring the record before transmission.</p>
<p>I don’t know what other overheads are associated with managed records. They seem to be more analogous to C++ classes and structs that exist on the stack instead of dynamically allocated on the free store (like all Delphi objects). Testing would be needed to see if there is any extra processing. In my basic testing I did not notice any differences.</p>
<p>Managed records allow us to have stack managed types that can box types that are dynamically allocated on the free store (colloquially called the heap). Dynamically allocated types can be constructed in <code class="language-plaintext highlighter-rouge">Initialize</code> and disposed in <code class="language-plaintext highlighter-rouge">Finalize</code> without developers needing to call constructors and box code in try-finally blocks. These types are scope managed.</p>
<h2 id="other-improvements-to-the-transmitter-class">Other Improvements to the Transmitter Class</h2>
<p>I realized in rewriting some of the code that signaling that does not require any configuration, or records that can be transmitted once configured based on their own <code class="language-plaintext highlighter-rouge">Initialize</code> code. So there are now three overloads for <code class="language-plaintext highlighter-rouge">Send<T></code>.</p>
<p>I also considered the need for end-users to control records themselves without a configuration callbacks, and to facilicate that, I added two overloads for <code class="language-plaintext highlighter-rouge">SendRecords</code></p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TTxer</span> <span class="p">=</span> <span class="k">class</span>
<span class="k">private</span>
<span class="k">public</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">SendRecord</span><span class="p">(</span><span class="n">ARecord</span><span class="p">:</span> <span class="n">PTxRec</span><span class="p">);</span> <span class="n">static</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">SendRecords</span><span class="p">(</span><span class="n">ARecordArray</span><span class="p">:</span> <span class="n">PTxRec</span><span class="p">;</span> <span class="n">ACount</span><span class="p">:</span> <span class="kt">integer</span><span class="p">);</span> <span class="n">static</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>();</span> <span class="k">overload</span><span class="p">;</span> <span class="c1">// signal type, no configured data
</span> <span class="k">class</span> <span class="k">procedure</span> <span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProc</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span> <span class="k">overload</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">ANumRecords</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span> <span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProcIter</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span> <span class="k">overload</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>The updated repository for this blog series code can be found <a href="https://github.com/schellingerhout/data-transmission-delphi/tree/Delphi10.4">here</a>. You will notice that its a branch named Delphi10.4, the original source code is still under the master branch</p>Jasper SchellingerhoutIn this blog post I will revisit my series from four years ago and update it with more current technologyAn Array Builder for Delphi2023-01-08T00:00:00+00:002023-01-08T00:00:00+00:00https://schellingerhout.github.io/utilities/array-builder-delphi<p>The design of an array builder in Delphi that holds records. Its purpose is to simplify the growth of an array as new elements are added, while improving speed by over-allocation</p>
<!--more-->
<h2 id="motivation">Motivation</h2>
<p>One of the common requirements while working with a dynamic array is the need to resize the array as new elements are added. We often have certain conditions that are checked before an element is added, and the initial size of the array may not be known.</p>
<p>Delphi does have classes that are wrappers around a dynamic array. The simple lists <code class="language-plaintext highlighter-rouge">TTList</code> and <code class="language-plaintext highlighter-rouge">TList<T></code> are examples of these, but these are heap managed types. Dynamic arrays are superior to lists as a result from functions. If we return lists there is always the question of ownership and who is responsible for memory management. Even worse, caller does not even need to receive the return value.</p>
<h2 id="a-simple-example">A Simple Example</h2>
<p>We may have a trivial implementation such as this:</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span> <span class="n">PrimesBelow</span><span class="p">(</span><span class="n">MaxValue</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Integer</span><span class="p">>;</span>
<span class="k">var</span>
<span class="n">Prime</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="k">while</span> <span class="n">Prime</span> <span class="p"><</span> <span class="n">MaxValue</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">Length</span><span class="p">(</span><span class="n">Result</span><span class="p">)</span> <span class="p">+</span> <span class="m">1</span><span class="p">);</span>
<span class="n">Result</span><span class="p">[</span><span class="n">High</span><span class="p">(</span><span class="n">result</span><span class="p">)]</span> <span class="p">:=</span> <span class="n">Prime</span><span class="p">;</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="n">NextPrime</span><span class="p">(</span><span class="n">Prime</span><span class="p">);</span> <span class="c1">// implementation not shown
</span> <span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>Not only is the code a bit ugly, it is quite inefficient to add one element at a time because dynamic array resizing involves copying the entire array.</p>
<h2 id="common-attempts-to-improve-performance">Common attempts to improve performance</h2>
<h3 id="resize-in-blocks">Resize in Blocks</h3>
<p>The first and most common attempt to improve is to allocate in chunks rather than one at time:</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span> <span class="n">PrimesBelow</span><span class="p">(</span><span class="n">MaxValue</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Integer</span><span class="p">>;</span>
<span class="k">var</span>
<span class="n">Prime</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">Count</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">const</span>
<span class="n">BlockSize</span><span class="p">:</span> <span class="m">100</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">Count</span> <span class="p">:=</span> <span class="m">0</span><span class="p">;</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="k">while</span> <span class="n">Prime</span> <span class="p"><</span> <span class="n">MaxValue</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">Count</span> <span class="p">></span> <span class="n">High</span><span class="p">(</span><span class="n">Result</span><span class="p">)</span> <span class="k">then</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">Length</span><span class="p">(</span><span class="n">Result</span><span class="p">)</span> <span class="p">+</span> <span class="n">BlockSize</span><span class="p">);</span>
<span class="n">Result</span><span class="p">[</span><span class="n">Count</span><span class="p">]</span> <span class="p">:=</span> <span class="n">Prime</span><span class="p">;</span>
<span class="k">Inc</span><span class="p">(</span><span class="n">Count</span><span class="p">);</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="n">NextPrime</span><span class="p">(</span><span class="n">Prime</span><span class="p">);</span> <span class="c1">// implementation not shown
</span> <span class="k">end</span><span class="p">;</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">Result</span><span class="p">,</span> <span class="n">Count</span><span class="p">);</span> <span class="c1">// we have to trim overallocation at the end
</span> <span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>We also have to keep track of the actual account and resize the final result as well. We have made the code even uglier, plus we allocate in chunks of a fixed size. We may want to be a bit smarter in our sizing.</p>
<h3 id="smart-initial-allocation">Smart Initial Allocation</h3>
<p>In the case of prime numbers there are <a href="https://en.wikipedia.org/wiki/Prime-counting_function">functions to estimate the number of values</a> that can be used in this case</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span> <span class="n">PrimesBelow</span><span class="p">(</span><span class="n">MaxValue</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Integer</span><span class="p">>;</span>
<span class="k">var</span>
<span class="n">Prime</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">Count</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">const</span>
<span class="n">BlockSize</span><span class="p">:</span> <span class="m">100</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">Count</span> <span class="p">:=</span> <span class="m">0</span><span class="p">;</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">EstimatedNumberOfPrimesBelow</span><span class="p">(</span><span class="n">MaxValue</span><span class="p">));</span> <span class="c1">// implementation not shown
</span>
<span class="k">while</span> <span class="n">Prime</span> <span class="p"><</span> <span class="n">MaxValue</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="c1">// we might not be able to rely on our estimation
</span> <span class="c1">// code and may need to check bounds anyway
</span> <span class="k">if</span> <span class="n">Count</span> <span class="p">></span> <span class="n">High</span><span class="p">(</span><span class="n">Result</span><span class="p">)</span> <span class="k">then</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">Length</span><span class="p">(</span><span class="n">Result</span><span class="p">)</span> <span class="p">+</span> <span class="n">BlockSize</span><span class="p">);</span>
<span class="n">Result</span><span class="p">[</span><span class="n">Count</span><span class="p">]</span> <span class="p">:=</span> <span class="n">Prime</span><span class="p">;</span>
<span class="k">Inc</span><span class="p">(</span><span class="n">Count</span><span class="p">);</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="n">NextPrime</span><span class="p">(</span><span class="n">Prime</span><span class="p">);</span> <span class="c1">// implementation not shown
</span> <span class="k">end</span><span class="p">;</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">Result</span><span class="p">,</span> <span class="n">Count</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>This code may be more performant, because of reducing the number of resizings of the array. However, the cost of the estimation function needs to be taken into account. Unfortunately, the code is still not very clean, in fact, uncertainty may lead us to still keep our bounds check and padding in place. We also probably would never have an exact size pre-calculated and we still need to fix the count at the end.</p>
<h3 id="smart-array-growth">Smart Array Growth</h3>
<p>The other common practice is to grow the dynamic arrays using a function of the current size. In the case of <code class="language-plaintext highlighter-rouge">TList</code> we resize by 1.5 times the previous size (some small values have their own steps). We could define a growth function such as this</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">procedure</span> <span class="n">GrowArray</span><span class="p">(</span><span class="k">var</span> <span class="k">Array</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Double</span><span class="p">>);</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="k">Array</span> <span class="p">=</span> <span class="nb">nil</span> <span class="k">then</span> <span class="c1">// an dynamic array in Delphi that has no elements is nil
</span> <span class="n">SetLength</span><span class="p">(</span><span class="k">Array</span><span class="p">,</span> <span class="m">4</span><span class="p">)</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">Length</span><span class="p">(</span><span class="k">Array</span><span class="p">)</span> <span class="p"><</span> <span class="m">5</span> <span class="k">then</span>
<span class="n">SetLength</span><span class="p">(</span><span class="k">Array</span><span class="p">,</span> <span class="m">8</span><span class="p">)</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">Length</span><span class="p">(</span><span class="k">Array</span><span class="p">)</span> <span class="p"><</span> <span class="m">9</span> <span class="k">then</span>
<span class="n">SetLength</span><span class="p">(</span><span class="k">Array</span><span class="p">,</span> <span class="m">16</span><span class="p">)</span>
<span class="k">else</span>
<span class="n">SetLength</span><span class="p">(</span><span class="k">Array</span><span class="p">,</span> <span class="p">(</span><span class="n">Length</span><span class="p">(</span><span class="k">Array</span><span class="p">)</span> <span class="p">*</span> <span class="m">3</span><span class="p">)</span> <span class="k">div</span> <span class="m">2</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>This function can be called instead of using the <code class="language-plaintext highlighter-rouge">SetLength(result, Length(Result) + BlockSize)</code> call we did earlier. For instance if our prime count estimation was too expensive vs. resizing arrays we could change the code as follows</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span> <span class="n">PrimesBelow</span><span class="p">(</span><span class="n">MaxValue</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Integer</span><span class="p">>;</span>
<span class="k">var</span>
<span class="n">Prime</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">Count</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">Count</span> <span class="p">:=</span> <span class="m">0</span><span class="p">;</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="k">while</span> <span class="n">Prime</span> <span class="p"><</span> <span class="n">MaxValue</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">Count</span> <span class="p">></span> <span class="n">High</span><span class="p">(</span><span class="n">Result</span><span class="p">)</span> <span class="k">then</span>
<span class="n">GrowArray</span><span class="p">(</span><span class="n">result</span><span class="p">);</span>
<span class="n">Result</span><span class="p">[</span><span class="n">Count</span><span class="p">]</span> <span class="p">:=</span> <span class="n">Prime</span><span class="p">;</span>
<span class="k">Inc</span><span class="p">(</span><span class="n">Count</span><span class="p">);</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="n">NextPrime</span><span class="p">(</span><span class="n">Prime</span><span class="p">);</span> <span class="c1">// implementation not shown
</span> <span class="k">end</span><span class="p">;</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">Result</span><span class="p">,</span> <span class="n">Count</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>If we create an implementation we may want to pass the growth function in so that we can grow according to a strategic pattern. I briefly touched on this in <a href="/memory%20management/record-pool-delphi/">my previous post on record pools</a>. We could define a more flexible function such as this</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span> <span class="n">GrowthFunction</span><span class="p">(</span><span class="n">Size</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">size</span> <span class="p">=</span> <span class="m">0</span> <span class="k">then</span>
<span class="k">exit</span><span class="p">(</span><span class="m">4</span><span class="p">);</span>
<span class="k">if</span> <span class="n">size</span> <span class="p"><</span> <span class="m">5</span> <span class="k">then</span>
<span class="k">exit</span><span class="p">(</span><span class="m">8</span><span class="p">);</span>
<span class="k">if</span> <span class="n">size</span> <span class="p"><</span> <span class="m">9</span> <span class="k">then</span>
<span class="k">exit</span><span class="p">(</span><span class="m">16</span><span class="p">);</span>
<span class="n">result</span> <span class="p">:=</span> <span class="p">(</span><span class="n">Size</span> <span class="p">*</span> <span class="m">3</span><span class="p">)</span> <span class="k">div</span> <span class="m">2</span><span class="p">)</span>
<span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>This function can then be passed grow the array as needed. I’ll discuss this a bit later.</p>
<h1 id="tarraybuilder-interface">TArrayBuilder<T> interface</T></h1>
<p>The pattern above works quite well, but we can package all of this actions into a record that does all this for us.</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">unit</span> <span class="n">ArrayBuilder</span><span class="p">;</span>
<span class="k">interface</span>
<span class="k">Type</span>
<span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="k">Record</span>
<span class="k">private</span>
<span class="n">FData</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="n">FCount</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">Grow</span><span class="p">;</span>
<span class="k">public</span>
<span class="k">class</span> <span class="k">function</span> <span class="n">Init</span><span class="p">(</span><span class="n">Size</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">):</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span> <span class="n">static</span><span class="p">;</span>
<span class="c1">// Record Initializers added in Delphi 10.4
</span> <span class="k">class</span> <span class="k">operator</span> <span class="n">Initialize</span><span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">procedure</span> <span class="n">Add</span><span class="p">(</span><span class="k">const</span> <span class="n">Element</span><span class="p">:</span> <span class="n">T</span><span class="p">);</span>
<span class="k">function</span> <span class="n">GetArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">implementation</span>
<span class="k">class</span> <span class="k">operator</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Initialize</span><span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">const</span>
<span class="n">DefaultRec</span><span class="p">:</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="p">();</span> <span class="c1">// count = 0, dynamic array empty
</span><span class="k">begin</span>
<span class="n">Dest</span> <span class="p">:=</span> <span class="n">DefaultRec</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">class</span> <span class="k">function</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Init</span><span class="p">(</span><span class="n">Size</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">):</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">Result</span><span class="p">.</span><span class="n">FData</span><span class="p">,</span> <span class="n">Size</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Add</span><span class="p">(</span><span class="k">const</span> <span class="n">Element</span><span class="p">:</span> <span class="n">T</span><span class="p">);</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">FCount</span> <span class="p">></span> <span class="n">High</span><span class="p">(</span><span class="n">FData</span><span class="p">)</span> <span class="k">then</span>
<span class="n">Grow</span><span class="p">;</span>
<span class="n">FData</span><span class="p">[</span><span class="n">FCount</span><span class="p">]</span> <span class="p">:=</span> <span class="n">Element</span><span class="p">;</span>
<span class="k">Inc</span><span class="p">(</span><span class="n">FCount</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Grow</span><span class="p">;</span>
<span class="k">var</span>
<span class="n">NewSize</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">FData</span> <span class="p">=</span> <span class="nb">nil</span> <span class="k">then</span>
<span class="n">NewSize</span> <span class="p">:=</span> <span class="m">4</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">Length</span><span class="p">(</span><span class="n">FData</span><span class="p">)</span> <span class="p"><</span> <span class="m">5</span> <span class="k">then</span>
<span class="n">NewSize</span> <span class="p">:=</span> <span class="m">8</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">Length</span><span class="p">(</span><span class="n">FData</span><span class="p">)</span> <span class="p"><</span> <span class="m">9</span> <span class="k">then</span>
<span class="n">NewSize</span> <span class="p">:=</span> <span class="m">16</span>
<span class="k">else</span>
<span class="n">NewSize</span> <span class="p">:=</span> <span class="p">(</span><span class="n">Length</span><span class="p">(</span><span class="n">FData</span><span class="p">)</span> <span class="p">*</span> <span class="m">3</span><span class="p">)</span> <span class="k">div</span> <span class="m">2</span><span class="p">;</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">FData</span><span class="p">,</span> <span class="n">NewSize</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">function</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">GetArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">FData</span><span class="p">,</span> <span class="n">FCount</span><span class="p">);</span>
<span class="n">Result</span> <span class="p">:=</span> <span class="n">FData</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">.</span>
</code></pre></div></div>
<p>Our code can now look like this:</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span> <span class="n">PrimesBelow</span><span class="p">(</span><span class="n">MaxValue</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Integer</span><span class="p">>;</span>
<span class="k">var</span>
<span class="n">Prime</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">PrimeBuilder</span><span class="p">:</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="kt">Integer</span><span class="p">>;</span>
<span class="k">begin</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="k">while</span> <span class="n">Prime</span> <span class="p"><</span> <span class="n">MaxValue</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="n">PrimeBuilder</span><span class="p">.</span><span class="n">Add</span><span class="p">(</span><span class="n">Prime</span><span class="p">)</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="n">NextPrime</span><span class="p">(</span><span class="n">Prime</span><span class="p">);</span> <span class="c1">// implementation not shown
</span> <span class="k">end</span><span class="p">;</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">PrimeBuilder</span><span class="p">.</span><span class="n">GetArray</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>If we had a fast estimator for the number of primes we could initialize with that to improve performance, while keeping the code still relatively clean</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">var</span>
<span class="n">Prime</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">PrimeBuilder</span><span class="p">:</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="kt">Integer</span><span class="p">>;</span>
<span class="k">begin</span>
<span class="n">PrimeBuilder</span> <span class="p">:=</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="kt">Integer</span><span class="p">>.</span><span class="n">Init</span><span class="p">(</span>
<span class="n">EstimatedNumberOfPrimesBelow</span><span class="p">(</span><span class="n">MaxValue</span><span class="p">));</span>
<span class="n">Prime</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="c1">//...
</span></code></pre></div></div>
<h2 id="custom-growth-function">Custom Growth Function</h2>
<p>Our Array Builder can now be extended by allowing a growth function that can either be held by the <code class="language-plaintext highlighter-rouge">TArrayBuilder<T></code> or passed into an overload of the <code class="language-plaintext highlighter-rouge">Add(Element: T)</code> method.</p>
<p>Here is an implementation using the function as a member of the <code class="language-plaintext highlighter-rouge">TArrayBuilder<T></code>:</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">unit</span> <span class="n">ArrayBuilder</span><span class="p">;</span>
<span class="k">interface</span>
<span class="k">Type</span>
<span class="n">TGrowthFunction</span> <span class="p">=</span> <span class="n">reference</span> <span class="k">to</span> <span class="k">function</span><span class="p">(</span><span class="n">CurrentSize</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">):</span> <span class="kt">Integer</span><span class="p">;</span>
<span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="k">Record</span>
<span class="k">private</span>
<span class="n">FData</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="n">FCount</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">;</span>
<span class="n">FGrowthFunction</span><span class="p">:</span> <span class="n">TGrowthFunction</span><span class="p">;</span>
<span class="k">public</span>
<span class="k">class</span> <span class="k">function</span> <span class="n">Init</span><span class="p">(</span><span class="n">Size</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">):</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span> <span class="n">static</span><span class="p">;</span>
<span class="c1">// Record Initializers added in Delphi 10.4
</span> <span class="k">class</span> <span class="k">operator</span> <span class="n">Initialize</span><span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">procedure</span> <span class="n">Add</span><span class="p">(</span><span class="k">const</span> <span class="n">Element</span><span class="p">:</span> <span class="n">T</span><span class="p">);</span>
<span class="k">function</span> <span class="n">GetArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="k">procedure</span> <span class="n">SetGrowthFunction</span><span class="p">(</span><span class="k">const</span> <span class="n">F</span><span class="p">:</span> <span class="n">TGrowthFunction</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="c1">// due to the way Generics resolve I have to make the default growth function public
</span> <span class="k">function</span> <span class="n">DefaultArrayBuilderGrowthFunction</span><span class="p">(</span><span class="n">Size</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">implementation</span>
<span class="k">function</span> <span class="n">DefaultArrayBuilderGrowthFunction</span><span class="p">(</span><span class="n">Size</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">size</span> <span class="p">=</span> <span class="m">0</span> <span class="k">then</span>
<span class="k">exit</span><span class="p">(</span><span class="m">4</span><span class="p">);</span>
<span class="k">if</span> <span class="n">size</span> <span class="p"><</span> <span class="m">5</span> <span class="k">then</span>
<span class="k">exit</span><span class="p">(</span><span class="m">8</span><span class="p">);</span>
<span class="k">if</span> <span class="n">size</span> <span class="p"><</span> <span class="m">9</span> <span class="k">then</span>
<span class="k">exit</span><span class="p">(</span><span class="m">16</span><span class="p">);</span>
<span class="n">result</span> <span class="p">:=</span> <span class="p">(</span><span class="n">Size</span> <span class="p">*</span> <span class="m">3</span><span class="p">)</span> <span class="k">div</span> <span class="m">2</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">class</span> <span class="k">operator</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Initialize</span><span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">const</span>
<span class="n">DefaultRec</span><span class="p">:</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="p">();</span> <span class="c1">// count = 0, dynamic array empty
</span><span class="k">begin</span>
<span class="n">Dest</span> <span class="p">:=</span> <span class="n">DefaultRec</span><span class="p">;</span>
<span class="n">Dest</span><span class="p">.</span><span class="n">FGrowthFunction</span> <span class="p">:=</span> <span class="n">DefaultArrayBuilderGrowthFunction</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">SetGrowthFunction</span><span class="p">(</span><span class="k">const</span> <span class="n">F</span><span class="p">:</span> <span class="n">TGrowthFunction</span><span class="p">);</span>
<span class="k">begin</span>
<span class="n">FGrowthFunction</span> <span class="p">:=</span> <span class="n">F</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">class</span> <span class="k">function</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Init</span><span class="p">(</span><span class="n">Size</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">):</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">Result</span><span class="p">.</span><span class="n">FData</span><span class="p">,</span> <span class="n">Size</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Add</span><span class="p">(</span><span class="k">const</span> <span class="n">Element</span><span class="p">:</span> <span class="n">T</span><span class="p">);</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">FCount</span> <span class="p">></span> <span class="n">High</span><span class="p">(</span><span class="n">FData</span><span class="p">)</span> <span class="k">then</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">FData</span><span class="p">,</span> <span class="n">FGrowthFunction</span><span class="p">(</span><span class="n">Length</span><span class="p">(</span><span class="n">FData</span><span class="p">)));</span>
<span class="n">FData</span><span class="p">[</span><span class="n">FCount</span><span class="p">]</span> <span class="p">:=</span> <span class="n">Element</span><span class="p">;</span>
<span class="k">Inc</span><span class="p">(</span><span class="n">FCount</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">function</span> <span class="n">TArrayBuilder</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">GetArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">FData</span><span class="p">,</span> <span class="n">FCount</span><span class="p">);</span>
<span class="n">Result</span> <span class="p">:=</span> <span class="n">FData</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">.</span>
</code></pre></div></div>
<p>We can now provide a custom growth function in this form</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">SomeArrayBuilder</span><span class="p">.</span><span class="n">SetGrowthFunction</span><span class="p">(</span>
<span class="k">function</span> <span class="p">(</span><span class="n">Size</span><span class="p">:</span> <span class="kt">integer</span><span class="p">):</span> <span class="kt">integer</span>
<span class="k">begin</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">Size</span> <span class="p">+</span> <span class="m">100</span><span class="p">;</span>
<span class="k">end</span>
<span class="p">);</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>I hope that you found this useful or at least interesting. Please download, clone or fork the <a href="https://github.com/schellingerhout/array-builder-delphi">source code</a>.</p>
<p>Comments and feedback are always welcomed</p>Jasper SchellingerhoutThe design of an array builder in Delphi that holds records. Its purpose is to simplify the growth of an array as new elements are added, while improving speed by over-allocationA Dynamic Record Pool with Stable Pointers in Delphi2022-12-10T00:00:00+00:002022-12-10T00:00:00+00:00https://schellingerhout.github.io/memory%20management/record-pool-delphi<p>The design of a memory pool in Delphi that holds records. Its purpose is to speed up dynamic memory allocation of a large number of the same record type.</p>
<!--more-->
<p>This pool can grow, while never moving records in memory (this is also called pointer stability).</p>
<h2 id="motivation">Motivation</h2>
<p>One of the common requirements while working in computational geometry is the need to dynamically create or allocate memory for records. However, heap memory operations are generally expensive and time consuming. In contrast, memory management on the stack is extremely efficient because memory is linearly allocated and disposed.</p>
<p>Unfortunately, stack memory is limited, so we can’t escape our usage of the free store. However, we can make our allocations more efficient. For instance, we can allocate blocks of memory instead of individual elements.</p>
<p>Many computational geometry algorithms depend on linked lists or doubly linked lists and therefore require pointer stability, meaning we cannot move records around in memory.</p>
<h2 id="dynamic-arrays-revisited">Dynamic Arrays Revisited</h2>
<p>Before we continue designing our record pool, first just a recap of how dynamic arrays work. For more detail on how dynamic arrays work, please reference my <a href="/data%20transmission/datatransmission2/">prior post on data transmission.</a></p>
<p>Dynamic arrays are smart pointers to the element type declared in their definition. For instance, <code class="language-plaintext highlighter-rouge">TArray<TMyRecord></code> is also a record pointer of type <code class="language-plaintext highlighter-rouge">PMyRecord=^TMyRecord</code>. The difference is that the dynamic array also stores the reference count and the number of elements right before the very first element in the array. This means our memory looks like this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Reference Count, Length, Element0, Element1, …
</code></pre></div></div>
<p>Our dynamic array holds the address of <code class="language-plaintext highlighter-rouge">Element0</code> and can therefore also be used as a simple pointer type.</p>
<p>Every time this dynamic array enters a scope its reference count is increased, and every time it exits a scope the reference count is decreased. Once the reference count hits zero the memory of the array is disposed.</p>
<h2 id="the-problem-with-dynamic-arrays">The Problem with Dynamic Arrays</h2>
<p>If we resize a dynamic array a new larger block is allocated, and the array’s memory is copied. This could be an expensive operation, but more notably the addresses of all the elements would change. This violates our requirement of pointer stability.</p>
<p>If we hold our records in dynamic arrays, we cannot resize them. The most practical way to prevent consumers of our memory pool from direct access is to make the dynamic array a private field of a record or class.</p>
<p>We can also expose a method that will give us the pointer to the next element in sequence. To maintain that we need to keep track with a counter that is incremented as we request the next item. This gives us pointer stability, but we also have the requirement that our pool must be able to grow.</p>
<p>A solution using a dynamic array of our records can either provide growth or pointer stability, but not both.</p>
<h2 id="pointers-revisited">Pointers Revisited</h2>
<p>In order to explain how we solve the dilemma of pointer stability and pool growth we need a brief review of pointers.</p>
<p>When I think of pointers, I consider them as Native Unsigned Integers. These numeric values are memory locations. We already discussed that dynamic arrays are pointer types and that resizing an array simply copies memory over to an array of the new size. This means that addresses of all the elements in the array are modified*. The pointer value of a dynamic array is the address of the first element and therefore also changes.</p>
<h2 id="dynamic-arrays-of-pointers">Dynamic Arrays of Pointers</h2>
<p>With our understanding of pointers as numbers it should be clear that if we resize an array that holds pointers, then the new array will hold the same pointers. Remember, this is essentially the same as resizing an array of Native Unsigned Integers. The memory locations of the pointers change when we resize, but the values they hold remain the same.</p>
<p>Below is an example of a memory block of a Dynamic with pointer value 1000 (64-bit).</p>
<table>
<thead>
<tr>
<th style="text-align: right">Address</th>
<th>Item</th>
<th style="text-align: right">Value</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">988</td>
<td>Reference Count</td>
<td style="text-align: right">1</td>
</tr>
<tr>
<td style="text-align: right">992</td>
<td>Length</td>
<td style="text-align: right">2</td>
</tr>
<tr>
<td style="text-align: right">1000</td>
<td>Pointer0</td>
<td style="text-align: right">100</td>
</tr>
<tr>
<td style="text-align: right">1008</td>
<td>Pointer1</td>
<td style="text-align: right">200</td>
</tr>
</tbody>
</table>
<p>Resizing the array to add an element moves the contiguous block of memory to a new location.</p>
<table>
<thead>
<tr>
<th style="text-align: right">Address</th>
<th>Item</th>
<th style="text-align: right">Value</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right">1988</td>
<td>Reference Count</td>
<td style="text-align: right">1</td>
</tr>
<tr>
<td style="text-align: right">1992</td>
<td>Length</td>
<td style="text-align: right">3</td>
</tr>
<tr>
<td style="text-align: right">2000</td>
<td>Pointer0</td>
<td style="text-align: right">100</td>
</tr>
<tr>
<td style="text-align: right">2008</td>
<td>Pointer1</td>
<td style="text-align: right">200</td>
</tr>
<tr>
<td style="text-align: right">2016</td>
<td>Pointer2</td>
<td style="text-align: right">0</td>
</tr>
</tbody>
</table>
<p>Note that while our entire array and its values are now at different locations, they still hold the same values. This means that our <code class="language-plaintext highlighter-rouge">Pointer0</code> and <code class="language-plaintext highlighter-rouge">Pointer1</code> are left unchanged.</p>
<h2 id="putting-it-all-together">Putting it all together</h2>
<p>Dynamic arrays are pointer types. While we can’t resize the arrays that hold our records, we can safely resize an array that holds the dynamic arrays of records. We already concluded that we can’t resize these dynamic arrays of records, but we can add new ones.</p>
<p>I use the term “row” to refer to the dynamic arrays that hold our records. Our pool can be defined as a generic type <code class="language-plaintext highlighter-rouge">TPool<T></code> and the array that holds the arrays of our type is declared as <code class="language-plaintext highlighter-rouge">TArray<TArray<T>></code></p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="k">Record</span>
<span class="k">private</span>
<span class="n">Rows</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>>;</span>
<span class="p">...</span>
<span class="k">End</span><span class="p">;</span>
</code></pre></div></div>
<p>To access a pointer from the Pool we’ll add a method named <code class="language-plaintext highlighter-rouge">New</code> that returns a pointer. This aids in consumers understanding that it is similar in behavior to the loose function <code class="language-plaintext highlighter-rouge">New</code>.</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="k">Record</span>
<span class="k">private</span>
<span class="k">Index</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span> <span class="c1">// initialized to 0
</span> <span class="n">Rows</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>>;</span>
<span class="k">Procedure</span> <span class="n">AddARow</span><span class="p">;</span>
<span class="k">public</span>
<span class="k">function</span> <span class="n">New</span><span class="p">:</span> <span class="kt">Pointer</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">New</code> can be implemented as follows</p>
<ol>
<li>If we have no rows or the index is beyond the last index of our last row, then
<ol>
<li>add a row and</li>
<li>set the column to zero.</li>
</ol>
</li>
<li>Return the address of the element at index in the last row</li>
<li>Increment the index</li>
</ol>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">function</span> <span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">New</span><span class="p">:</span> <span class="kt">Pointer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="p">(</span><span class="n">Rows</span> <span class="p">=</span> <span class="nb">nil</span><span class="p">)</span> <span class="k">or</span> <span class="p">(</span><span class="k">Index</span> <span class="p">></span> <span class="n">High</span><span class="p">(</span><span class="n">Rows</span><span class="p">[</span><span class="n">High</span><span class="p">(</span><span class="n">Rows</span><span class="p">)]))</span> <span class="k">then</span>
<span class="n">AddARow</span><span class="p">;</span> <span class="c1">// also reset Index to 0
</span>
<span class="n">result</span> <span class="p">:=</span> <span class="p">@</span><span class="n">Rows</span><span class="p">[</span><span class="n">High</span><span class="p">(</span><span class="n">Rows</span><span class="p">)][</span><span class="k">Index</span><span class="p">];</span>
<span class="k">inc</span><span class="p">(</span><span class="k">Index</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>We may need some flexibility in how to size new rows. For instance, we may not want to make every row the same length. We may grow the size of the rows as we allocate more rows as a way to reduce the number of allocations we make. For instance, each row could be made larger by a factor of the prior row, or we could decide the size of the row based on the overall size of all rows already allocated. In some cases we might have a very good estimation of the size and we don’t want to aggressively grow our pool. Since the needs are determined by each use case, we could design our type to accept a method that would dictate the growth of the pool.</p>
<p>In the solution below you can see one example of how we can implement our pool. I allocate rows of a fixed size and I don’t grow new rows at all.</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">procedure</span> <span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">AddARow</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">Rows</span><span class="p">,</span> <span class="n">Length</span><span class="p">(</span><span class="n">Rows</span><span class="p">)</span> <span class="p">+</span> <span class="m">1</span><span class="p">);</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">Rows</span><span class="p">[</span><span class="n">High</span><span class="p">(</span><span class="n">Rows</span><span class="p">)],</span> <span class="n">RowSize</span><span class="p">);</span>
<span class="k">Index</span> <span class="p">:=</span> <span class="m">0</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
</code></pre></div></div>
<p>We can declare a constant for RowSize for the number of entries, but we can get creative and allocate rows that fit a specific size in bytes. For instance, if we want 2KB rows we can declare our constant like this:</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">const</span>
<span class="n">RowSize</span> <span class="p">=</span> <span class="n">Trunc</span><span class="p">(</span><span class="m">2048</span><span class="p">/</span><span class="n">SizeOf</span><span class="p">(</span><span class="n">T</span><span class="p">));</span>
</code></pre></div></div>
<p>Feel free to adjust this part of the code as needed. Consider potentially passing in a method for determining growth. In the complete code below you can see I also added an initializer, this is a feature that only exists in Delphi 10.4 and later. For older versions of Delphi, you can use a constructor that takes arguments, for instance you can set the initial row size or pass a growth function.</p>
<div class="language-pascal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">unit</span> <span class="n">Pool</span><span class="p">;</span>
<span class="k">interface</span>
<span class="k">type</span>
<span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="k">Record</span>
<span class="k">private</span>
<span class="k">const</span>
<span class="c1">// We allocate rows of about 2KB each
</span> <span class="n">RowSize</span> <span class="p">=</span> <span class="n">Trunc</span><span class="p">(</span><span class="m">2048</span><span class="p">/</span><span class="n">SizeOf</span><span class="p">(</span><span class="n">T</span><span class="p">));</span>
<span class="k">var</span>
<span class="n">Rows</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>>;</span>
<span class="k">Index</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">AddARow</span><span class="p">;</span>
<span class="k">public</span>
<span class="k">function</span> <span class="n">New</span><span class="p">:</span> <span class="kt">Pointer</span><span class="p">;</span>
<span class="c1">// Record Initializers added in Delphi 10.4
</span> <span class="k">class</span> <span class="k">operator</span> <span class="n">Initialize</span> <span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">End</span><span class="p">;</span>
<span class="k">implementation</span>
<span class="k">procedure</span> <span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">AddARow</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">Rows</span><span class="p">,</span> <span class="n">Length</span><span class="p">(</span><span class="n">Rows</span><span class="p">)</span> <span class="p">+</span> <span class="m">1</span><span class="p">);</span>
<span class="c1">// instead of a fixed RowSize consider growing rows
</span> <span class="c1">// as implementation demands
</span> <span class="n">SetLength</span><span class="p">(</span><span class="n">Rows</span><span class="p">[</span><span class="n">High</span><span class="p">(</span><span class="n">Rows</span><span class="p">)],</span> <span class="n">RowSize</span><span class="p">);</span>
<span class="k">Index</span> <span class="p">:=</span> <span class="m">0</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">class</span> <span class="k">operator</span> <span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Initialize</span><span class="p">(</span><span class="k">out</span> <span class="n">Dest</span><span class="p">:</span> <span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">const</span>
<span class="n">DefaultRec</span> <span class="p">:</span> <span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="p">();</span>
<span class="k">begin</span>
<span class="n">Dest</span> <span class="p">:=</span> <span class="n">DefaultRec</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">function</span> <span class="n">TPool</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">New</span><span class="p">:</span> <span class="kt">Pointer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="p">(</span><span class="n">Rows</span> <span class="p">=</span> <span class="nb">nil</span><span class="p">)</span> <span class="k">or</span> <span class="p">(</span><span class="k">Index</span> <span class="p">></span> <span class="n">High</span><span class="p">(</span><span class="n">Rows</span><span class="p">[</span><span class="n">High</span><span class="p">(</span><span class="n">Rows</span><span class="p">)]))</span> <span class="k">then</span>
<span class="n">AddARow</span><span class="p">;</span>
<span class="n">result</span> <span class="p">:=</span> <span class="p">@</span><span class="n">Rows</span><span class="p">[</span><span class="n">High</span><span class="p">(</span><span class="n">Rows</span><span class="p">)][</span><span class="k">Index</span><span class="p">];</span>
<span class="k">inc</span><span class="p">(</span><span class="k">Index</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">.</span>
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>I hope that you found this useful or at least interesting. Please download, clone or fork the <a href="https://github.com/schellingerhout/record-pool-delphi">source code</a>.</p>
<p>Comments and feedback are always welcomed</p>Jasper SchellingerhoutThe design of a memory pool in Delphi that holds records. Its purpose is to speed up dynamic memory allocation of a large number of the same record type.A Successful Object Edit Pattern2019-10-17T00:00:00+00:002019-10-17T00:00:00+00:00https://schellingerhout.github.io/design%20patterns/objecteditparttern<p>A successful Object-Oriented Design Pattern for editing objects. This pattern allows free editing of objects, while allowing cancellation and preservation of the original object reference.
<!--more--></p>
<p>I will guide you through my thought processes in developing a code pattern that I have used successfully in multiple implementations.</p>
<p>The goals of the object edit pattern is as follows:</p>
<ul>
<li>Editing an object should allow free manipulation of any of its values</li>
<li>Canceling an edit should leave the original object unchanged</li>
<li>Committing an edit should leave the original object’s pointer or reference unchanged</li>
</ul>
<p>I will present this pattern using Delphi, but the concepts transcend any single programming language.</p>
<h2 id="risk-free-editing-of-object">Risk-free Editing of Object</h2>
<p>This goal proves to be a challenge in that we may manipulate any number of properties at any number of levels. The implementation of an undo system may be complex. The use of a light weight editable object is brittle and requires constant maintenance. The simplest solution to this problem is to edit an identical independent copy of the original object.</p>
<p>To facilitate editing copies, each of my editable objects will need to have a copy constructor (here named <code class="language-plaintext highlighter-rouge">CreateCopy</code>):</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="n">TABC</span> <span class="p">=</span> <span class="k">Class</span><span class="p">(</span><span class="n">TBase</span><span class="p">)</span>
<span class="k">public</span>
<span class="k">Constructor</span> <span class="n">Create</span><span class="p">;</span>
<span class="k">Constructor</span> <span class="n">CreateCopy</span><span class="p">(</span><span class="n">AABC</span><span class="p">:</span> <span class="n">TABC</span><span class="p">);</span>
<span class="k">Destructor</span> <span class="n">Destroy</span><span class="p">;</span> <span class="k">override</span><span class="p">;</span>
<span class="k">function</span> <span class="n">Clone</span><span class="p">:</span> <span class="n">TBase</span><span class="p">;</span> <span class="k">override</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>We can also add a virtual method <code class="language-plaintext highlighter-rouge">Clone</code> at the base class that each class can override. Adding a virtual method that accesses the copy constructor is handy in cases where we don’t need to deal with specific types, such as cases where we need to duplicate all the elements of a list. Its implementation would be:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">function</span> <span class="n">TABC</span><span class="p">.</span><span class="n">Clone</span><span class="p">:</span> <span class="n">TBase</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">TABC</span><span class="p">.</span><span class="n">CreateCopy</span><span class="p">(</span><span class="n">self</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h2 id="canceling-leaves-original-object-unchanged">Canceling leaves original object unchanged</h2>
<p>This goal is already accomplished with our editing of a copied object, provided that we implement our copy constructor according to the following rules:</p>
<ul>
<li>Composed objects are duplicated with a copy constructor.</li>
<li>Weak references to values of reference types are assigned directly. This includes
<ul>
<li>object and pointer types</li>
<li>reference counted types: strings, dynamic arrays, interfaces</li>
</ul>
</li>
</ul>
<p>This new requirement means that all composed objects also need a copy constructor. Composed lists can have their elements duplicated with the use of the <code class="language-plaintext highlighter-rouge">Clone</code> virtual method, instead of checking each class and calling each specific copy constructor by type. The only problem situation is composed interfaced types that are edited along with the object. This odd scenario would require special handling, I consider this an exceptional case and is not addressed by this pattern.</p>
<h2 id="committing-edits-leaves-the-original-object-pointer-unchanged">Committing Edits Leaves the Original Object Pointer Unchanged</h2>
<p>The safety provided by editing the copied object also introduces a problem. We have a new object, with a new reference. All references to the original object will need to be updated. This can be resolved by “assigning” the edited copy to the original object. An assignment method that takes an object and assign its fields, references and composed objects according to similar rules to our copy constructor will serve this purpose. We can even use a properly defined assignment method in our copy constructor to prevent duplication of code. Our copy constructor would then simply look like this.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">procedure</span> <span class="n">TABC</span><span class="p">.</span><span class="n">CreateCopy</span><span class="p">(</span><span class="n">AABC</span><span class="p">:</span> <span class="n">TABC</span><span class="p">);</span>
<span class="k">begin</span>
<span class="n">Create</span><span class="p">;</span> <span class="c1">// NOT inherited create
</span> <span class="n">self</span><span class="p">.</span><span class="n">Assign</span><span class="p">(</span><span class="n">AABC</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Our base class implementation could look like this</p>
<h2 id="assignable-class-implementation">Assignable class implementation</h2>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TBase</span> <span class="p">=</span> <span class="k">Class</span>
<span class="k">procedure</span> <span class="n">Assign</span><span class="p">(</span><span class="n">ASource</span><span class="p">:</span> <span class="n">TBase</span><span class="p">);</span> <span class="k">virtual</span><span class="p">;</span> <span class="k">abstract</span><span class="p">;</span>
<span class="k">function</span> <span class="n">Clone</span><span class="p">:</span> <span class="n">TBase</span><span class="p">;</span> <span class="k">virtual</span><span class="p">;</span> <span class="k">abstract</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>And descendants like this</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TDescendant</span> <span class="p">=</span> <span class="k">Class</span><span class="p">(</span><span class="n">TBase</span><span class="p">)</span>
<span class="k">begin</span>
<span class="k">Constructor</span> <span class="n">Create</span><span class="p">;</span>
<span class="k">Constructor</span> <span class="n">CreateCopy</span><span class="p">(</span><span class="n">ASource</span><span class="p">:</span> <span class="n">TDescendant</span><span class="p">);</span>
<span class="k">Destructor</span> <span class="n">Destroy</span><span class="p">;</span> <span class="k">override</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">Assign</span><span class="p">(</span><span class="n">ASource</span><span class="p">:</span> <span class="n">TBase</span><span class="p">);</span> <span class="n">overide</span><span class="p">;</span>
<span class="k">function</span> <span class="n">Clone</span><span class="p">:</span> <span class="n">TBase</span><span class="p">;</span> <span class="k">virtual</span><span class="p">;</span> <span class="k">abstract</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Thus, our complexity is shifted from our copy constructor and clone methods towards our implementation of an assignment method.</p>
<p>Here is an example of what our <code class="language-plaintext highlighter-rouge">Assign</code> method could look like on a complex object</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">procedure</span> <span class="n">TMyComplexObject</span><span class="p">.</span><span class="n">Assign</span><span class="p">(</span><span class="n">ASource</span><span class="p">:</span> <span class="n">TBase</span><span class="p">);</span>
<span class="k">var</span>
<span class="n">LSource</span><span class="p">:</span> <span class="n">TMyComplexObject</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">inherited</span><span class="p">;</span>
<span class="k">if</span> <span class="n">ASource</span> <span class="k">is</span> <span class="n">TMyComplexObject</span> <span class="k">then</span>
<span class="k">begin</span>
<span class="n">LSource</span> <span class="p">:=</span> <span class="n">TMyComplexObject</span><span class="p">(</span><span class="n">ASource</span><span class="p">);</span>
<span class="n">FField1</span> <span class="p">:=</span> <span class="n">LSource</span><span class="p">.</span><span class="n">FField1</span><span class="p">;</span>
<span class="n">FComposedObject</span><span class="p">.</span><span class="n">Assign</span><span class="p">(</span><span class="n">LSource</span><span class="p">.</span><span class="n">FComposedObject</span><span class="p">);</span>
<span class="n">FComposedList</span><span class="p">.</span><span class="n">Assign</span><span class="p">(</span><span class="n">LSource</span><span class="p">.</span><span class="n">FComposedList</span><span class="p">);</span>
<span class="n">TListHelper</span><span class="p">.</span><span class="n">Assign</span><span class="p">(</span><span class="n">FNonAssignableList</span><span class="p">,</span> <span class="n">LSource</span><span class="p">.</span><span class="n">FNonAssignableList</span><span class="p">);</span>
<span class="n">FWeakReference</span> <span class="p">:=</span> <span class="n">LSource</span><span class="p">.</span><span class="n">FWeakReference</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Besides the rules of copy constructors, here re-appropriated for assignment there are a few best practices:</p>
<ul>
<li>Assign fields instead of properties. We should be concerned with data preservation, properties may have side-effects</li>
<li>Composed objects must allow assignment</li>
<li>Composed lists must allow assignment or you will need to create classes that can facilitate deep copying</li>
</ul>
<p>Delphi provides a standard base class called <code class="language-plaintext highlighter-rouge">TPersistent</code> that provides a public virtual method <code class="language-plaintext highlighter-rouge">Assign</code> and a protected virtual counterpart <code class="language-plaintext highlighter-rouge">AssignTo</code>. We could use that as a base if desired, but there is no clear of convincing argument to use it, except in the case of requiring decendency from this class (for example serialization purposes)</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TBase</span> <span class="p">=</span> <span class="k">Class</span><span class="p">(</span><span class="n">TPersistent</span><span class="p">)</span>
<span class="k">procedure</span> <span class="n">Assign</span><span class="p">(</span><span class="n">ASource</span><span class="p">:</span> <span class="n">TPersistent</span><span class="p">);</span> <span class="n">overide</span><span class="p">;</span>
<span class="k">function</span> <span class="n">Clone</span><span class="p">:</span> <span class="n">TBase</span><span class="p">;</span> <span class="k">virtual</span><span class="p">;</span> <span class="k">abstract</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="p">...</span>
<span class="k">procedure</span> <span class="n">TBase</span><span class="p">.</span><span class="n">Assign</span><span class="p">(</span><span class="n">ASource</span><span class="p">:</span> <span class="n">TPersistent</span><span class="p">);</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="k">not</span> <span class="p">(</span><span class="n">ASource</span> <span class="k">is</span> <span class="n">TBase</span><span class="p">)</span> <span class="k">then</span>
<span class="k">inherited</span><span class="p">;</span> <span class="c1">// will raise an exception
</span><span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Decendent classes would look like this:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TDescendant</span> <span class="p">=</span> <span class="k">Class</span><span class="p">(</span><span class="n">TBase</span><span class="p">)</span>
<span class="k">begin</span>
<span class="k">Constructor</span> <span class="n">Create</span><span class="p">;</span>
<span class="k">Constructor</span> <span class="n">CreateCopy</span><span class="p">(</span><span class="n">ASource</span><span class="p">:</span> <span class="n">TDescendant</span><span class="p">);</span>
<span class="k">Destructor</span> <span class="n">Destroy</span><span class="p">;</span> <span class="k">override</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">Assign</span><span class="p">(</span><span class="n">ASource</span><span class="p">:</span> <span class="n">TPersistent</span><span class="p">);</span> <span class="n">overide</span><span class="p">;</span>
<span class="k">function</span> <span class="n">Clone</span><span class="p">:</span> <span class="n">TBase</span><span class="p">;</span> <span class="k">virtual</span><span class="p">;</span> <span class="k">abstract</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h2 id="our-editor">Our Editor</h2>
<p>Once we have our class with copy constructor and assignment method our implementation of an editor is quite trivial.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">Interface</span>
<span class="k">Type</span>
<span class="n">TABCEditor</span> <span class="p">=</span> <span class="k">class</span>
<span class="k">class</span> <span class="k">function</span> <span class="n">Edit</span><span class="p">(</span><span class="k">const</span> <span class="n">ASubject</span><span class="p">:</span> <span class="n">TABC</span><span class="p">)</span> <span class="p">:</span> <span class="n">TModalResult</span><span class="p">;</span> <span class="n">static</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">implementation</span>
<span class="p">...</span>
<span class="k">class</span> <span class="k">function</span> <span class="n">TABCEditor</span><span class="p">.</span><span class="n">Edit</span><span class="p">(</span><span class="k">const</span> <span class="n">ASubject</span><span class="p">:</span> <span class="n">TABC</span><span class="p">)</span> <span class="p">:</span> <span class="n">TModalResult</span><span class="p">;</span>
<span class="k">var</span>
<span class="n">LABCEditDialog</span><span class="p">:</span> <span class="n">TABCEditDialog</span><span class="p">;</span>
<span class="n">LEdited</span><span class="p">:</span> <span class="n">TABC</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">LABCEditDialog</span> <span class="p">:=</span> <span class="n">TABCEditDialog</span><span class="p">.</span><span class="n">Create</span><span class="p">(</span><span class="nb">nil</span><span class="p">);</span>
<span class="n">LEdited</span> <span class="p">:=</span> <span class="n">TABC</span><span class="p">.</span><span class="n">CreateCopy</span><span class="p">(</span><span class="n">ASubject</span><span class="p">);</span>
<span class="k">try</span>
<span class="n">LABCEditDialog</span><span class="p">.</span><span class="n">ABC</span> <span class="p">:=</span> <span class="n">LEdited</span><span class="p">;</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">LABCEditDialog</span><span class="p">.</span><span class="n">ShowModal</span><span class="p">;</span>
<span class="k">if</span> <span class="n">result</span> <span class="p">=</span> <span class="n">mrOK</span> <span class="k">then</span>
<span class="n">ASubject</span><span class="p">.</span><span class="n">Assign</span><span class="p">(</span><span class="n">LEdited</span><span class="p">);</span>
<span class="k">finally</span>
<span class="n">LEdited</span><span class="p">.</span><span class="n">Free</span><span class="p">;</span>
<span class="n">LABCEditDialog</span><span class="p">.</span><span class="n">Free</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Our edit dialog can manipulate the object at will, knowing that if we hit cancel no edits are preserved, and if we commit our changes, we will assign them to the original object.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope you find the code pattern useful. If you do, please be so kind to link to this article</p>Jasper SchellingerhoutA successful Object-Oriented Design Pattern for editing objects. This pattern allows free editing of objects, while allowing cancellation and preservation of the original object reference.Data Transmission with Delphi (Part 3: Transmitting and Interpreting Data)2019-09-12T00:00:00+00:002019-09-12T00:00:00+00:00https://schellingerhout.github.io/data%20transmission/datatransmission3<p>In this blog post I will continue with arrays in communicating cross-platform via DLLs.</p>
<!--more-->
<p>The purpose of this blog series is to understand how to transfer raw data at high speed with direct access using pointers to structures that can be supported by most programming platforms.</p>
<h2 id="quick-recap">Quick Recap</h2>
<h3 id="records-can-interpret-data-in-different-ways">Records can interpret data in different ways</h3>
<p>As mentioned in prior sections that pointers are simply numbers that correspond to places in memory and that types associated with that address are the way we interpret the data. In many cases data can be interpreted as compatible data types. Here is an example of we could define a customer record at a transmitter:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="cm">{$Z4}</span> <span class="c1">//integer enum
</span> <span class="n">TxRectTypeEnum</span> <span class="p">=</span> <span class="p">(</span><span class="n">TxRectType_AccountHolder</span><span class="p">,</span> <span class="n">TxRectType_CustomerInfo</span><span class="p">);</span>
<span class="cm">{$Z1}</span>
<span class="n">TxCustomerInfoRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span>
<span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="n">UID</span><span class="p">:</span> <span class="n">TGUID</span><span class="p">;</span>
<span class="n">ID</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">Name</span><span class="p">,</span>
<span class="n">Address1</span><span class="p">,</span>
<span class="n">Address2</span> <span class="p">:</span> <span class="k">String</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span></code></pre></figure>
<p>We can interpret the same data received as a pointer in a generic way at the receiver as follows</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="n">RxCustomerInfoRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span>
<span class="n">RecType</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">;</span>
<span class="n">UID</span><span class="p">:</span> <span class="n">TGUID</span><span class="p">;</span>
<span class="n">ID</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">Name</span><span class="p">,</span>
<span class="n">Address1</span><span class="p">,</span>
<span class="n">Address2</span> <span class="p">:</span> <span class="kt">PChar</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">RecType</code> can be interpreted as a <code class="language-plaintext highlighter-rouge">Integer</code> and the <code class="language-plaintext highlighter-rouge">String</code> can be interpreted as <code class="language-plaintext highlighter-rouge">PChar</code>. Similarly to the <code class="language-plaintext highlighter-rouge">String</code> field we could interpret a type of <code class="language-plaintext highlighter-rouge">TArray<Double></code> as <code class="language-plaintext highlighter-rouge">PDouble</code>;</p>
<h3 id="someone-needs-to-manage-the-memory">Someone needs to manage the memory</h3>
<p>In the scenario described above the sender holds the record that has the <code class="language-plaintext highlighter-rouge">String</code> type. This type just like dynamic arrays (like <code class="language-plaintext highlighter-rouge">TArray<Double></code>) are reference counted types. Referencing the underlying data of these types as either <code class="language-plaintext highlighter-rouge">PChar</code> for <code class="language-plaintext highlighter-rouge">String</code>, or <code class="language-plaintext highlighter-rouge">PDouble</code> for <code class="language-plaintext highlighter-rouge">TArray<Double></code> does not increment the reference count. This means that as soon as the dynamic array or string is out of scope the memory is disposed and the pointer is invalid.</p>
<p class="notice"><em>Note:</em> Dynamic arrays and strings must be kept alive via a reference as long as there is a pointer to their underlying data.</p>
<p>There are a few ways to manage this.</p>
<ul>
<li>Receiving party allocates memory for the transmitting party to fill with data. The receiving party disposes the memory after it processed it (and at its convenience).</li>
<li>Transmitting party allocates memory for the receiving party to copy or process completely (no references held). The transmitting party disposes the memory
<ul>
<li>once the send method returns, or</li>
<li>waits to be notified that the receiving party is done with the data, or</li>
<li>the receiving party terminates the session</li>
</ul>
</li>
</ul>
<p>The scenario of allocating memory to be filled is very common in the Windows API, this is often a two-step call. The first call would pass the buffer as <code class="language-plaintext highlighter-rouge">nil</code> and receive the size that you need to allocate.</p>
<p>In the rest of this discussion I will focus on the easier method of synchronous transmission where I assume data is received and copied or completely processed into other types at the receiver before I return. This way the transmitter can dispose as soon as the method call into the receiver returns</p>
<h3 id="abstract-types-can-be-used-to-direct-data">Abstract types can be used to direct data</h3>
<p>In <a href="/data%20transmission/datatransmission1/">Section 1</a> I covered the idea of abstract structured types. The idea was that if we had a record that shared part of its structure with another, we could safely interpret a pointer (memory address) as the smaller abstract of these types (sometimes called a header record). This is the same concept used in Windows messaging where <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_HDR</code> is essentially the header part of other records such as <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_DEVICEINTERFACE</code> or <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_VOLUME</code>. The determination of how the memory could be fully interpreted was contained in <code class="language-plaintext highlighter-rouge">dbcv_devicetype</code>.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="p">...</span>
<span class="k">case</span> <span class="n">LPDeviceBroadcastHeader</span><span class="p">^.</span><span class="n">dbch_devicetype</span> <span class="k">of</span>
<span class="n">DBT_DEVTYP_DEVICEINTERFACE</span><span class="p">:</span>
<span class="k">begin</span>
<span class="n">LPBroadcastDeviceIntf</span> <span class="p">:=</span> <span class="n">PDEV_BROADCAST_DEVICEINTERFACE</span><span class="p">(</span><span class="n">LPDeviceBroadcastHeader</span><span class="p">);</span>
<span class="n">LUsbDeviceName</span> <span class="p">:=</span> <span class="kt">PChar</span><span class="p">(@</span><span class="n">LPBroadcastDeviceIntf</span><span class="p">^.</span><span class="n">dbcc_name</span><span class="p">);</span>
<span class="p">...</span>
<span class="k">end</span><span class="p">;</span>
<span class="n">DBT_DEVTYP_VOLUME</span><span class="p">:</span>
<span class="k">begin</span>
<span class="n">LPBroadcastVolume</span> <span class="p">:=</span> <span class="n">PDEV_BROADCAST_VOLUME</span><span class="p">(</span><span class="n">LPDeviceBroadcastHeader</span><span class="p">);</span>
<span class="c1">// use LPBroadcastVolume.dbcv_unitmask to find volume information
</span> <span class="p">...</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="p">...</span> </code></pre></figure>
<p>Our receiver can direct data in the same way:</p>
<ul>
<li>read part of the data,</li>
<li>use the partial data to determine the type and size of memory to read</li>
<li>read the data as defined by the underlying type</li>
</ul>
<h2 id="building-our-api">Building our API</h2>
<p>I will create a sample data environment containing Lines, Arcs, PolyLines and GeometryLists. Sender and Receiver records will be able to interpret data in compatible ways, with the Sender responsible for data lifetime</p>
<h3 id="the-senders-data">The Sender’s data</h3>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="cm">{$Z4}</span> <span class="c1">//integer enum
</span> <span class="n">TxRectTypeEnum</span> <span class="p">=</span> <span class="p">(</span><span class="n">TxRectType_Point</span><span class="p">,</span> <span class="n">TxRectType_Line</span><span class="p">,</span> <span class="n">TxRectType_Arc</span><span class="p">,</span> <span class="n">TxRectType_Polyline</span><span class="p">,</span> <span class="n">TxRectType_GeometryList</span><span class="p">);</span>
<span class="cm">{$Z1}</span>
<span class="k">const</span>
<span class="n">TxRectType_Undefined</span> <span class="p">=</span> <span class="n">TxRectTypeEnum</span><span class="p">(-</span><span class="m">1</span><span class="p">);</span>
<span class="k">Type</span>
<span class="c1">//The base memory block of all parameter recs, also serves as a signal parameter (no data transmitted)
</span> <span class="n">PTxRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxRec</span><span class="p">;</span>
<span class="n">TxRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span> <span class="c1">// Integer
</span> <span class="k">End</span><span class="p">;</span>
<span class="n">PointRec</span> <span class="p">=</span> <span class="k">Record</span> <span class="c1">//not transmitted directly, composed type
</span> <span class="n">X</span><span class="p">,</span> <span class="n">Y</span> <span class="p">:</span> <span class="kt">double</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PTxPointRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxPointRec</span><span class="p">;</span>
<span class="n">TxPointRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="c1">//Point Specific
</span> <span class="n">p</span><span class="p">:</span> <span class="n">PointRec</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PTxLineRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxLineRec</span><span class="p">;</span>
<span class="n">TxLineRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="c1">//Line Specific
</span> <span class="n">p1</span><span class="p">,</span> <span class="n">p2</span> <span class="p">:</span> <span class="n">PointRec</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PTxArcRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxArcRec</span><span class="p">;</span>
<span class="n">TxArcRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="c1">// Arc Specific
</span> <span class="n">p</span> <span class="p">:</span> <span class="n">PointRec</span><span class="p">;</span>
<span class="n">CCW</span><span class="p">:</span> <span class="kt">Boolean</span><span class="p">;</span> <span class="c1">//in Delphi a Boolean has size of Byte
</span> <span class="n">StartAngle</span><span class="p">,</span> <span class="n">EndAngle</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PTxPolyLineRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxPolyLineRec</span><span class="p">;</span>
<span class="n">TxPolyLineRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="c1">// Polyline Specific
</span> <span class="n">VertexCount</span><span class="p">:</span> <span class="n">Uint32</span><span class="p">;</span>
<span class="n">Vertices</span> <span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">PointRec</span><span class="p">>;</span> <span class="c1">// since x can have a valid value of zero,
</span> <span class="c1">// we need a count, we can't use null termination
</span> <span class="k">End</span><span class="p">;</span>
<span class="n">PTxGeometryListRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxGeometryListRec</span><span class="p">;</span>
<span class="n">TxGeometryListRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="c1">// PolyLineArcRec Specific
</span> <span class="n">Geometry</span> <span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">PTxRec</span><span class="p">>;</span> <span class="c1">// since we have an array of pointers,
</span> <span class="c1">// this array can be null terminated
</span> <span class="k">End</span><span class="p">;</span></code></pre></figure>
<h3 id="the-receivers-data">The Receiver’s data</h3>
<p>Dynamic arrrays are mapped to pointer types.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="cm">{$Z4}</span> <span class="c1">//integer enum
</span> <span class="n">RxRectTypeEnum</span> <span class="p">=</span> <span class="p">(</span><span class="n">RxRectType_Point</span><span class="p">,</span> <span class="n">RxRectType_Line</span><span class="p">,</span> <span class="n">RxRectType_Arc</span><span class="p">,</span> <span class="n">RxRectType_Polyline</span><span class="p">,</span>
<span class="n">RxRectType_GeometryList</span><span class="p">);</span>
<span class="cm">{$Z1}</span>
<span class="k">const</span>
<span class="n">RxRectType_Undefined</span> <span class="p">=</span> <span class="n">RxRectTypeEnum</span><span class="p">(-</span><span class="m">1</span><span class="p">);</span>
<span class="k">Type</span>
<span class="c1">//The base memory block of all parameter recs,
// also serves as a signal parameter (no data transmitted)
</span>
<span class="n">PPRxRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">PRxRec</span><span class="p">;</span> <span class="c1">//array of pointers to RxRec
</span> <span class="n">PRxRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">RxRec</span><span class="p">;</span>
<span class="n">RxRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">RxRectTypeEnum</span><span class="p">;</span> <span class="c1">// Integer
</span> <span class="k">End</span><span class="p">;</span>
<span class="n">PPointRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">PointRec</span><span class="p">;</span>
<span class="n">PointRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">X</span><span class="p">,</span> <span class="n">Y</span> <span class="p">:</span> <span class="kt">double</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PRxPointRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">RxPointRec</span><span class="p">;</span>
<span class="n">RxPointRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">RxRectTypeEnum</span><span class="p">;</span>
<span class="c1">//Point Specific
</span> <span class="n">p</span><span class="p">:</span> <span class="n">PointRec</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PRxLineRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">RxLineRec</span><span class="p">;</span>
<span class="n">RxLineRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">RxRectTypeEnum</span><span class="p">;</span>
<span class="c1">//Line Specific
</span> <span class="n">p1</span><span class="p">,</span> <span class="n">p2</span> <span class="p">:</span> <span class="n">PointRec</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PRxArcRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">RxArcRec</span><span class="p">;</span>
<span class="n">RxArcRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">RxRectTypeEnum</span><span class="p">;</span>
<span class="c1">// Arc Specific
</span> <span class="n">p</span> <span class="p">:</span> <span class="n">PointRec</span><span class="p">;</span>
<span class="n">CCW</span><span class="p">:</span> <span class="kt">Boolean</span><span class="p">;</span> <span class="c1">//in Delphi a Boolean has size of Byte
</span> <span class="n">StartAngle</span><span class="p">,</span> <span class="n">EndAngle</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PRxPolyLineRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">RxPolyLineRec</span><span class="p">;</span>
<span class="n">RxPolyLineRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">RxRectTypeEnum</span><span class="p">;</span>
<span class="c1">// Polyline Specific
</span> <span class="n">VertexCount</span><span class="p">:</span> <span class="n">Uint32</span><span class="p">;</span>
<span class="n">Vertices</span> <span class="p">:</span> <span class="n">PPointRec</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PRxGeometryListRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">RxGeometryListRec</span><span class="p">;</span>
<span class="n">RxGeometryListRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// Common header
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span> <span class="c1">//UInt32
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">RxRectTypeEnum</span><span class="p">;</span>
<span class="c1">// GeometryListRec Specific
</span> <span class="n">Geometry</span> <span class="p">:</span> <span class="n">PPRxRec</span><span class="p">;</span> <span class="c1">// null terminated
</span> <span class="k">End</span><span class="p">;</span></code></pre></figure>
<h3 id="receivers-dll-export">Receiver’s DLL Export</h3>
<p>The Receiver can export a method of this format:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">procedure</span> <span class="n">SendTxRecord</span><span class="p">(</span><span class="n">APRxRec</span> <span class="p">:</span> <span class="n">PRxRec</span><span class="p">);</span> <span class="k">stdcall</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">case</span> <span class="n">APRXRec</span><span class="p">.</span><span class="n">RecType</span> <span class="k">of</span>
<span class="n">RxRectType_Point</span> <span class="p">:</span>
<span class="n">ReceivePoint</span><span class="p">(</span><span class="n">PRxPointRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">));</span>
<span class="n">RxRectType_Line</span> <span class="p">:</span>
<span class="n">ReceiveLine</span><span class="p">(</span><span class="n">PRxLineRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">));</span>
<span class="n">RxRectType_Arc</span> <span class="p">:</span>
<span class="n">ReceiveArc</span><span class="p">(</span><span class="n">PRxArcRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">));</span>
<span class="n">RxRectType_Polyline</span> <span class="p">:</span>
<span class="n">ReceivePolline</span><span class="p">(</span><span class="n">PRxPolyLineRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">));</span>
<span class="n">RxRectType_GeometryList</span> <span class="p">:</span>
<span class="n">ReceivePolline</span><span class="p">(</span><span class="n">RxGeometryListRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">));</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">exports</span> <span class="n">SendTxRecord</span><span class="p">;</span></code></pre></figure>
<p>It seems strange that the sender would have a method called “Send”, but the receiver will not use it internally, instead a the Sender can link to it as</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"> <span class="k">procedure</span> <span class="n">SendTxRecord</span><span class="p">(</span><span class="n">APTxRec</span> <span class="p">:</span> <span class="n">PTxRec</span><span class="p">);</span> <span class="k">stdcall</span><span class="p">;</span> <span class="k">external</span> <span class="s">'MyDLL.dll'</span></code></pre></figure>
<p>We can also add a similar method to process arrays of data that can be traversed with pointermath as discussed in Section 2.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">procedure</span> <span class="n">SendTxRecords</span><span class="p">(</span><span class="n">APRxRec</span> <span class="p">:</span> <span class="n">PRxRec</span><span class="p">;</span> <span class="n">ACount</span><span class="p">:</span> <span class="kt">integer</span><span class="p">);</span> <span class="k">stdcall</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">case</span> <span class="n">APRXRec</span><span class="p">.</span><span class="n">RecType</span> <span class="k">of</span>
<span class="n">RxRectType_Point</span> <span class="p">:</span>
<span class="n">ReceivePoints</span><span class="p">(</span><span class="n">PRxPointRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">),</span> <span class="n">ACount</span><span class="p">);</span>
<span class="n">RxRectType_Line</span> <span class="p">:</span>
<span class="n">ReceiveLines</span><span class="p">(</span><span class="n">PRxLineRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">),</span> <span class="n">ACount</span><span class="p">);</span>
<span class="n">RxRectType_Arc</span> <span class="p">:</span>
<span class="n">ReceiveArcs</span><span class="p">(</span><span class="n">PRxArcRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">),</span> <span class="n">ACount</span><span class="p">);</span>
<span class="n">RxRectType_Polyline</span> <span class="p">:</span>
<span class="n">ReceivePollines</span><span class="p">(</span><span class="n">PRxPolyLineRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">),</span> <span class="n">ACount</span><span class="p">);</span>
<span class="n">RxRectType_GeometryList</span> <span class="p">:</span>
<span class="n">ReceivePollines</span><span class="p">(</span><span class="n">RxGeometryListRec</span><span class="p">(</span><span class="n">APRxRec</span><span class="p">),</span> <span class="n">ACount</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span> </code></pre></figure>
<h3 id="preparing-data-for-transmission">Preparing Data for Transmission</h3>
<p>Each of our datatypes will be ready to receive the relevant information to transmit, but the values that are needed for processing on the receiver side such as <code class="language-plaintext highlighter-rouge">size</code> and <code class="language-plaintext highlighter-rouge">rectype</code> will need to be populated for every record before we can fill it with the data that we want to transmit.</p>
<p>Delphi does not have a way to auto-initialize records. Only the smart pointer types: Strings and Interfaces are automatically initialized as null. <a href="https://stackoverflow.com/questions/39392920/how-can-delphi-records-be-initialized-automatically">You could exploit the use of properties along with a string or interface field on the record to initialize it on demand</a>, but that will not help us in the transmission of this record. We are passing an abstract type and there are no virtual calls on record structures. Even if these did exist, support would vary by programming language. Also, we really want our records to be a data map of memory and nothing more.</p>
<p>Delphi provides for a simple syntax to declare record constants, for instance:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">const</span>
<span class="n">DefaultTxArcRec</span><span class="p">:</span> <span class="n">TTxArcRec</span> <span class="p">=</span> <span class="p">();</span></code></pre></figure>
<p>This constant as defined would be a default record with all memory zeroed out. This means integer and floating numeric values are zero and booleans are false. Zeroed out memory also means pointers are nil, which in turn means dynamic arrays are empty and strings are empty strings.</p>
<p>We can do even better with our constant, when we declare fields and values using <code class="language-plaintext highlighter-rouge">field: value</code> syntax then only the declared fields receive values other than 0 all other values are still blank. Here is the improved constant:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">const</span>
<span class="n">DefaultTxArcRec</span><span class="p">:</span> <span class="n">TTxArcRec</span> <span class="p">=</span>
<span class="p">(</span>
<span class="n">Size</span><span class="p">:</span> <span class="n">SizeOf</span><span class="p">(</span><span class="n">TTxArcRec</span><span class="p">);</span>
<span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectType_Arc</span><span class="p">;</span>
<span class="p">);</span></code></pre></figure>
<p>Constant record declarations can also be nested with the same considerations.</p>
<p>So transmission process would be :</p>
<ul>
<li>Obtain a record filled with appropriate <code class="language-plaintext highlighter-rouge">size</code> and <code class="language-plaintext highlighter-rouge">rectype</code> values</li>
<li>Populate the data to transmit</li>
<li>Transmit the record</li>
<li>If the record was dynamically allocated then dispose it after transmission</li>
</ul>
<h3 id="generalizing-transmission">Generalizing transmission</h3>
<p>The process of transmission as described above follows a predictable path and may lend itself well to using generics and anonymous methods. The problem is that pointers to records do not give us virtual type info, but we could use the typeinfo of a specific record type to generalize our data.</p>
<p>Here is the basic idea: When sending a record call a generic method that gets specialized by the record type <code class="language-plaintext highlighter-rouge">Send<TxLineRec>(...)</code>, we could also pass an anonymous callback method so we can be populate the data in the record (beside the header). This anonymous method will give the caller the record to manipulate via a var parameter. If we transmit a list we’ll have to pass a count and receive an anonymous callback that presents both the record and its index in the list.</p>
<p>To facilitate the generic calls, I’ll wrap these <code class="language-plaintext highlighter-rouge">Send</code> calls in a class</p>
<h2 id="the-transmitter-class">The Transmitter Class</h2>
<h3 id="interface">Interface</h3>
<p>Here is an example of what my transmitter class could look like:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">Type</span>
<span class="n">TSendConfigProc</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="n">reference</span> <span class="k">to</span> <span class="k">procedure</span><span class="p">(</span><span class="k">var</span> <span class="n">A</span><span class="p">:</span> <span class="n">T</span><span class="p">);</span>
<span class="n">TSendConfigProcIter</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="n">reference</span> <span class="k">to</span> <span class="k">procedure</span><span class="p">(</span><span class="k">var</span> <span class="n">A</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span> <span class="n">AIndex</span><span class="p">:</span> <span class="kt">integer</span><span class="p">);</span>
<span class="n">TTxer</span> <span class="p">=</span> <span class="k">class</span>
<span class="k">public</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">SendRecord</span><span class="p">(</span><span class="n">ARecord</span><span class="p">:</span> <span class="n">PTxRec</span><span class="p">);</span> <span class="n">static</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">SendRecords</span><span class="p">(</span><span class="n">ARecordArray</span><span class="p">:</span> <span class="n">PTxRec</span><span class="p">;</span> <span class="n">ACount</span><span class="p">:</span> <span class="kt">integer</span><span class="p">);</span> <span class="n">static</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProc</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span> <span class="k">overload</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">ANumRecords</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span> <span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProcIter</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span> <span class="k">overload</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>For our transmitter class we’ll add the ability to send records and an array of records with a <code class="language-plaintext highlighter-rouge">Send</code> command that is generic and will take either an anonymous method to configure a record, or a count and an anonymous method that will configure a record, plus provide the index of the active item.</p>
<p>The sample transmitter class also has <code class="language-plaintext highlighter-rouge">SendRecord</code> and <code class="language-plaintext highlighter-rouge">SendRecords</code> that are direct one-to-one wrappers of the DLL signatures. I’ll explain the reason for these a bit later.</p>
<h3 id="generic-methods">Generic Methods</h3>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProc</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">Var</span>
<span class="n">L</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">L</span> <span class="p">:=</span> <span class="n">TxRec</span><span class="p">.</span><span class="n">Default</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="n">AConfigureProc</span><span class="p">(</span><span class="n">L</span><span class="p">);</span>
<span class="n">SendRecord</span><span class="p">(@</span><span class="n">L</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">ANumRecords</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span> <span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProcIter</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">Var</span>
<span class="n">LDynArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">LDefault</span> <span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">LDynArray</span><span class="p">,</span> <span class="n">ANumRecords</span><span class="p">);</span>
<span class="n">LDefault</span> <span class="p">:=</span> <span class="n">TxRec</span><span class="p">.</span><span class="n">Default</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="n">ANumRecords</span><span class="p">-</span><span class="m">1</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="n">LDynArray</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">:=</span> <span class="n">LDefault</span><span class="p">;</span>
<span class="n">AConfigureProc</span><span class="p">(</span><span class="n">LDynArray</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">i</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="n">SendRecords</span><span class="p">(@</span><span class="n">LDynArray</span><span class="p">[</span><span class="m">0</span><span class="p">],</span> <span class="n">ANumRecords</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>These calls facilitate the steps in our process as follows</p>
<ul>
<li>We obtain a record filled with appropriate <code class="language-plaintext highlighter-rouge">size</code> and <code class="language-plaintext highlighter-rouge">rectype</code> values by our <code class="language-plaintext highlighter-rouge">TxRec.Default<T></code> method</li>
<li>The API Consumer populate the data to transmit via the anonymous methods of types <code class="language-plaintext highlighter-rouge">TSendConfigProc<T></code> and <code class="language-plaintext highlighter-rouge">TSendConfigProcIter<T></code></li>
<li>The record is transmitted via <code class="language-plaintext highlighter-rouge">SendRecord</code> and <code class="language-plaintext highlighter-rouge">SendRecords</code></li>
<li>The records are disposed when the <code class="language-plaintext highlighter-rouge">L: T</code> variable and the reference counted <code class="language-plaintext highlighter-rouge">TArray<T></code> dynamic array going out of scope</li>
</ul>
<p>We will examine each of these in more detail</p>
<h3 id="initializing-records">Initializing Records</h3>
<p>Delphi constants allow for a loophole to modify values that should not change, all you have to do is obtain a pointer to the constant and you can manipulate the values. This makes using constants with pointer types particularly dangerous.</p>
<p>In this code you will see that I keep the scope of referencing the constant via pointer as small as possible.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">function</span> <span class="n">TxRec</span><span class="p">.</span><span class="n">Default</span><span class="p"><</span><span class="n">T</span><span class="p">>:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">var</span>
<span class="n">PT</span><span class="p">:</span> <span class="p">^</span><span class="n">T</span><span class="p">;</span> <span class="c1">// this will be a pointer to a const, do not modify values via this pointer
</span><span class="k">begin</span>
<span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxPointRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultPointRec</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxLineRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultLineRec</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxArcRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultArcRec</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxPolyLineRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultPolyLineRec</span>
<span class="k">else</span> <span class="k">if</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">T</span><span class="p">)</span> <span class="p">=</span> <span class="n">TypeInfo</span><span class="p">(</span><span class="n">TxGeometryListRec</span><span class="p">)</span> <span class="k">then</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="p">@</span><span class="n">DefaultGeometryListRec</span>
<span class="k">else</span>
<span class="n">PT</span> <span class="p">:=</span> <span class="nb">nil</span><span class="p">;</span> <span class="c1">// raise exception
</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">PT</span><span class="p">^;</span> <span class="c1">//We Copy value, so the constant is not inadvertently modified
</span><span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>You may recall that I added methods <code class="language-plaintext highlighter-rouge">SendRecord</code> and <code class="language-plaintext highlighter-rouge">SendRecords</code> to my transmitter class, the reason I did so was that a generic method cannot use the imported DLL method directly unless it is in the interface section of the unit. Similarly here: The defaultrecords will need to be declared in the interface section with the records. Unfortunately, even though records support constant declaration within the record name scope, we cannot declare a self constant such as we can with classes. So we’ll have to declare these record constants separate from our records in the interface section of the unit.</p>
<h3 id="populating-the-record">Populating the Record</h3>
<p>The anonymous method passed by the consumer of the API serves as a way to populate the record</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"> <span class="n">TTXer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">TxLineRec</span><span class="p">>(</span>
<span class="k">procedure</span><span class="p">(</span><span class="k">var</span> <span class="n">ARec</span><span class="p">:</span> <span class="n">TxLineRec</span><span class="p">)</span>
<span class="k">begin</span>
<span class="n">ARec</span><span class="p">.</span><span class="n">p1</span><span class="p">.</span><span class="n">x</span> <span class="p">:=</span> <span class="m">0.5</span><span class="p">;</span>
<span class="n">ARec</span><span class="p">.</span><span class="n">p1</span><span class="p">.</span><span class="n">y</span> <span class="p">:=</span> <span class="m">0.25</span><span class="p">;</span>
<span class="n">ARec</span><span class="p">.</span><span class="n">p2</span><span class="p">.</span><span class="n">x</span> <span class="p">:=</span> <span class="m">1.0</span><span class="p">;</span>
<span class="n">ARec</span><span class="p">.</span><span class="n">p2</span><span class="p">.</span><span class="n">y</span> <span class="p">:=</span> <span class="m">2.0</span><span class="p">;</span>
<span class="k">end</span>
<span class="p">);</span>
<span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">TxPolyLineRec</span><span class="p">>(</span><span class="n">FPolylines</span><span class="p">.</span><span class="n">Count</span><span class="p">,</span>
<span class="k">Procedure</span><span class="p">(</span><span class="k">var</span> <span class="n">ARec</span><span class="p">:</span> <span class="n">TxPolyLineRec</span><span class="p">;</span> <span class="n">AIdx</span><span class="p">:</span> <span class="kt">integer</span><span class="p">)</span>
<span class="k">begin</span>
<span class="n">ARec</span><span class="p">.</span><span class="n">VertexCount</span> <span class="p">:=</span> <span class="n">Length</span><span class="p">(</span><span class="n">FPolylines</span><span class="p">[</span><span class="n">AIdx</span><span class="p">].</span><span class="n">Vertices</span><span class="p">);</span>
<span class="n">ARec</span><span class="p">.</span><span class="n">Vertices</span> <span class="p">:=</span> <span class="n">FPolylines</span><span class="p">[</span><span class="n">AIdx</span><span class="p">].</span><span class="n">Vertices</span><span class="p">;</span>
<span class="k">end</span>
<span class="p">);</span></code></pre></figure>
<p>Smart reference types such as strings, interfaces and dynamic arrays that are cast as dumb pointer types must be kept alive for the duration of transmission. In the case above assume <code class="language-plaintext highlighter-rouge">Vertices</code> on our polline objects in our list is a <code class="language-plaintext highlighter-rouge">TArray<PointRec></code> and we have one on our record so the reference will be kept alive. If we had a field on our record of type <code class="language-plaintext highlighter-rouge">PPointRec</code> then we should ensure that the list element’s array does not get modified.</p>
<h3 id="transmitting-the-record">Transmitting the Record</h3>
<p>The pointer to the record or array of records are handed off to the dll which should process the data synchronously (at least with the method that creates, transmits and disposes the data). We could add our DLL imports to the interface section and call them directly, or keep them in the implementation section and call them via the class. The class methods will just pass the parameters along</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">SendRecord</span><span class="p">(</span><span class="n">ARecord</span><span class="p">:</span> <span class="n">PTxRec</span><span class="p">);</span>
<span class="k">begin</span>
<span class="n">SendTxRecord</span><span class="p">(</span><span class="n">ARecord</span><span class="p">);</span> <span class="c1">//dll call
</span><span class="k">end</span><span class="p">;</span>
<span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">SendRecords</span><span class="p">(</span><span class="n">ARecordArray</span><span class="p">:</span> <span class="n">PTxRec</span><span class="p">;</span> <span class="n">ACount</span><span class="p">:</span> <span class="kt">integer</span><span class="p">);</span>
<span class="k">begin</span>
<span class="n">SendTxRecords</span><span class="p">(</span><span class="n">ARecordArray</span><span class="p">,</span> <span class="n">ACount</span><span class="p">);</span> <span class="c1">//dll call
</span><span class="k">end</span><span class="p">;</span></code></pre></figure>
<h3 id="disposal-of-the-record">Disposal of the Record</h3>
<p>Disposal is done via reference counting of the dynamic array in the case of an array transmission, and in the case of a single record it will be disposed once it exists the scope of the <code class="language-plaintext highlighter-rouge">Send</code> method. We could also allocate a record via <code class="language-plaintext highlighter-rouge">New</code>. In that case we’d do something like this</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">procedure</span> <span class="n">TTxer</span><span class="p">.</span><span class="n">Send</span><span class="p"><</span><span class="n">T</span><span class="p">>(</span><span class="n">AConfigureProc</span><span class="p">:</span> <span class="n">TSendConfigProc</span><span class="p"><</span><span class="n">T</span><span class="p">>);</span>
<span class="k">Var</span>
<span class="n">LPT</span><span class="p">:</span> <span class="p">^</span><span class="n">T</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">New</span><span class="p">(</span><span class="n">LPT</span><span class="p">);</span>
<span class="k">try</span>
<span class="n">LPT</span><span class="p">^</span> <span class="p">:=</span> <span class="n">TxRec</span><span class="p">.</span><span class="n">Default</span><span class="p"><</span><span class="n">T</span><span class="p">>;</span>
<span class="n">AConfigureProc</span><span class="p">(</span><span class="n">LPT</span><span class="p">^);</span>
<span class="n">SendRecord</span><span class="p">(</span><span class="n">PTxRec</span><span class="p">(</span><span class="n">LPT</span><span class="p">));</span>
<span class="k">finally</span>
<span class="n">Dispose</span><span class="p">(</span><span class="n">LPT</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>I personal prefer the stack cleaning up the variable, but there may be cases where this method may be justified.</p>
<h2 id="conclusion">Conclusion</h2>
<p>The repository for this blog series code can be found <a href="https://github.com/schellingerhout/data-transmission-delphi">here</a>.</p>
<p>That concludes this series on data transmission with records and arrays. Hopefully you’ll be able to extend these concepts to other programming languages. C\C++ should be easy candidates for handling data transmission in this way, for .Net you’d probably have to write a wrapper class in C++\CLI because it may be tricky to write the proper PInvoke headers to process data.</p>Jasper SchellingerhoutIn this blog post I will continue with arrays in communicating cross-platform via DLLs.Data Transmission with Delphi (Part 2: Arrays and Pointer Math)2019-02-14T00:00:00+00:002019-02-14T00:00:00+00:00https://schellingerhout.github.io/data%20transmission/datatransmission2<p>In this blog post I will continue with arrays in communicating cross-platform via DLLs.</p>
<!--more-->
<p>The purpose of this blog series is to understand how to transfer raw data at high speed with direct access using pointers to structures that can be supported by most programming platforms.</p>
<h2 id="arrays">Arrays</h2>
<p>Arrays are contiguous blocks of items of the same size. In Delphi arrays can be declared explicitly in a few ways</p>
<h3 id="fixed-length-arrays">Fixed Length Arrays</h3>
<p>If we know the size of the array of data we need to declare, or if the size never varies, then the array can be declared as follows:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">var</span>
<span class="n">LStaticArray</span> <span class="p">:</span> <span class="k">Array</span><span class="p">[</span><span class="m">0..9</span><span class="p">]</span> <span class="k">of</span> <span class="kt">Double</span><span class="p">;</span></code></pre></figure>
<p>The notation within <code class="language-plaintext highlighter-rouge">[]</code> allows for a start and end index, or you can provide an enumerated type as an index. You should usually start the index range at 0, but if you can not, then the examples below will need to be offset by the value of the start index.</p>
<p>At declaration of the array a block of memory will now exist consisting of 10 doubles in a row. The address of a static array is the same as the address of its first element. This means <code class="language-plaintext highlighter-rouge">@LStaticArray</code> and <code class="language-plaintext highlighter-rouge">@LStaticArray[0]</code> return the same value.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">Assert</span><span class="p">(@</span><span class="n">LStaticArray</span> <span class="p">=</span> <span class="p">@</span><span class="n">LStaticArray</span><span class="p">[</span><span class="m">0</span><span class="p">]);</span> </code></pre></figure>
<p>If we call <code class="language-plaintext highlighter-rouge">SizeOf</code> on the static array we get the total size of all the elements.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">Assert</span><span class="p">(</span><span class="n">SizeOf</span><span class="p">(</span><span class="n">LStaticArray</span><span class="p">)</span> <span class="p">=</span> <span class="n">SizeOf</span><span class="p">(</span><span class="n">LStaticArray</span><span class="p">[</span><span class="m">0</span><span class="p">])</span> <span class="p">*</span> <span class="n">Length</span><span class="p">(</span><span class="n">LStaticArray</span><span class="p">));</span></code></pre></figure>
<p>The value of <code class="language-plaintext highlighter-rouge">SizeOf</code> is the same size as the structure in memory, because the distance from element to element (stride) is simply the size of the element type.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">Assert</span><span class="p">(</span><span class="n">NativeUInt</span><span class="p">(@</span><span class="n">LStaticArray</span><span class="p">[</span><span class="m">1</span><span class="p">])</span> <span class="p">-</span> <span class="n">NativeUInt</span><span class="p">(@</span><span class="n">LStaticArray</span><span class="p">[</span><span class="m">0</span><span class="p">])</span> <span class="p">=</span> <span class="n">SizeOf</span><span class="p">(</span><span class="kt">Double</span><span class="p">));</span></code></pre></figure>
<p>In the example above <code class="language-plaintext highlighter-rouge">@StaticArray[0]</code> and <code class="language-plaintext highlighter-rouge">@StaticArray[1]</code> are pointers to our first and second element. We can see that their addresses differ by the size of the element: <code class="language-plaintext highlighter-rouge">SizeOf(Double)</code>.In production code many developers would rather use <code class="language-plaintext highlighter-rouge">SizeOf(LStaticArray[0])</code> because the code will continue to work if the type of the array needs to change.</p>
<p>If we assign one fixed array to another we copy by value. The array does not move around in memory but stays at its initial location. Assignment simply copies the entire memory content.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="n">TTenDoubles</span> <span class="p">=</span> <span class="k">Array</span><span class="p">[</span><span class="m">0..9</span><span class="p">]</span> <span class="k">of</span> <span class="kt">Double</span><span class="p">;</span>
<span class="k">var</span>
<span class="n">LStaticArray</span> <span class="p">:</span> <span class="n">TTenDoubles</span><span class="p">;</span>
<span class="n">LStaticArray2</span> <span class="p">:</span> <span class="n">TTenDoubles</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">LStaticArray2</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="p">:=</span> <span class="s">'123.456'</span><span class="p">;</span>
<span class="n">LStaticArray</span> <span class="p">:=</span> <span class="n">LStaticArray2</span><span class="p">;</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">LStaticArray</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="p">=</span> <span class="n">LStaticArray2</span><span class="p">[</span><span class="m">0</span><span class="p">]);</span> <span class="c1">// values match
</span> <span class="n">Assert</span><span class="p">(@</span><span class="n">LStaticArray</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="p"><></span> <span class="p">@</span><span class="n">LStaticArray2</span><span class="p">[</span><span class="m">0</span><span class="p">]);</span> <span class="c1">// addresses do not!
</span><span class="k">end</span><span class="p">;</span></code></pre></figure>
<p class="notice"><em>Note:</em> We had to declare a type to allow assignment, if you’d like to understand why then read about the rules of <a href="http://docwiki.embarcadero.com/RADStudio/Rio/en/Overloads_and_Type_Compatibility_in_Generics">assignment compatibility</a>.</p>
<p>When we assign a record, the data contained within the size of the record’s definition is copied, but the memory locations remain the same. Static arrays and record structures can have an overhead due to copy by value. If we were to declare a method that takes our fixed length array or record as a parameter, we should usually declare the parameter as <code class="language-plaintext highlighter-rouge">var</code> or <code class="language-plaintext highlighter-rouge">const</code>.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">procedure</span> <span class="n">foo</span><span class="p">(</span><span class="n">ADblArray</span><span class="p">:</span> <span class="n">TTenDoubles</span><span class="p">);</span>
<span class="k">begin</span>
<span class="c1">// When I was called, I copied all the values passed into ADblArray
</span><span class="k">end</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">goo</span><span class="p">(</span><span class="k">var</span> <span class="n">ADblArray</span><span class="p">:</span> <span class="n">TTenDoubles</span><span class="p">);</span>
<span class="k">begin</span>
<span class="c1">// I hold a reference to the original record passed via ADblArray
</span><span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>If we declare a fixed array as a field in our record it will contribute to the size of the record. The memory will be contained within the record structure. This provides for simple memory management.</p>
<p>Fixed arrays can easily be used cross platform when defined as part of our records.</p>
<h3 id="dynamic-arrays">Dynamic Arrays</h3>
<p>Dynamic arrays are declared without specifying the number of elements:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">var</span>
<span class="n">LDynamicArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Double</span><span class="p">>;</span>
<span class="n">LDynamicArray2</span><span class="p">:</span> <span class="k">Array</span> <span class="k">of</span> <span class="kt">Double</span><span class="p">;</span></code></pre></figure>
<p>Even though these two arrays are the same, I recommend that you use the generic syntax (<code class="language-plaintext highlighter-rouge">TArray<T></code>) because it allows us to assign arrays of the same type. Dynamic arrays are allocated by calling <code class="language-plaintext highlighter-rouge">SetLength()</code></p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">var</span>
<span class="n">LDynamicArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Double</span><span class="p">>;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">,</span> <span class="m">10</span><span class="p">);</span> <span class="c1">//allocate an array of 10 doubles
</span><span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Like our static array the distance between elements is the size of the elements</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">Assert</span><span class="p">(</span><span class="n">NativeUInt</span><span class="p">(@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">1</span><span class="p">])</span> <span class="p">-</span> <span class="n">NativeUInt</span><span class="p">(@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">0</span><span class="p">])</span> <span class="p">=</span> <span class="n">SizeOf</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">0</span><span class="p">]));</span> </code></pre></figure>
<p>However, the memory address of dynamic array is not the same as address of the first element:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">Assert</span><span class="p">(@</span><span class="n">LDynamicArray</span> <span class="p"><></span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">0</span><span class="p">]);</span></code></pre></figure>
<p>If we change our dynamic array size we will find that the pointer to the dynamic array stays the same, while the pointer to the data (the raw memory array), may change. <code class="language-plaintext highlighter-rouge">SetLength</code> also copies the memory of the array to the newly allocated array as needed.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">var</span>
<span class="n">LDynamicArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Double</span><span class="p">>;</span>
<span class="n">lPtrArray</span><span class="p">:</span> <span class="kt">Pointer</span><span class="p">;</span>
<span class="n">LPtrArrayData</span><span class="p">:</span> <span class="kt">Pointer</span><span class="p">;</span>
<span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">,</span> <span class="m">10</span><span class="p">);</span>
<span class="n">lPtrArray</span> <span class="p">:=</span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">;</span>
<span class="n">LPtrArrayData</span> <span class="p">:=</span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">0</span><span class="p">];</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="m">9</span> <span class="k">do</span>
<span class="n">LDynamicArray</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">:=</span> <span class="kt">Double</span><span class="p">(</span><span class="n">i</span> <span class="p">*</span> <span class="m">1.2</span><span class="p">);</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">,</span> <span class="m">100</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">lPtrArray</span> <span class="p">=</span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">);</span> <span class="c1">// Memory address of dynamic array is the same after SetLength
</span> <span class="n">Assert</span><span class="p">(</span><span class="n">LPtrArrayData</span> <span class="p"><></span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">0</span><span class="p">]);</span> <span class="c1">// The raw data array moved
</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="m">9</span> <span class="k">do</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">=</span> <span class="kt">Double</span><span class="p">(</span><span class="n">i</span> <span class="p">*</span> <span class="m">1.2</span><span class="p">));</span> <span class="c1">// The values of the first 10 elements remain the same
</span>
<span class="n">LPtrArrayData</span> <span class="p">:=</span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">0</span><span class="p">];</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">,</span> <span class="m">5</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">lPtrArray</span> <span class="p">=</span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">);</span> <span class="c1">// Memory address of dynamic array is the same after SetLength
</span> <span class="n">Assert</span><span class="p">(</span><span class="n">LPtrArrayData</span> <span class="p"><></span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">0</span><span class="p">]);</span> <span class="c1">// The raw data array moved
</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="m">4</span> <span class="k">do</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">=</span> <span class="kt">Double</span><span class="p">(</span><span class="n">i</span> <span class="p">*</span> <span class="m">1.2</span><span class="p">));</span> <span class="c1">//the remaining 5 elements remained the same
</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p class="notice"><em>Note:</em> When calling <code class="language-plaintext highlighter-rouge">SetLength</code> we may get the same memory address for our raw data, usually when shrinking the array. You may need to grow or shrink the array by many elements to force a new address.</p>
<p>So what exactly is a dynamic array? It is a pointer variable type that holds the address to the first element of data.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">Assert</span><span class="p">(</span><span class="kt">Pointer</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">)</span> <span class="p">=</span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="m">0</span><span class="p">]);</span></code></pre></figure>
<p>Let us think of pointers in simple terms again, they are essentially just numeric values. If we assign an integer variable to another their values match, even though the variables that hold these values have their own independent addresses. Similarly, each dynamic array variable has its own address, but once we assign a dynamic array the value held in this variable is shared between the two.</p>
<p>However, as soon as we call <code class="language-plaintext highlighter-rouge">SetLength()</code> on a dynamic array a new value may be written, so the arrays may independent again.</p>
<p>Since assigning dynamic arrays sets the two variables to refer to the same data, we may encounter some unexpected concequences.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">var</span>
<span class="n">LDynamicArray</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Double</span><span class="p">>;</span>
<span class="n">LDynamicArray2</span><span class="p">:</span> <span class="n">TArray</span><span class="p"><</span><span class="kt">Double</span><span class="p">>;</span>
<span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="n">V</span><span class="p">:</span> <span class="kt">double</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">try</span>
<span class="n">SetLength</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">,</span> <span class="m">10</span><span class="p">);</span>
<span class="n">LDynamicArray2</span> <span class="p">:=</span> <span class="n">LDynamicArray</span><span class="p">;</span>
<span class="n">Assert</span><span class="p">(@</span><span class="n">LDynamicArray2</span> <span class="p"><></span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">);</span> <span class="c1">// The Dynamic arrays have different addresses
</span>
<span class="n">Assert</span><span class="p">(</span><span class="kt">Pointer</span><span class="p">(</span><span class="n">LDynamicArray2</span><span class="p">)</span> <span class="p">=</span> <span class="kt">Pointer</span><span class="p">(</span><span class="n">LDynamicArray</span><span class="p">);</span> <span class="c1">// But they refer to the same raw data!
</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="m">9</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="n">V</span> <span class="p">:=</span> <span class="n">i</span> <span class="p">*</span> <span class="m">1.2</span><span class="p">;</span>
<span class="n">LDynamicArray</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">:=</span> <span class="n">V</span><span class="p">;</span> <span class="c1">// modifying the elements of one array
</span> <span class="n">Assert</span><span class="p">(</span><span class="n">LDynamicArray2</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">=</span> <span class="n">V</span><span class="p">);</span> <span class="c1">// affects the other
</span> <span class="n">Assert</span><span class="p">(@</span><span class="n">LDynamicArray2</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">=</span> <span class="p">@</span><span class="n">LDynamicArray</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span> <span class="c1">// because the data exists in the same memory location
</span> <span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>The benefit of this behavior is that we can have large amounts of data passed around using dynamic arrays. Dynamic arrays are reference counted and automatically disposed. Any API that uses the raw data pointer in a dynamic array must ensure that:</p>
<ul>
<li>The dynamic array is referenced for as long as we need a pointer to the raw data</li>
<li>The pointer to the raw data needs to be updated after <code class="language-plaintext highlighter-rouge">SetLength</code> is called against the dynamic array</li>
</ul>
<p class="notice"><em>Note:</em> If you are interested in how dynamic arrays know the element count and number of references look at <code class="language-plaintext highlighter-rouge">TDynArrayRec</code> in <code class="language-plaintext highlighter-rouge">system.pas</code>. This record is stored right before the first element of the array.</p>
<p>Dynamic arrays, strings and interfaces are reference counted and are automatically initialized to nil.</p>
<p>One last note on dynamic arrays. Since a dynamic array is simply a pointer to an array, it should not be a surprise that <code class="language-plaintext highlighter-rouge">SizeOf(LDynamicArray)</code> for any dynamic array is always <code class="language-plaintext highlighter-rouge">SizeOf(Pointer)</code>.</p>
<h3 id="records-with-variable-length-arrays">Records with Variable Length Arrays</h3>
<p>We can declare records that hold arrays where we use the syntax of a fixed length array (typically with a single element), but where the actual size of the array varies.</p>
<p>In Part 1 we encountered one of these:</p>
<p><a href="https://docs.microsoft.com/en-us/windows/desktop/api/dbt/ns-dbt-_dev_broadcast_deviceinterface_w">DEV_BROADCAST_DEVICEINTERFACE</a>. Which is defined as</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">typedef</span> <span class="k">struct</span> <span class="n">_DEV_BROADCAST_DEVICEINTERFACE_W</span> <span class="p">{</span>
<span class="n">DWORD</span> <span class="n">dbcc_size</span><span class="p">;</span>
<span class="n">DWORD</span> <span class="n">dbcc_devicetype</span><span class="p">;</span>
<span class="n">DWORD</span> <span class="n">dbcc_reserved</span><span class="p">;</span>
<span class="n">GUID</span> <span class="n">dbcc_classguid</span><span class="p">;</span>
<span class="kt">wchar_t</span> <span class="n">dbcc_name</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="p">}</span> <span class="n">DEV_BROADCAST_DEVICEINTERFACE_W</span><span class="p">,</span> <span class="o">*</span><span class="n">PDEV_BROADCAST_DEVICEINTERFACE_W</span><span class="p">;</span></code></pre></figure>
<p>Which I translated to Delphi as</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">PDEV_BROADCAST_DEVICEINTERFACE</span> <span class="p">=</span> <span class="p">^</span><span class="n">DEV_BROADCAST_DEVICEINTERFACE</span><span class="p">;</span>
<span class="n">DEV_BROADCAST_DEVICEINTERFACE</span> <span class="p">=</span> <span class="k">record</span>
<span class="n">dbcc_size</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbcc_devicetype</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span> <span class="c1">// = DBT_DEVTYP_DEVICEINTERFACE
</span> <span class="n">dbcc_reserved</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbcc_classguid</span><span class="p">:</span> <span class="n">TGUID</span><span class="p">;</span>
<span class="n">dbcc_name</span><span class="p">:</span> <span class="kt">Char</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Strictly speaking, my direct conversion of <code class="language-plaintext highlighter-rouge">dbcc_name</code> should have been declared as <code class="language-plaintext highlighter-rouge">dbcc_name: array[0..0] of Char;</code>, a fixed length array of one character! Well, the actual name of the device is actually longer than just one character. The record’s size as described in <code class="language-plaintext highlighter-rouge">dbcc_size</code> includes the length of the string contained in <code class="language-plaintext highlighter-rouge">dbcc_name</code> including the null character.</p>
<p>In Delphi <code class="language-plaintext highlighter-rouge">pChar</code> is a pointer to <code class="language-plaintext highlighter-rouge">Char</code>. We can read characters beyond that first character by indexing (e.g. <code class="language-plaintext highlighter-rouge">LMyPChar[5]</code>). Delphi provides support to convert from <code class="language-plaintext highlighter-rouge">pChar</code> to <code class="language-plaintext highlighter-rouge">String</code>. When a variable of type <code class="language-plaintext highlighter-rouge">pChar</code> is converted to <code class="language-plaintext highlighter-rouge">String</code> the characters are read until a null character (character with value 0) is reached. This means that we can read a null terminated character array without knowing its size, or in this case without reading the size of the record.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="c1">// WMDeviceChange sample in lesson1
</span> <span class="n">LUsbDeviceName</span> <span class="p">:=</span> <span class="kt">PChar</span><span class="p">(@</span><span class="n">LPBroadcastDeviceIntf</span><span class="p">^.</span><span class="n">dbcc_name</span><span class="p">);</span></code></pre></figure>
<p class="notice--warning"><strong>Watch out!</strong> Assigning a variable of type <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_DEVICEINTERFACE</code> will transfer the <code class="language-plaintext highlighter-rouge">dbcc_size</code>, but only the first character of <code class="language-plaintext highlighter-rouge">dbcc_name</code>.</p>
<p>Typically records such as these are used in APIs where a single record is passed by reference. Since the size of <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_DEVICEINTERFACE</code> varies we cannot place it in an array. We can place a series of these records in a row, but indexing will not work because we do not have a fixed stride length between elements. In that case we would have to move our pointer by the size of each record in turn.</p>
<p>To create a record such as this you would allocate memory equal to the size of the record definition plus the length of the character string. That size value would then also be written to the <code class="language-plaintext highlighter-rouge">dbcc_size</code> field.</p>
<p>Bonus: A dynamic array has a record header, <code class="language-plaintext highlighter-rouge">TDynArrayRec</code>, written before the raw data of the array. Knowing this, you may notice the similarity between the dynamic array data and the <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_DEVICEINTERFACE</code> record. Both have headers before the raw data. Both are allocated with space for the header and the number of elements in the data. In the case of dynamic arrays the <code class="language-plaintext highlighter-rouge">Setlength()</code> function allocates the header and data space for us and returns the address of the first element of the array data. If we want to read the number of elements or the reference count we move a pointer to the raw data back by
<code class="language-plaintext highlighter-rouge">SizeOf(NativeInt)</code> to read the <code class="language-plaintext highlighter-rouge">Length: NativeInt</code> and back by an additional 4 bytes to read <code class="language-plaintext highlighter-rouge">RefCnt: Integer</code>. Next I will discuss ways we can move our pointer.</p>
<h2 id="pointer-math">Pointer Math</h2>
<p>We can pass a pointer to the first element of our array to our DLL, we can also pass the number of elements. If we know what stride to take from the start of one item to the next, then we can walk our pointer by simply adding (or deleting) the size of this stride to the numeric value of the pointer. As we walk our pointer and de-reference a memory address, we can read the items in our array.</p>
<p>The question then is, what is the stride between my elements? Can we safely use the size of our record?</p>
<p>In Part 1 we saw that our records are aligned to memory boundaries, but the record may also be padded at its end. If another record of the same type is now placed in memory directly after the first, then it’s starting address will be correct, and no alignment is needed. This end padding counts towards the size of the record, which means that we can use the size of the record as our stride from one element to the next.</p>
<h3 id="inc-and-dec">Inc and Dec</h3>
<p>Calling <code class="language-plaintext highlighter-rouge">Inc</code> and <code class="language-plaintext highlighter-rouge">Dec</code> increments and decrements our pointer’s value by the size of the pointer’s underlying type.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="n">PTestRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TTestRec</span><span class="p">;</span>
<span class="n">TTestRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">Field1</span><span class="p">:</span> <span class="kt">word</span><span class="p">;</span>
<span class="n">Field2</span><span class="p">:</span> <span class="kt">byte</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="k">var</span>
<span class="n">LTestPtr</span><span class="p">:</span> <span class="n">PTestRec</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">LTestPtr</span> <span class="p">:=</span> <span class="nb">nil</span><span class="p">;</span> <span class="c1">// nil has an integer value of 0
</span> <span class="k">Inc</span><span class="p">(</span><span class="n">LTestPtr</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">NativeUInt</span><span class="p">(</span><span class="n">LTestPtr</span><span class="p">)</span> <span class="p">=</span> <span class="n">SizeOf</span><span class="p">(</span><span class="n">TTestRec</span><span class="p">));</span>
<span class="n">Dec</span><span class="p">(</span><span class="n">LTestPtr</span><span class="p">);</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">LTestPtr</span> <span class="p">=</span> <span class="nb">nil</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>As we call <code class="language-plaintext highlighter-rouge">Inc</code> on our pointer starting at the first element it will simply advance and reference element by element.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">var</span>
<span class="n">LTestPtr</span><span class="p">:</span> <span class="n">PTestRec</span><span class="p">;</span>
<span class="n">LTestArray</span> <span class="p">:</span> <span class="k">Array</span><span class="p">[</span><span class="m">0..5</span><span class="p">]</span> <span class="k">of</span> <span class="n">TTestRec</span><span class="p">;</span>
<span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">LTestPtr</span> <span class="p">:=</span> <span class="p">@</span><span class="n">LTestArray</span><span class="p">[</span><span class="m">0</span><span class="p">];</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="m">5</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">LTestPtr</span> <span class="p">=</span> <span class="p">@</span><span class="n">LTestArray</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
<span class="k">Inc</span><span class="p">(</span><span class="n">LTestPtr</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>So, with a strongly typed pointer and the count we can easily access all of our elements in the array in a sequential fashion.</p>
<h3 id="treat-the-pointer-as-a-number">Treat the Pointer as a number</h3>
<p>To avoid the <code class="language-plaintext highlighter-rouge">Inc</code> and <code class="language-plaintext highlighter-rouge">Dev</code> side effect of modifying the pointer, we can also do the math ourselves. All we must do is multiply the size of the element with the number of elements to walk and then add to our starting point.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">var</span>
<span class="n">LTestPtr</span><span class="p">:</span> <span class="n">PTestRec</span><span class="p">;</span>
<span class="n">LTestArray</span> <span class="p">:</span> <span class="k">Array</span><span class="p">[</span><span class="m">0..5</span><span class="p">]</span> <span class="k">of</span> <span class="n">TTestRec</span><span class="p">;</span>
<span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">LTestPtr</span> <span class="p">:=</span> <span class="p">@</span><span class="n">LTestArray</span><span class="p">[</span><span class="m">0</span><span class="p">];</span> <span class="c1">//starting point
</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="m">5</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="n">Assert</span><span class="p">(</span> <span class="n">PTestRec</span><span class="p">(</span><span class="n">NativeUInt</span><span class="p">(</span><span class="n">LTestPtr</span><span class="p">)</span> <span class="p">+</span> <span class="n">i</span> <span class="p">*</span><span class="n">SizeOf</span><span class="p">(</span><span class="n">TTestRec</span><span class="p">))</span> <span class="p">=</span> <span class="p">@</span><span class="n">LTestArray</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>There is a shorthand for this notation</p>
<h3 id="pointer-indexing">Pointer Indexing</h3>
<p>Writing <code class="language-plaintext highlighter-rouge">PTestRec[i]</code> is essentially the same as <code class="language-plaintext highlighter-rouge">PTestRec(NativeUInt(LTestPtr) + i *SizeOf(TTestRec))^</code> and allows us to use the same indexing notation as we would use with arrays. This notation works with a few built-in types, most notably <code class="language-plaintext highlighter-rouge">PChar</code>, but we can use the <code class="language-plaintext highlighter-rouge">{$POINTERMATH ON}</code> directive to allow indexing on our own pointer types</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">var</span>
<span class="n">LTestPtr</span><span class="p">:</span> <span class="n">PTestRec</span><span class="p">;</span>
<span class="n">LTestArray</span> <span class="p">:</span> <span class="k">Array</span><span class="p">[</span><span class="m">0..5</span><span class="p">]</span> <span class="k">of</span> <span class="n">TTestRec</span><span class="p">;</span>
<span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">LTestPtr</span> <span class="p">:=</span> <span class="p">@</span><span class="n">LTestArray</span><span class="p">[</span><span class="m">0</span><span class="p">];</span> <span class="c1">// our pointer reference the first element in our array
</span>
<span class="cm">{$POINTERMATH ON}</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="m">5</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="n">Assert</span><span class="p">(@</span><span class="n">LTestPtr</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">=</span> <span class="p">@</span><span class="n">LTestArray</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span> <span class="c1">// we index based on our pointer in the same way as the array
</span> <span class="k">end</span><span class="p">;</span>
<span class="c1">// Modifying via our array or using our indexed pointer modifies the same data
</span> <span class="n">LTestArray</span><span class="p">[</span><span class="m">2</span><span class="p">].</span><span class="n">Field1</span> <span class="p">:=</span> <span class="m">12</span><span class="p">;</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">LTestPtr</span><span class="p">[</span><span class="m">2</span><span class="p">].</span><span class="n">Field1</span> <span class="p">=</span> <span class="m">12</span><span class="p">);</span>
<span class="n">LTestPtr</span><span class="p">[</span><span class="m">3</span><span class="p">].</span><span class="n">Field2</span> <span class="p">:=</span> <span class="m">34</span><span class="p">;</span>
<span class="n">Assert</span><span class="p">(</span><span class="n">LTestArray</span><span class="p">[</span><span class="m">3</span><span class="p">].</span><span class="n">Field2</span> <span class="p">=</span> <span class="m">34</span><span class="p">);</span>
<span class="cm">{$POINTERMATH OFF}</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Most of you would have indexed <code class="language-plaintext highlighter-rouge">PChar</code>. The index syntax here is the same, but just expanded for our own pointer types. <code class="language-plaintext highlighter-rouge">PChar</code> is different from most other data pointers in that it usually does not need to be paired with a size. A <code class="language-plaintext highlighter-rouge">PChar</code> is usually terminated by a null character. We can potentially index data beyond the end of our array with other pointer types and we need to have the element count passed via our API. With the count and a pointer to a known structure type (or some way to derive this information), we can read data in arrays in most programming languages</p>
<p>Bonus: You may be wondering why our spacing of elements work out so easily. Well in Delphi all data types are multiples of others (1, 2, 4, 8 etc.) and so they align to multiples of their size. There is one type that violates this rule :<code class="language-plaintext highlighter-rouge">Extended</code>, and only in 32-bit. In 32-bit <code class="language-plaintext highlighter-rouge">Extended</code> has 10 bytes, but in 64-bit it is an alias for <code class="language-plaintext highlighter-rouge">Double</code>. This type is the only type that sometimes requires the rarely used <code class="language-plaintext highlighter-rouge">packed array</code>. I will not cover that type since it is hard to use cross-platform and we should avoid it in APIs.</p>
<h2 id="section-conclusion">Section Conclusion</h2>
<p>Arrays are contiguous blocks of same size values. Pointers to the raw data of arrays can be used to traverse the elements in an array. To successfully traverse an array in our API we need a pointer to the start of the data of the array, the size of the elements and the number of elements to read.</p>
<p>In the next section I will cover the basic API of our DLL that will allow it to receive records and arrays.</p>Jasper SchellingerhoutIn this blog post I will continue with arrays in communicating cross-platform via DLLs.Data Transmission with Delphi (Part 1: Pointers and Structures)2019-02-11T00:00:00+00:002019-02-11T00:00:00+00:00https://schellingerhout.github.io/data%20transmission/datatransmission1<p>In this blog post I will start to describe the data structures and the use of pointers in communicating cross-platform via DLLs.</p>
<!--more-->
<p>The purpose of this blog series is to understand how to transfer raw data at high speed with direct access using pointers to structures that can be supported by most programming platforms.</p>
<p>When we transmit data between an application and a DLL, we operate in the same memory space. This means that we can read memory directly via pointers. This allows high speed transmission of data by directly reading memory as common structures shared between the application and DLLs.</p>
<h2 id="pointer-primer">Pointer Primer</h2>
<p>Many tutorials on pointers exist already and this is not a replacement for those, just a basic primer and refresher on high level concepts.</p>
<p>Think of memory as a grid of boxes where each box has a sequential identifier (or address). If we have a memory position in this grid as well as knowledge about the number of boxes to read beyond that position, and their interpretation, then two systems with this common understanding can create an efficient system of directly sharing information.</p>
<p>In the model described above a pointer is simply the address of a box as described above. I like to think of pointers as “native unsigned integers”. Thinking about pointers this way helps keep our sanity when passing pointers by value or by reference. The value held by the pointer behaves in the same way as a numeric value passed by value or by reference.</p>
<p>Let us look at a sample of a pointer type defined in System.SysUtils.pas</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">PThreadInfo</span> <span class="p">=</span> <span class="p">^</span><span class="n">TThreadInfo</span><span class="p">;</span>
<span class="n">TThreadInfo</span> <span class="p">=</span> <span class="k">record</span>
<span class="n">Next</span><span class="p">:</span> <span class="n">PThreadInfo</span><span class="p">;</span>
<span class="n">ThreadID</span><span class="p">:</span> <span class="n">TThreadID</span><span class="p">;</span>
<span class="n">Active</span><span class="p">:</span> <span class="kt">Integer</span><span class="p">;</span>
<span class="n">RecursionCount</span><span class="p">:</span> <span class="kt">Cardinal</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">PThreadInfo</code> is a pointer type to <code class="language-plaintext highlighter-rouge">TThreadInfo</code>. The type <code class="language-plaintext highlighter-rouge">PThreadInfo</code> has exacly the same size as <code class="language-plaintext highlighter-rouge">Pointer</code>, which is the same size as <code class="language-plaintext highlighter-rouge">NativeUInt</code> (32-bit or 64-bit based on the platform). So, if the <code class="language-plaintext highlighter-rouge">PThreadInfo</code> essentially just holds a numeric value just like any other pointer, then why declare it in this way? Well, as I mentioned earlier, if we know how much memory we need to read and how to interpret its values then we know what the data represents. When we de-reference a typed pointer variable like <code class="language-plaintext highlighter-rouge">LMyThreadPtr: PThreadInfo</code>, by calling in this form: <code class="language-plaintext highlighter-rouge">LMyThreadPtr^</code>, then we interpret the memory at the position value held by <code class="language-plaintext highlighter-rouge">LMyThreadPtr</code> as memory with a stucture defined by <code class="language-plaintext highlighter-rouge">TThreadInfo</code>.</p>
<p>Besides an understanding of the data at the memory location of the pointer, the strongly typed pointer also has benefits in an area called “pointer math”. In short, this means that if we increment our pointer it will not always increment by 1, but by the size of the associated type. Pointer math also allows for indexing memory positions based on a pointer type. I will explain in more detail in a future post.</p>
<p>Now we have both parts to make our communication between application and DLL possible: a memory address and an interpretation of that memory. Let us look at an example of what our communication might look like</p>
<h2 id="abstract-structured-types-in-window-messaging">Abstract Structured Types in Window Messaging</h2>
<p>Windows messaging deals with lots of varying information and types in a generic way. Let us look at one scenario of windows messaging how this works with abstract treatment of pointers.</p>
<p>The windows message <a href="https://docs.microsoft.com/en-us/windows/desktop/DevIO/wm-devicechange">WM_DEVICECHANGE</a> can inform us of device changes on our system. Once we receive this method and the <code class="language-plaintext highlighter-rouge">wmparam</code> has a value of <a href="https://docs.microsoft.com/en-us/windows/desktop/DevIO/dbt-devicearrival">DBT_DEVICEARRIVAL</a> we know that a device arrived. Then we can read the <code class="language-plaintext highlighter-rouge">lparam</code> for information for device. In the windows message strucutre <code class="language-plaintext highlighter-rouge">lparam</code> is simply a <code class="language-plaintext highlighter-rouge">NativeInt</code> (i.e. a numeric value), and for this windows message we can treat it as a pointer to <a href="https://docs.microsoft.com/en-us/windows/desktop/api/Dbt/ns-dbt-_dev_broadcast_hdr">DEV_BROADCAST_HDR</a></p>
<p>The structure and its strongly typed pointer looks like this in Delphi:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">PDEV_BROADCAST_HDR</span> <span class="p">=</span> <span class="p">^</span><span class="n">DEV_BROADCAST_HDR</span><span class="p">;</span>
<span class="n">DEV_BROADCAST_HDR</span> <span class="p">=</span> <span class="k">record</span>
<span class="n">dbch_size</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbch_devicetype</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbch_reserved</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>The information we can glean form this is limited to essentially just the device type, so how can we get more detail? If you read the documentation for <a href="https://docs.microsoft.com/en-us/windows/desktop/api/Dbt/ns-dbt-_dev_broadcast_hdr">DEV_BROADCAST_HDR</a>. You will see that for each of the values of the device type it tells you that the structure is not really <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_HDR</code>, but some other structure like <a href="https://docs.microsoft.com/en-us/windows/desktop/api/dbt/ns-dbt-_dev_broadcast_deviceinterface_a">DEV_BROADCAST_DEVICEINTERFACE</a> or <a href="https://docs.microsoft.com/en-us/windows/desktop/api/dbt/ns-dbt-_dev_broadcast_volume">DEV_BROADCAST_VOLUME</a>. You may be confused at this point, but let us look at those two types as translated to Delphi code:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">PDEV_BROADCAST_DEVICEINTERFACE</span> <span class="p">=</span> <span class="p">^</span><span class="n">DEV_BROADCAST_DEVICEINTERFACE</span><span class="p">;</span>
<span class="n">DEV_BROADCAST_DEVICEINTERFACE</span> <span class="p">=</span> <span class="k">record</span>
<span class="n">dbcc_size</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbcc_devicetype</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span> <span class="c1">// = DBT_DEVTYP_DEVICEINTERFACE
</span> <span class="n">dbcc_reserved</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbcc_classguid</span><span class="p">:</span> <span class="n">TGUID</span><span class="p">;</span>
<span class="n">dbcc_name</span><span class="p">:</span> <span class="kt">Char</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="n">PDEV_BROADCAST_VOLUME</span> <span class="p">=</span> <span class="p">^</span><span class="n">DEV_BROADCAST_VOLUME</span><span class="p">;</span>
<span class="n">DEV_BROADCAST_VOLUME</span> <span class="p">=</span> <span class="k">record</span>
<span class="n">dbcv_size</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbcv_devicetype</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span> <span class="c1">// = DBT_DEVTYP_VOLUME
</span> <span class="n">dbcv_reserved</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbcv_unitmask</span><span class="p">:</span> <span class="kt">DWORD</span><span class="p">;</span>
<span class="n">dbcv_flags</span><span class="p">:</span> <span class="kt">Word</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Look at the two records above and compare them to <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_HDR</code>. You will notice that the first part of their data is the same. <code class="language-plaintext highlighter-rouge">DEV_BROADCAST_HDR</code> is an abstract structure. Once we cast our pointer from the header type <code class="language-plaintext highlighter-rouge">PDEV_BROADCAST_HDR</code> to the specific type (say <code class="language-plaintext highlighter-rouge">PDEV_BROADCAST_DEVICEINTERFACE</code>) we can now read more information. All of this works of course if both parties have a clear and unambiguous understanding of the size of data that needs to be read.</p>
<p>Without going into detail. Here is an example of how we would use this</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">procedure</span> <span class="n">TFoo</span><span class="p">.</span><span class="n">WMDeviceChange</span><span class="p">(</span><span class="k">var</span> <span class="n">AMessage</span><span class="p">:</span> <span class="n">TMessage</span><span class="p">);</span>
<span class="k">var</span>
<span class="n">LUsbDeviceName</span><span class="p">:</span> <span class="k">string</span><span class="p">;</span>
<span class="n">LPDeviceBroadcastHeader</span><span class="p">:</span> <span class="n">PDEV_BROADCAST_HDR</span><span class="p">;</span>
<span class="n">LPBroadcastDeviceIntf</span><span class="p">:</span> <span class="n">PDEV_BROADCAST_DEVICEINTERFACE</span><span class="p">;</span>
<span class="n">LPBroadcastVolume</span><span class="p">:</span> <span class="n">PDEV_BROADCAST_VOLUME</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="p">(</span><span class="n">AMessage</span><span class="p">.</span><span class="n">wParam</span> <span class="p">=</span> <span class="n">DBT_DEVICEARRIVAL</span><span class="p">)</span> <span class="k">then</span>
<span class="k">begin</span>
<span class="n">LPDeviceBroadcastHeader</span> <span class="p">:=</span> <span class="n">PDEV_BROADCAST_HDR</span><span class="p">(</span><span class="n">AMessage</span><span class="p">.</span><span class="n">LParam</span><span class="p">);</span>
<span class="k">case</span> <span class="n">LPDeviceBroadcastHeader</span><span class="p">^.</span><span class="n">dbch_devicetype</span> <span class="k">of</span>
<span class="n">DBT_DEVTYP_DEVICEINTERFACE</span><span class="p">:</span>
<span class="k">begin</span>
<span class="n">LPBroadcastDeviceIntf</span> <span class="p">:=</span> <span class="n">PDEV_BROADCAST_DEVICEINTERFACE</span><span class="p">(</span><span class="n">LPDeviceBroadcastHeader</span><span class="p">);</span>
<span class="n">LUsbDeviceName</span> <span class="p">:=</span> <span class="kt">PChar</span><span class="p">(@</span><span class="n">LPBroadcastDeviceIntf</span><span class="p">^.</span><span class="n">dbcc_name</span><span class="p">);</span>
<span class="p">...</span>
<span class="k">end</span><span class="p">;</span>
<span class="n">DBT_DEVTYP_VOLUME</span><span class="p">:</span>
<span class="k">begin</span>
<span class="n">LPBroadcastVolume</span> <span class="p">:=</span> <span class="n">PDEV_BROADCAST_VOLUME</span><span class="p">(</span><span class="n">LPDeviceBroadcastHeader</span><span class="p">);</span>
<span class="c1">// use LPBroadcastVolume.dbcv_unitmask to find volume information
</span> <span class="p">...</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h2 id="shared-structures-naive-attempt">Shared Structures (Naive Attempt)</h2>
<p>As long as our application and DLL have a common understanding of the structure they need to pass the call could be a simple one</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">procedure</span> <span class="n">foo</span><span class="p">(</span><span class="n">AData</span><span class="p">:</span> <span class="kt">Pointer</span><span class="p">);</span> <span class="k">stdcall</span><span class="p">;</span></code></pre></figure>
<p>The DLL receiving the information can cast to the correct pointer type and read the memory at that address, but this would not be very extensible and reduce readability of the source code. Look at the windows message handler routine we defined earlier. In that case we use the less abstract <code class="language-plaintext highlighter-rouge">PDEV_BROADCAST_HDR</code> type and cast to the appropriate <code class="language-plaintext highlighter-rouge">PDEV_BROADCAST_*</code> pointer type based on <code class="language-plaintext highlighter-rouge">dbch_devicetype</code>. We can do the same in our shared structure (record) definitions.</p>
<p>Let use start by creating our header or “base” record type. As seen above we need a type so we can cast to correct pointer type. An enumeration would work fine</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="n">TxRectTypeEnum</span> <span class="p">=</span> <span class="p">(</span><span class="n">TxRectType_Undefined</span><span class="p">,</span> <span class="n">TxRectType_Line</span><span class="p">,</span> <span class="n">TxRectType_Arc</span><span class="p">);</span>
<span class="n">PTxRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxRec</span><span class="p">;</span>
<span class="n">TxRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span></code></pre></figure>
<p>Next we can define our Line and Arc records, but we have to keep in mind that the first data needs to match that of TxRec.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="n">PTxRecArc</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxRecArc</span><span class="p">;</span>
<span class="n">TxRecArc</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// common to TxRec
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span> <span class="c1">// = TxRectType_Arc
</span>
<span class="c1">// TxRecArc data start
</span> <span class="n">CenterX</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">CenterY</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">StartAng</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">EndAng</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">Radius</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">CCW</span><span class="p">:</span> <span class="kt">boolean</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PTxRecLine</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxRecLine</span><span class="p">;</span>
<span class="n">TxRecLine</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// common to TxRec
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span> <span class="c1">// = TxRectType_Line
</span>
<span class="c1">// TxRecLine data start
</span> <span class="n">P1X</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">P1Y</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">P2X</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">P2Y</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
</code></pre></figure>
<p>The type alone will work for single records transmitted by our API. We will need the size if we read a complex stream of data (for instance from a file). It would be prudent to also add the size to our base type. If we ever encounter a stream that has a record that we don’t understand we can skip that section of memory.</p>
<h2 id="shared-structures-slightly-improved">Shared Structures (Slightly Improved)</h2>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">type</span>
<span class="n">PTxRec</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxRec</span><span class="p">;</span>
<span class="n">TxRec</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">Size</span><span class="p">:</span> <span class="kt">DWord</span><span class="p">;</span>
<span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PTxRecArc</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxRecArc</span><span class="p">;</span>
<span class="n">TxRecArc</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// common to TxRec
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">DWord</span><span class="p">;</span>
<span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span> <span class="c1">// = TxRectType_Arc
</span>
<span class="c1">// TxRecArc data start
</span> <span class="n">CenterX</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">CenterY</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">StartAng</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">EndAng</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">Radius</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">CCW</span><span class="p">:</span> <span class="kt">boolean</span>
<span class="k">End</span><span class="p">;</span>
<span class="n">PTxRecLine</span> <span class="p">=</span> <span class="p">^</span><span class="n">TxRecLine</span><span class="p">;</span>
<span class="n">TxRecLine</span> <span class="p">=</span> <span class="k">Record</span>
<span class="c1">// common to TxRec
</span> <span class="n">Size</span><span class="p">:</span> <span class="kt">DWord</span><span class="p">;</span>
<span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span> <span class="c1">// = TxRectType_Line
</span>
<span class="c1">// TxRecLine data start
</span> <span class="n">P1X</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">P1Y</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">P2X</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">P2Y</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="k">End</span><span class="p">;</span>
</code></pre></figure>
<p>Records don’t always occupy the memory size equivalent to the size of their component values, so adding the size allow receivers to read the correct amount of data when reading streams. There is another issue to consider: data members and records themselves have the data aligned to certain size boundaries. Since all systems reading the memory need a clear understanding of the data, we need to understand how this data can be represented in an unambiguous way. The <code class="language-plaintext highlighter-rouge">Size</code> value can be read by both parties as a first line of defense to know that records are represented in a similar way, but for complete safety we need to ensure that our data alignment matches.</p>
<h2 id="record-alignment-and-padding">Record Alignment and Padding</h2>
<p>Records are usually <a href="https://en.wikipedia.org/wiki/Data_structure_alignment">padded and aligned</a>, this means they have their data members and the record as a whole are padded to certain size multiples. This speeds up memory access and indexing of data. We could declare our records as <code class="language-plaintext highlighter-rouge">Packed</code> to prevent alignment or we could use explicit compiler defines (<code class="language-plaintext highlighter-rouge">$A</code> or <code class="language-plaintext highlighter-rouge">$ALIGN</code>) to set the alignment. As long as all parties involved in the transfer of the data have the same understanding of the data structure the data can be read correctly.</p>
<p>Records are by aligned by the smaller of:</p>
<ul>
<li>The record alignment setting. By default, 8-byte (Quad Word)</li>
<li>The size of the largest element</li>
</ul>
<p>Record fields are aligned by the smaller of:</p>
<ul>
<li>The alignment of fields of its own size</li>
<li>The alignment of the record (see above)</li>
</ul>
<p>By placing larger members before smaller ones, we could create a more compact structure because we would eliminate data member alignment requirements. This is true because data types are usually multiples of smaller types (1,2,4,8 bytes). 8 Byte alignment is the same standard used by the Windows SDK and the reason why we could do a verbatim translation of the C++ struct to a Delphi record.</p>
<p>If we now look at our header record the <code class="language-plaintext highlighter-rouge">RecType: TxRectTypeEnum</code> gains no space benefit defined as a type with a size of 1 byte. By default, enumerated types are byte, but since our <code class="language-plaintext highlighter-rouge">Size</code> is <code class="language-plaintext highlighter-rouge">DWord</code> the record will be aligned by at least 4 bytes. This means that the size of the record will be 8 instead of 5. In our Line and Arc records it means that a pad of 3 bytes will be added before <code class="language-plaintext highlighter-rouge">CenterX</code> and before <code class="language-plaintext highlighter-rouge">P1X</code> in <code class="language-plaintext highlighter-rouge">TxRecArc</code> and <code class="language-plaintext highlighter-rouge">TxRecLine</code> respectively. Similarly, since our record will be aligned to a boundary of 8 bytes (since it contains <code class="language-plaintext highlighter-rouge">Double</code>), a pad of 7 bytes is added to the end.</p>
<p>In memory our Record really looks like this (I added comments where padding is done)</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TxRecArc</span> <span class="p">=</span> <span class="k">Record</span>
<span class="n">Size</span><span class="p">:</span> <span class="kt">DWord</span><span class="p">;</span> <span class="c1">// 4 bytes. Offset 0
</span> <span class="n">RecType</span><span class="p">:</span> <span class="n">TxRectTypeEnum</span><span class="p">;</span> <span class="c1">// = TxRectType_Arc //1 byte //Offset 4
</span>
<span class="c1">//Pad1 : Array[0..2] of Byte; // implicit pad added to align CenterX to align to a multiple of 8 bytes
</span> <span class="n">CenterX</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span> <span class="c1">// offset 8
</span> <span class="n">CenterY</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">StartAng</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">EndAng</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">Radius</span><span class="p">:</span> <span class="kt">Double</span><span class="p">;</span>
<span class="n">CCW</span><span class="p">:</span> <span class="kt">boolean</span><span class="p">;</span> <span class="c1">// 1 byte. offset 48
</span> <span class="c1">//Pad2 : Array[0..6] of Byte; // implicit pad added to align record with the next record
</span><span class="k">End</span><span class="p">;</span></code></pre></figure>
<p>Since enumerated types are typically 32-bit in other programming languages we might as well set our <code class="language-plaintext highlighter-rouge">TxRectTypeEnum</code> to be 4 bytes using (the <code class="language-plaintext highlighter-rouge">$Z</code> compiler directive). Doing so this <code class="language-plaintext highlighter-rouge">Pad1</code> will disappear and we will be more compatible with C++. Also, since we have padding after CCW we could use <code class="language-plaintext highlighter-rouge">WordBool</code> or <code class="language-plaintext highlighter-rouge">LongBool</code> datatypes without growing our record footprint</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="cm">{$Z4}</span>
<span class="n">TxRectTypeEnum</span> <span class="p">=</span> <span class="p">(</span><span class="n">TxRectType_Undefined</span><span class="p">,</span> <span class="n">TxRectType_Line</span><span class="p">,</span> <span class="n">TxRectType_Arc</span><span class="p">);</span>
<span class="cm">{$Z1}</span></code></pre></figure>
<p>If we use “standard” alignment rules (8 Byte) we remain compatible with Windows SDK standards. Most applications can easily be configured to use our records and we won’t have surprises about alignment of fields or records.</p>
<p>I highly recommend staying within the 8-Byte (Quad Word) record alignment scheme. It is well known and widely used. If parties use this alignment the fields will be aligned as expected.</p>
<h2 id="section-conclusion">Section Conclusion</h2>
<p>Pointers are numeric values. Strongly typed pointers allow us to understand the structure and values at the address held by a pointer. Casting a pointer allows for a different or extended interpretation of the data at a memory address. As long as two parties have common interpretation of the structure of memory data can be easily read or written by either.</p>
<p>In the next section I will expand on pointer math and the transmission of arrays of data.</p>Jasper SchellingerhoutIn this blog post I will start to describe the data structures and the use of pointers in communicating cross-platform via DLLs.Implementing the Sieve of Eratosthenes in Delphi2017-12-01T00:00:00+00:002017-12-01T00:00:00+00:00https://schellingerhout.github.io/algorithms/sieve-of-eratosthenes<p>The Sieve of Eratosthenes is a very fast, but memory intensive algorithm to find the primes under a given value.</p>
<!--more-->
<p>I first read about this method when I was about 12 in a book called “Mathematics: a Human Edeavor” by Harold R. Jacobs. I had forgotten about the details of this method until I stumbled on it again the other day. Here is how I developed a solution in Delphi:</p>
<h2 id="the-algortihm">The algortihm</h2>
<p>Start by creating a field of numbers from 3 up to desired maximum.</p>
<p><strong>3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63</strong></p>
<p>Mark multiples of 2 starting with 2<sup>2</sup></p>
<p><strong>3</strong>, <s>4</s>, <strong>5</strong>, <s>6</s>, <strong>7</strong>, <s>8</s>, <strong>9</strong>, <s>10</s>, <strong>11</strong>, <s>12</s>, <strong>13</strong>, <s>14</s>, <strong>15</strong>, <s>16</s>, <strong>17</strong>, <s>18</s>, <strong>19</strong>, <s>20</s>, <strong>21</strong>, <s>22</s>, <strong>23</strong>, <s>24</s>, <strong>25</strong>, <s>26</s>, <strong>27</strong>, <s>28</s>, <strong>29</strong>, <s>30</s>, <strong>31</strong>, <s>32</s>, <strong>33</strong>, <s>34</s>, <strong>35</strong>, <s>36</s>, <strong>37</strong>, <s>38</s>, <strong>39</strong>, <s>40</s>, <strong>41</strong>, <s>42</s>, <strong>43</strong>, <s>44</s>, <strong>45</strong>, <s>46</s>, <strong>47</strong>, <s>48</s>, <strong>49</strong>, <s>50</s>, <strong>51</strong>, <s>52</s>, <strong>53</strong>, <s>54</s>, <strong>55</strong>, <s>56</s>, <strong>57</strong>, <s>58</s>, <strong>59</strong>, <s>60</s>, <strong>61</strong>, <s>62</s>, <strong>63</strong></p>
<p>After this we repeatedly find the lowest unmarked value (which will always be prime) and mark its multiple starting with its square</p>
<p>Mark multiples of 3 starting with 3<sup>2</sup></p>
<p><strong>3</strong>, <s>4</s>, <strong>5</strong>, <s>6</s>, <strong>7</strong>, <s>8</s>, <s>9</s>, <s>10</s>, <strong>11</strong>, <s>12</s>, <strong>13</strong>, <s>14</s>, <s>15</s>, <s>16</s>, <strong>17</strong>, <s>18</s>, <strong>19</strong>, <s>20</s>, <s>21</s>, <s>22</s>, <strong>23</strong>, <s>24</s>, <strong>25</strong>, <s>26</s>, <s>27</s>, <s>28</s>, <strong>29</strong>, <s>30</s>, <strong>31</strong>, <s>32</s>, <s>33</s>, <s>34</s>, <strong>35</strong>, <s>36</s>, <strong>37</strong>, <s>38</s>, <s>39</s>, <s>40</s>, <strong>41</strong>, <s>42</s>, <strong>43</strong>, <s>44</s>, <s>45</s>, <s>46</s>, <strong>47</strong>, <s>48</s>, <strong>49</strong>, <s>50</s>, <s>51</s>, <s>52</s>, <strong>53</strong>, <s>54</s>, <strong>55</strong>, <s>56</s>, <s>57</s>, <s>58</s>, <strong>59</strong>, <s>60</s>, <strong>61</strong>, <s>62</s>, <s>63</s></p>
<p>The next unmarked value greater than 3 is 5. Mark multiples of 5 starting with 5<sup>2</sup></p>
<p><strong>3</strong>, <s>4</s>, <strong>5</strong>, <s>6</s>, <strong>7</strong>, <s>8</s>, <s>9</s>, <s>10</s>, <strong>11</strong>, <s>12</s>, <strong>13</strong>, <s>14</s>, <s>15</s>, <s>16</s>, <strong>17</strong>, <s>18</s>, <strong>19</strong>, <s>20</s>, <s>21</s>, <s>22</s>, <strong>23</strong>, <s>24</s>, <s>25</s>, <s>26</s>, <s>27</s>, <s>28</s>, <strong>29</strong>, <s>30</s>, <strong>31</strong>, <s>32</s>, <s>33</s>, <s>34</s>, <s>35</s>, <s>36</s>, <strong>37</strong>, <s>38</s>, <s>39</s>, <s>40</s>, <strong>41</strong>, <s>42</s>, <strong>43</strong>, <s>44</s>, <s>45</s>, <s>46</s>, <strong>47</strong>, <s>48</s>, <strong>49</strong>, <s>50</s>, <s>51</s>, <s>52</s>, <strong>53</strong>, <s>54</s>, <s>55</s>, <s>56</s>, <s>57</s>, <s>58</s>, <strong>59</strong>, <s>60</s>, <strong>61</strong>, <s>62</s>, <s>63</s></p>
<p>7 is the next unmarked value. So we mark multiples of 7 starting at 7<sup>2</sup></p>
<p><strong>3</strong>, <s>4</s>, <strong>5</strong>, <s>6</s>, <strong>7</strong>, <s>8</s>, <s>9</s>, <s>10</s>, <strong>11</strong>, <s>12</s>, <strong>13</strong>, <s>14</s>, <s>15</s>, <s>16</s>, <strong>17</strong>, <s>18</s>, <strong>19</strong>, <s>20</s>, <s>21</s>, <s>22</s>, <strong>23</strong>, <s>24</s>, <s>25</s>, <s>26</s>, <s>27</s>, <s>28</s>, <strong>29</strong>, <s>30</s>, <strong>31</strong>, <s>32</s>, <s>33</s>, <s>34</s>, <s>35</s>, <s>36</s>, <strong>37</strong>, <s>38</s>, <s>39</s>, <s>40</s>, <strong>41</strong>, <s>42</s>, <strong>43</strong>, <s>44</s>, <s>45</s>, <s>46</s>, <strong>47</strong>, <s>48</s>, <s>49</s>, <s>50</s>, <s>51</s>, <s>52</s>, <strong>53</strong>, <s>54</s>, <s>55</s>, <s>56</s>, <s>57</s>, <s>58</s>, <strong>59</strong>, <s>60</s>, <strong>61</strong>, <s>62</s>, <s>63</s></p>
<p>11 is the next prime, but 11<sup>2</sup> is outside our max. We are done and all unmarked values are prime. You can see how this algorithm can be very quick esp. since we only have to mark prime multiples and then only multiples greater than the square of each prime in turn.</p>
<h2 id="what-is-the-downside-to-this-method">What is the downside to this method?</h2>
<p>We need at least one bit of storage per number that we want to mark. For simplicity if our bitfields start at 0 (and I’ll show why that is a good idea soon) and we want to mark primes up to max(uint32) 4,294,967,295. Then we would need 512 MB just for the bits. We could shrink the bitfield by only storing the odd values and starting from 3.</p>
<p>The algorithm is very fast, but the memory constraints make it problematic to get large primes.</p>
<p>Supposing that we will need to use memory for other purposes, such as storing the prime factorization of numbers we should consider building and running the application as a 64-bit binary to allow easy access to more memory.</p>
<h2 id="implementation-in-delphi">Implementation in Delphi</h2>
<h3 id="the-bitfield">The Bitfield</h3>
<p>For the purposes of keeping the implementation simple and still gaining speed I will make the bitfield starting at 0 and a bit for each value. Index 0 represents 0, index n represents n. This keeps the logic very simple. I simply index by each value directly and set the bit. But, what if we want to store values larger than our sample bit-field of 64 bits? We could simply keep an array of 64-bit fields</p>
<p>for example we could have <code class="language-plaintext highlighter-rouge">FBitFieldArray : Array[0..2] of UInt64;</code> to represent 0 to 191</p>
<p><code class="language-plaintext highlighter-rouge">FBitFieldArray[0]</code> would be 0 to 63, <code class="language-plaintext highlighter-rouge">FBitFieldArray[1]</code> would be 64 to 127, <code class="language-plaintext highlighter-rouge">FBitFieldArray[2]</code> would be 128 to 191. Indexing on field [0] is easy, we just use our number, but what about the other fields. In Delphi integer division is done with <code class="language-plaintext highlighter-rouge">Div</code>, while the <code class="language-plaintext highlighter-rouge">Mod</code> method gets the remainder (modulus) of a number after division. Both can be done at the same time with <code class="language-plaintext highlighter-rouge">DivMod</code>. Since we have blocks of 64 we can divide by 64 to get the block we are in and the remainder would be the position inside that block</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">DivMod</span><span class="p">(</span><span class="k">index</span><span class="p">,</span> <span class="m">64</span><span class="p">,</span> <span class="n">LBlockIndex</span><span class="p">,</span> <span class="n">LBlockBitIndex</span><span class="p">);</span></code></pre></figure>
<p>We will build our code based on 3 blocks and then we will generalize later</p>
<h3 id="2-is-the-odd-man-out">2 is the odd man out</h3>
<p>All primes are odd except for 2, which forces us to have a check every iteration of the loop for this exceptional case. In all other cases we can check for our next prime value starting with the previous prime’s position plus 2.</p>
<p>To eliminate this the check for 2 each iteration (and since 2 is a trivial case) we can set all the bits in our bitfield for 2s first before repeating the same process for primes 3 and up.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">LMultiple</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="n">LNonPrime</span> <span class="p">:=</span> <span class="m">4</span><span class="p">;</span> <span class="c1">//start at 2^2
</span><span class="k">while</span> <span class="n">LNonPrime</span> <span class="p"><=</span> <span class="m">191</span> <span class="k">do</span> <span class="c1">// maximum value represented by block [2]
</span><span class="k">begin</span>
<span class="err"> </span> <span class="n">DivMod</span><span class="p">(</span><span class="n">LNonPrime</span><span class="p">,</span> <span class="m">64</span><span class="p">,</span> <span class="n">LBlockIndex</span><span class="p">,</span> <span class="n">LBlockBitIndex</span><span class="p">);</span>
<span class="err"> </span>
<span class="c1">// Set bit LBlockBitIndex in FValueBitField[LBlockIndex] to 1
</span>
<span class="err"> </span> <span class="k">inc</span><span class="p">(</span><span class="n">LMultiple</span><span class="p">);</span>
<span class="err"> </span> <span class="n">LNonPrime</span> <span class="p">:=</span> <span class="m">2</span> <span class="p">*</span> <span class="n">LMultiple</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>But, this is not needed. We know that setting every even value will just yield a bit pattern of alternating ones and zeros 101010101. We do have a special case though; 2 is not prime and, of course, 0 and 1 are not prime numbers either. Since binary 1010<sub>2</sub> is just hexadecimal 5<sub>16</sub> we can set all blocks to <code class="language-plaintext highlighter-rouge">$5555555555555555</code> except for the very first block. The only difference for the first block is that the first 4 bits would be 0011<sub>2</sub> or 3<sub>16</sub>. So, we can set the first block to <code class="language-plaintext highlighter-rouge">$5555555555555553</code></p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">FValueBitField</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="p">:=</span> <span class="p">$</span><span class="m">5555555555555553</span><span class="p">;</span>
<span class="n">FValueBitField</span><span class="p">[</span><span class="m">1</span><span class="p">]</span> <span class="p">:=</span> <span class="p">$</span><span class="m">5555555555555555</span><span class="p">;</span>
<span class="n">FValueBitField</span><span class="p">[</span><span class="m">2</span><span class="p">]</span> <span class="p">:=</span> <span class="n">FValueBitField</span><span class="p">[</span><span class="m">1</span><span class="p">];</span></code></pre></figure>
<p>Now we can move on to real algorithm.</p>
<h3 id="eliminate-multiples-of-primes">Eliminate multiples of primes</h3>
<p>Starting with prime 3 we can now mark multiples of 3 as not prime. We have to exclude 3 itself of course, but we already covered 2*3 so we don’t need to do 3*2 as well. Therefore we can start at 3*3. This logic applies as we move to each next prime, so we can always start with multiples of our prime starting with its square</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">LPrime</span> <span class="p">:=</span> <span class="m">3</span><span class="p">;</span>
<span class="k">repeat</span>
<span class="err"> </span> <span class="n">LMultiple</span> <span class="p">:=</span> <span class="n">LPrime</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LNonPrime</span> <span class="p">:=</span> <span class="n">LPrime</span> <span class="p">*</span> <span class="n">LMultiple</span><span class="p">;</span>
<span class="err"> </span> <span class="k">while</span> <span class="n">LNonPrime</span> <span class="p"><=</span> <span class="n">FMaxValue</span> <span class="k">do</span>
<span class="err"> </span> <span class="k">begin</span>
<span class="err"> </span> <span class="n">DivMod</span><span class="p">(</span><span class="n">LNonPrime</span><span class="p">,</span> <span class="m">64</span><span class="p">,</span> <span class="n">LBlockIndex</span><span class="p">,</span> <span class="n">LBlockBitIndex</span><span class="p">);</span>
<span class="err"> </span>
<span class="c1">// Set bit LBlockBitIndex in FValueBitField[LBlockIndex]
</span>
<span class="err"> </span> <span class="k">inc</span><span class="p">(</span><span class="n">LMultiple</span><span class="p">);</span>
<span class="err"> </span> <span class="n">LNonPrime</span> <span class="p">:=</span> <span class="n">LPrime</span> <span class="p">*</span> <span class="n">LMultiple</span><span class="p">;</span>
<span class="err"> </span> <span class="k">end</span><span class="p">;</span>
<span class="err"> </span>
<span class="c1">// Set LPrime to the next prime
</span><span class="k">until</span> <span class="p">(</span><span class="n">LPrime</span> <span class="p">>=</span> <span class="m">14</span><span class="p">);</span> <span class="c1">// 14 ^ 2 is greater than 191</span></code></pre></figure>
<h3 id="setting-the-bit-corresponding-to-a-composite-number">Setting the bit corresponding to a composite number</h3>
<p>Next we need to find a way to set a specific bit by index. <code class="language-plaintext highlighter-rouge">LBlockBitIndex</code> tells us the bit we need to set, but how do we actually set a bit based on that index? In numeric terms each bit n represents 2<sup>n</sup>, therefore we want the value 2<sup>LBlockBitIndex</sup>.</p>
<p>However, we don’t actually need to calculate a power of 2, we can use bit shifting. The <code class="language-plaintext highlighter-rouge">shl</code> method in Delphi shifts bits left. For instance if we start with 10000<sub>2</sub> and shift 2 left we’d get 1000000<sub>2</sub>. Using <code class="language-plaintext highlighter-rouge">1 shl LBlockBitIndex</code> we will shift the value by the number of bits. At first glance this looks OK, but what size is the <code class="language-plaintext highlighter-rouge">1</code> constant used in that expression? Its unlikely to be 64-bit, which means that shifts over the size of the <code class="language-plaintext highlighter-rouge">1</code> constant will give us incorrect results. To ensure that we shift for a 64-bit unsigned buffer value we can make it explicit by casting <code class="language-plaintext highlighter-rouge">UInt64(1) shl LBlockBitIndex</code></p>
<p class="notice--warning"><strong>Watch out!</strong> Don’t assume the sizes of declared constants either used in expressions or declared with a const statement. If you need a specific size, make it explicit</p>
<p>Now that we have a value with the bit all we need to do is to apply a bitwise <code class="language-plaintext highlighter-rouge">or</code> with the correct block in the field. This way we will set the bit at that index and leave all other bits in the field as they were before</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">FValueBitField</span><span class="p">[</span><span class="n">LBlockIndex</span><span class="p">]</span> <span class="p">:=</span> <span class="n">FValueBitField</span><span class="p">[</span><span class="n">LBlockIndex</span><span class="p">]</span> <span class="k">or</span>
<span class="err"> </span> <span class="p">(</span><span class="n">UInt64</span><span class="p">(</span><span class="m">1</span><span class="p">)</span> <span class="k">shl</span> <span class="n">LBlockBitIndex</span><span class="p">);</span></code></pre></figure>
<p>Delphi uses the <code class="language-plaintext highlighter-rouge">or</code> operator for logical and bitwise <code class="language-plaintext highlighter-rouge">or</code> operations. The type of operation depends on the operands, in this case we have a bitwise logical operation.</p>
<h3 id="finding-the-next-prime">Finding the next prime</h3>
<p>The sieve eliminates primes by marking multiples so if we start with our last prime and move bit by bit until we find an unmarked bit then that bit will represent our next prime. So given we our field how do we do this?</p>
<p>We know that we have checked all values up to p (our last prime). p is 3 or larger so we know we can check for even unmarked bits p+2 or higher. In a loop for n we can shift right by p+n+2 (n > 0) until the right most bit is 0.</p>
<p>Of course as we increment to get the next prime we may loop over to the next block</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">function</span> <span class="n">TSieveOfEratosthenes</span><span class="p">.</span><span class="n">GetNextPrime</span><span class="p">(</span><span class="n">APreviousPrime</span>
<span class="err"> </span> <span class="p">:</span> <span class="n">uint32</span><span class="p">):</span> <span class="n">uint32</span><span class="p">;</span>
<span class="k">var</span>
<span class="err"> </span> <span class="n">LBlock</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LBlockIndex</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LBlockBitIndex</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="k">begin</span>
<span class="err"> </span> <span class="n">DivMod</span><span class="p">(</span><span class="n">APreviousPrime</span> <span class="p">+</span> <span class="m">2</span><span class="p">,</span> <span class="m">64</span><span class="p">,</span> <span class="n">LBlockIndex</span><span class="p">,</span> <span class="n">LBlockBitIndex</span><span class="p">);</span>
<span class="err"> </span> <span class="k">repeat</span>
<span class="err"> </span> <span class="n">LBlock</span> <span class="p">:=</span> <span class="n">FValueBitField</span><span class="p">[</span><span class="n">LBlockIndex</span><span class="p">]</span> <span class="k">shr</span> <span class="n">LBlockBitIndex</span><span class="p">;</span>
<span class="err"> </span> <span class="k">if</span> <span class="n">LBlockBitIndex</span> <span class="p">>=</span> <span class="m">64</span> <span class="k">then</span>
<span class="err"> </span> <span class="k">begin</span>
<span class="err"> </span> <span class="n">LBlockBitIndex</span> <span class="p">:=</span> <span class="m">0</span><span class="p">;</span>
<span class="err"> </span> <span class="k">inc</span><span class="p">(</span><span class="n">LBlockIndex</span><span class="p">);</span>
<span class="err"> </span> <span class="k">end</span>
<span class="err"> </span> <span class="k">else</span>
<span class="err"> </span> <span class="k">inc</span><span class="p">(</span><span class="n">LBlockBitIndex</span><span class="p">);</span>
<span class="err"> </span> <span class="k">until</span> <span class="p">(</span><span class="n">LBlock</span> <span class="k">and</span> <span class="m">1</span> <span class="p">=</span> <span class="m">0</span><span class="p">);</span>
<span class="err"> </span> <span class="n">result</span> <span class="p">:=</span> <span class="n">LBlockIndex</span> <span class="p">*</span> <span class="m">64</span> <span class="p">+</span> <span class="n">LBlockBitIndex</span> <span class="p">-</span> <span class="m">1</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>This code is not fully optimized, but shows the process of finding the next unset bit. Lastly since we convert the Block and Bit index to a number which will be our next prime (we subtract 1 because our bitindex is one out of phase).</p>
<h3 id="generalizing-the-sample">Generalizing the sample</h3>
<p>First lets remedy the fact that we use “magic numbers” all over the place. While it is fine to write comments to explain the use of numbers that are hard to derive from context. Instead of using 64 for our block size we can declare a constant</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">const</span>
<span class="err"> </span> <span class="n">BlockSize</span> <span class="p">=</span> <span class="m">64</span><span class="p">;</span></code></pre></figure>
<p>We make assumptions that the number of blocks are fixed. That may not be the case, we want to make the sieve more flexible and allow it to expand according to demands</p>
<p>Let start by defining <code class="language-plaintext highlighter-rouge">MaxValue</code>; this will be the cap when searching for primes. To represent that value we would need <code class="language-plaintext highlighter-rouge">Ceil(MaxValue/BlockSize)</code> blocks in our field, so that leads us to <code class="language-plaintext highlighter-rouge">MaxBlockIndex := Floor(MaxValue/BlockSize)</code>.</p>
<p>We know that we start marking multiples of squares of primes, therefore we don’t have to check primes greater than the integer square root of <code class="language-plaintext highlighter-rouge">MaxValue</code>. This gives us <code class="language-plaintext highlighter-rouge">MaxValueISqrt := Trunc(Sqrt(MaxValue))</code>.</p>
<p>Since we can set all of these as dependency on a single value we can write something like this:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">procedure</span> <span class="n">TSieveOfEratosthenes</span><span class="p">.</span><span class="n">SetMaxValue</span><span class="p">(</span><span class="k">const</span> <span class="k">Value</span><span class="p">:</span> <span class="n">uint32</span><span class="p">);</span>
<span class="k">begin</span>
<span class="err"> </span> <span class="n">FMaxValue</span> <span class="p">:=</span> <span class="k">Value</span><span class="p">;</span>
<span class="err"> </span> <span class="n">FMaxValueISqrt</span> <span class="p">:=</span> <span class="n">Trunc</span><span class="p">(</span><span class="n">Sqrt</span><span class="p">(</span><span class="n">FMaxValue</span><span class="p">));</span>
<span class="err"> </span> <span class="n">FMaxBlockIndex</span> <span class="p">:=</span> <span class="n">floor</span><span class="p">(</span><span class="n">FMaxValue</span> <span class="p">/</span> <span class="n">BlockSize</span><span class="p">);</span>
<span class="err"> </span> <span class="n">SetLength</span><span class="p">(</span><span class="n">FValueBitField</span><span class="p">,</span> <span class="n">FMaxBlockIndex</span> <span class="p">+</span> <span class="m">1</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>With these changes our main loop will now look more readable. Here is our main routine that creates the prime mask</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">procedure</span> <span class="n">TSieveOfEratosthenes</span><span class="p">.</span><span class="n">GetPrimeMask</span><span class="p">;</span>
<span class="k">var</span>
<span class="err"> </span> <span class="n">i</span><span class="p">:</span> <span class="n">uint32</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LPrime</span><span class="p">:</span> <span class="n">uint32</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LMultiple</span><span class="p">:</span> <span class="n">uint32</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LNonPrime</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LBlockIndex</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LBlockBitIndex</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="k">begin</span>
<span class="c1">//we could set the initial state of the blocks. since
</span> <span class="c1">// multiples of 2 are really trivial bit patterns
</span>
<span class="c1">// first block has repeat(0101) 0011
</span> <span class="n">FValueBitField</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="p">:=</span> <span class="n">UInt64</span><span class="p">($</span><span class="m">5555555555555553</span><span class="p">);</span>
<span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">1</span> <span class="k">to</span> <span class="n">FMaxBlockIndex</span> <span class="k">do</span>
<span class="k">begin</span>
<span class="c1">// other blocks have repeat(0101)
</span> <span class="n">FValueBitField</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">:=</span> <span class="n">UInt64</span><span class="p">($</span><span class="m">5555555555555555</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LPrime</span> <span class="p">:=</span> <span class="m">3</span><span class="p">;</span>
<span class="err"> </span> <span class="k">repeat</span>
<span class="err"> </span> <span class="n">LMultiple</span> <span class="p">:=</span> <span class="n">LPrime</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LNonPrime</span> <span class="p">:=</span> <span class="n">LPrime</span> <span class="p">*</span> <span class="n">LMultiple</span><span class="p">;</span>
<span class="err"> </span> <span class="k">while</span> <span class="n">LNonPrime</span> <span class="p"><=</span> <span class="n">FMaxValue</span> <span class="k">do</span>
<span class="err"> </span> <span class="k">begin</span>
<span class="err"> </span> <span class="n">DivMod</span><span class="p">(</span><span class="n">LNonPrime</span><span class="p">,</span> <span class="n">BlockSize</span><span class="p">,</span> <span class="n">LBlockIndex</span><span class="p">,</span> <span class="n">LBlockBitIndex</span><span class="p">);</span>
<span class="err"> </span>
<span class="err"> </span> <span class="n">FValueBitField</span><span class="p">[</span><span class="n">LBlockIndex</span><span class="p">]</span> <span class="p">:=</span> <span class="n">FValueBitField</span><span class="p">[</span><span class="n">LBlockIndex</span><span class="p">]</span> <span class="k">or</span>
<span class="err"> </span> <span class="p">(</span><span class="n">UInt64</span><span class="p">(</span><span class="m">1</span><span class="p">)</span> <span class="k">shl</span> <span class="n">LBlockBitIndex</span><span class="p">);</span>
<span class="err"> </span> <span class="k">inc</span><span class="p">(</span><span class="n">LMultiple</span><span class="p">);</span>
<span class="err"> </span> <span class="n">LNonPrime</span> <span class="p">:=</span> <span class="n">LPrime</span> <span class="p">*</span> <span class="n">LMultiple</span><span class="p">;</span>
<span class="err"> </span> <span class="k">end</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LPrime</span> <span class="p">:=</span> <span class="n">GetNextPrime</span><span class="p">(</span><span class="n">LPrime</span><span class="p">);</span>
<span class="err"> </span> <span class="k">until</span> <span class="p">(</span><span class="n">LPrime</span> <span class="p">>=</span> <span class="n">FMaxValueISqrt</span><span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>We can also clean up our <code class="language-plaintext highlighter-rouge">GetNextPrime</code> to add in our new variables:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">function</span> <span class="n">TSieveOfEratosthenes</span><span class="p">.</span><span class="n">GetNextPrime</span><span class="p">(</span><span class="n">APreviousPrime</span>
<span class="err"> </span> <span class="p">:</span> <span class="n">uint32</span><span class="p">):</span> <span class="n">uint32</span><span class="p">;</span>
<span class="k">var</span>
<span class="err"> </span> <span class="n">LBlock</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LBlockIndex</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LBlockBitIndex</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="k">begin</span>
<span class="err"> </span> <span class="n">DivMod</span><span class="p">(</span><span class="n">APreviousPrime</span> <span class="p">+</span> <span class="m">2</span><span class="p">,</span> <span class="n">BlockSize</span><span class="p">,</span> <span class="n">LBlockIndex</span><span class="p">,</span> <span class="n">LBlockBitIndex</span><span class="p">);</span>
<span class="err"> </span> <span class="k">repeat</span>
<span class="err"> </span> <span class="n">LBlock</span> <span class="p">:=</span> <span class="n">FValueBitField</span><span class="p">[</span><span class="n">LBlockIndex</span><span class="p">]</span> <span class="k">shr</span> <span class="n">LBlockBitIndex</span><span class="p">;</span>
<span class="err"> </span> <span class="k">if</span> <span class="n">LBlockBitIndex</span> <span class="p">>=</span> <span class="n">BlockSize</span> <span class="k">then</span>
<span class="err"> </span> <span class="k">begin</span>
<span class="err"> </span> <span class="n">LBlockBitIndex</span> <span class="p">:=</span> <span class="m">0</span><span class="p">;</span>
<span class="err"> </span> <span class="k">inc</span><span class="p">(</span><span class="n">LBlockIndex</span><span class="p">);</span>
<span class="err"> </span> <span class="k">end</span>
<span class="err"> </span> <span class="k">else</span>
<span class="err"> </span> <span class="k">inc</span><span class="p">(</span><span class="n">LBlockBitIndex</span><span class="p">);</span>
<span class="err"> </span> <span class="k">until</span> <span class="p">(</span><span class="n">LBlock</span> <span class="k">and</span> <span class="m">1</span> <span class="p">=</span> <span class="m">0</span><span class="p">);</span>
<span class="err"> </span> <span class="n">result</span> <span class="p">:=</span> <span class="n">LBlockIndex</span> <span class="p">*</span> <span class="n">BlockSize</span> <span class="p">+</span> <span class="n">LBlockBitIndex</span> <span class="p">-</span> <span class="m">1</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h3 id="finding-primes-up-to-a-maximum">Finding Primes up to a Maximum</h3>
<p>The process would be to mark all multiples primes using the method above up to the squareroot of the maximum. And then traverse the field and return the indexes of all the zeros. Assume <code class="language-plaintext highlighter-rouge">NumberOfBitsSet</code> is a method that returns the number of 1s in the field, we will cover that in a moment.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">function</span> <span class="n">TSieveOfEratosthenes</span><span class="p">.</span><span class="n">GetPrimes</span><span class="p">(</span><span class="k">const</span> <span class="n">AMaxValue</span><span class="p">:</span> <span class="n">uint32</span><span class="p">)</span>
<span class="err"> </span> <span class="p">:</span> <span class="n">ArrayOfUInt32</span><span class="p">;</span>
<span class="k">var</span>
<span class="err"> </span> <span class="n">LPrime</span><span class="p">:</span> <span class="n">Uint32</span><span class="p">;</span>
<span class="err"> </span> <span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="k">begin</span>
<span class="err"> </span> <span class="n">SetMaxValue</span><span class="p">(</span><span class="n">AMaxValue</span><span class="p">);</span>
<span class="err"> </span> <span class="n">GetPrimeMask</span><span class="p">;</span>
<span class="err"> </span> <span class="n">SetLength</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="n">FMaxValue</span> <span class="p">-</span> <span class="n">NumberOfBitsSet</span><span class="p">);</span>
<span class="err"> </span> <span class="n">result</span><span class="p">[</span><span class="m">0</span><span class="p">]</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="err"> </span> <span class="n">result</span><span class="p">[</span><span class="m">1</span><span class="p">]</span> <span class="p">:=</span> <span class="m">3</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LPrime</span> <span class="p">:=</span> <span class="m">3</span><span class="p">;</span>
<span class="err"> </span>
<span class="n">i</span> <span class="p">:=</span> <span class="m">2</span><span class="p">;</span>
<span class="err"> </span> <span class="k">while</span> <span class="n">LPrime</span> <span class="p"><=</span> <span class="n">FMaxValue</span> <span class="k">do</span>
<span class="err"> </span> <span class="k">begin</span>
<span class="err"> </span> <span class="n">LPrime</span> <span class="p">:=</span> <span class="n">GetNextPrime</span><span class="p">(</span><span class="n">LPrime</span><span class="p">);</span>
<span class="err"> </span> <span class="n">result</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">:=</span> <span class="n">LPrime</span><span class="p">;</span>
<span class="err"> </span> <span class="k">inc</span><span class="p">(</span><span class="n">i</span><span class="p">);</span>
<span class="err"> </span> <span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h3 id="couting-bits">Couting bits</h3>
<p>In order to allocate exactle enough memory to return our primes in the method above we needed to know the number of bits set. The difference between the max value and the number of bits set gives us our number of primes. I found <a href="https://stackoverflow.com/questions/2709430/count-number-of-bits-in-a-64-bit-long-big-integer">this nugget</a> that I converted from C to Delphi. All I do finally is looping through the blocks and accumulating the count</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">class</span> <span class="k">function</span> <span class="n">TSieveOfEratosthenes</span><span class="p">.</span><span class="n">NumberOfBitsSet</span><span class="p">:</span> <span class="n">uint32</span><span class="p">;</span>
<span class="k">var</span>
<span class="err"> </span> <span class="n">i</span><span class="p">:</span> <span class="kt">integer</span><span class="p">;</span>
<span class="err"> </span> <span class="n">LBitFieldCount</span><span class="p">:</span> <span class="n">UInt64</span><span class="p">;</span>
<span class="k">begin</span>
<span class="err"> </span> <span class="n">result</span> <span class="p">:=</span> <span class="m">0</span><span class="p">;</span>
<span class="err"> </span> <span class="k">for</span> <span class="n">i</span> <span class="p">:=</span> <span class="m">0</span> <span class="k">to</span> <span class="n">FMaxBlockIndex</span> <span class="k">do</span>
<span class="err"> </span> <span class="k">begin</span>
<span class="err"> </span> <span class="n">LBitFieldCount</span> <span class="p">:=</span> <span class="n">FValueBitField</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">-</span> <span class="p">((</span><span class="n">FValueBitField</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">shr</span> <span class="m">1</span><span class="p">)</span> <span class="k">and</span>
<span class="err"> </span> <span class="n">UInt64</span><span class="p">($</span><span class="m">5555555555555555</span><span class="p">));</span>
<span class="err"> </span> <span class="n">LBitFieldCount</span> <span class="p">:=</span> <span class="p">(</span><span class="n">LBitFieldCount</span> <span class="k">and</span> <span class="n">UInt64</span><span class="p">($</span><span class="m">3333333333333333</span><span class="p">))</span> <span class="p">+</span>
<span class="err"> </span> <span class="p">((</span><span class="n">LBitFieldCount</span> <span class="k">shr</span> <span class="m">2</span><span class="p">)</span> <span class="k">and</span> <span class="n">UInt64</span><span class="p">($</span><span class="m">3333333333333333</span><span class="p">));</span>
<span class="err"> </span> <span class="n">LBitFieldCount</span> <span class="p">:=</span>
<span class="err"> </span> <span class="kt">byte</span><span class="p">((((</span><span class="n">LBitFieldCount</span> <span class="p">+</span> <span class="p">(</span><span class="n">LBitFieldCount</span> <span class="k">shr</span> <span class="m">4</span><span class="p">))</span> <span class="k">and</span>
<span class="err"> </span> <span class="n">UInt64</span><span class="p">($</span><span class="n">F0F0F0F0F0F0F0F</span><span class="p">))</span> <span class="p">*</span> <span class="n">UInt64</span><span class="p">($</span><span class="m">101010101010101</span><span class="p">))</span> <span class="k">shr</span> <span class="m">56</span><span class="p">);</span>
<span class="err"> </span> <span class="n">result</span> <span class="p">:=</span> <span class="n">result</span> <span class="p">+</span> <span class="n">LBitFieldCount</span><span class="p">;</span>
<span class="err"> </span> <span class="k">end</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>I hope that you found this useful or at least interesting. Please download, clone or fork the <a href="https://github.com/schellingerhout/sieve-of-eratosthenes-delphi">source code</a>.</p>Jasper SchellingerhoutThe Sieve of Eratosthenes is a very fast, but memory intensive algorithm to find the primes under a given value.Active Object Design Pattern in Delphi (Part 3): Futures (Promises)2017-11-25T00:00:00+00:002017-11-25T00:00:00+00:00https://schellingerhout.github.io/design%20patterns/activeobject-pattern3<p>In this blog post I will continue to describe the process to develop a solution in Delphi using the Active Object Design Pattern. Specifically, we will see how Future Values are delivered</p>
<!--more-->
<p>If you missed <a href="/design%20patterns/activeobject-pattern1/">Part 1: Method Requests</a> or <a href="/design%20patterns/activeobject-pattern2/">Part 2: The Scheduler</a>. I would recommend that you read those first before proceeding.</p>
<p>In the third part of the deep dive into the Active Object pattern I want to look at the future value. Unlike the IFuture that is a Task in Delphi, this form of future is simple a container that waits to be filled and triggers the event to release once filled.</p>
<p>We have already looked at the interface we would need in <a href="/design%20patterns/activeobject-pattern1/">Part 1: Method Requests</a>. For reference, here it is again</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">IFutureValue</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="k">interface</span>
<span class="k">function</span> <span class="n">GetValue</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">property</span> <span class="k">Value</span><span class="p">:</span> <span class="n">T</span> <span class="k">read</span> <span class="n">GetValue</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Below is the class defininition for the Future Value.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TFutureValue</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="p">=</span> <span class="k">class</span><span class="p">(</span><span class="n">TInterfacedObject</span><span class="p">,</span> <span class="n">IFutureValue</span><span class="p"><</span><span class="n">T</span><span class="p">>)</span>
<span class="k">private</span>
<span class="n">FResultSet</span><span class="p">:</span> <span class="kt">boolean</span><span class="p">;</span>
<span class="n">FResult</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="n">FValueReadyEvent</span><span class="p">:</span> <span class="n">TLightWeightEvent</span><span class="p">;</span> <span class="c1">// consider balance between TEvent and TLightweight event
</span> <span class="k">function</span> <span class="n">GetValue</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">function</span> <span class="n">Wait</span><span class="p">:</span> <span class="kt">boolean</span><span class="p">;</span>
<span class="k">function</span> <span class="n">GetValueReadyEvent</span><span class="p">:</span> <span class="n">TLightWeightEvent</span><span class="p">;</span>
<span class="k">protected</span>
<span class="k">property</span> <span class="n">ValueReadyEvent</span><span class="p">:</span> <span class="n">TLightWeightEvent</span> <span class="k">read</span> <span class="n">GetValueReadyEvent</span><span class="p">;</span>
<span class="k">public</span>
<span class="k">destructor</span> <span class="n">Destroy</span><span class="p">;</span> <span class="k">override</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">SetValue</span><span class="p">(</span><span class="n">AResult</span><span class="p">:</span> <span class="n">T</span><span class="p">);</span> <span class="c1">// done by methodrequest wrapping a future
</span> <span class="k">property</span> <span class="k">Value</span><span class="p">:</span> <span class="n">T</span> <span class="k">read</span> <span class="n">GetValue</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>Lets examine some of the basics of the definition:</p>
<p>The property <code class="language-plaintext highlighter-rouge">Value</code> uses the <code class="language-plaintext highlighter-rouge">GetValue</code> method to <code class="language-plaintext highlighter-rouge">Wait</code> for the <code class="language-plaintext highlighter-rouge">FValueReadyEvent</code> and then returns <code class="language-plaintext highlighter-rouge">FResult</code>. The boolean <code class="language-plaintext highlighter-rouge">FResultSet</code> is used in a technique called Quick-Check Locking. I will disuss that when we look at the implementation of the class.</p>
<p><code class="language-plaintext highlighter-rouge">SetValue</code> is called by the service that supplies the value (in this case MethodRequest via theScheduler). When <code class="language-plaintext highlighter-rouge">SetValue</code> is called, the <code class="language-plaintext highlighter-rouge">FValueReadyEvent</code> is set and <code class="language-plaintext highlighter-rouge">FResultSet</code> is set to <code class="language-plaintext highlighter-rouge">true</code>.</p>
<p>The <code class="language-plaintext highlighter-rouge">GetValueReadyEvent</code> is used to lazy load the <code class="language-plaintext highlighter-rouge">FValueReadyEvent</code> when first needed.</p>
<p>Here follows the implementation:</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">destructor</span> <span class="n">TFutureValue</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Destroy</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">FValueReadyEvent</span><span class="p">.</span><span class="n">Free</span><span class="p">;</span>
<span class="k">inherited</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">function</span> <span class="n">TFutureValue</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">GetValue</span><span class="p">:</span> <span class="n">T</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">Wait</span><span class="p">;</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">FResult</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">function</span> <span class="n">TFutureValue</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">GetValueReadyEvent</span><span class="p">:</span> <span class="n">TLightWeightEvent</span><span class="p">;</span>
<span class="k">var</span>
<span class="n">LEvent</span><span class="p">:</span> <span class="n">TLightWeightEvent</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">FValueReadyEvent</span> <span class="p">=</span> <span class="nb">nil</span> <span class="k">then</span>
<span class="k">begin</span>
<span class="n">LEvent</span> <span class="p">:=</span> <span class="n">TLightWeightEvent</span><span class="p">.</span><span class="n">Create</span><span class="p">;</span>
<span class="k">if</span> <span class="n">TInterlocked</span><span class="p">.</span><span class="n">CompareExchange</span><span class="p"><</span><span class="n">TLightWeightEvent</span><span class="p">>(</span><span class="n">FValueReadyEvent</span><span class="p">,</span> <span class="n">LEvent</span><span class="p">,</span>
<span class="nb">nil</span><span class="p">)</span> <span class="p"><></span> <span class="nb">nil</span> <span class="k">then</span>
<span class="n">LEvent</span><span class="p">.</span><span class="n">Free</span><span class="p">;</span>
<span class="k">if</span> <span class="n">FResultSet</span> <span class="k">then</span> <span class="c1">//can't see this happening, but here just in case
</span> <span class="n">FValueReadyEvent</span><span class="p">.</span><span class="n">SetEvent</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">FValueReadyEvent</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">TFutureValue</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">SetValue</span><span class="p">(</span><span class="n">AResult</span><span class="p">:</span> <span class="n">T</span><span class="p">);</span>
<span class="k">begin</span>
<span class="c1">//raise exception if set twice!
</span>
<span class="n">FResult</span> <span class="p">:=</span> <span class="n">AResult</span><span class="p">;</span> <span class="c1">// no-one can read until we set the flag anyway, so no need for interlocked
</span> <span class="c1">// interchange, which is hard to do with generics anyway
</span> <span class="n">FResultSet</span> <span class="p">:=</span> <span class="nb">true</span><span class="p">;</span> <span class="c1">//don't think interlock exchange is needed on a boolean
</span> <span class="n">GetValueReadyEvent</span><span class="p">.</span><span class="n">SetEvent</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">function</span> <span class="n">TFutureValue</span><span class="p"><</span><span class="n">T</span><span class="p">>.</span><span class="n">Wait</span><span class="p">:</span> <span class="kt">boolean</span><span class="p">;</span>
<span class="k">begin</span>
<span class="k">if</span> <span class="n">FResultSet</span> <span class="k">then</span> <span class="c1">//set after value is set
</span> <span class="n">result</span> <span class="p">:=</span> <span class="nb">true</span>
<span class="k">else</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">ValueReadyEvent</span><span class="p">.</span><span class="n">WaitFor</span><span class="p">(</span><span class="n">INFINITE</span><span class="p">)</span> <span class="p"><></span> <span class="n">TWaitResult</span><span class="p">.</span><span class="n">wrTimeout</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>For simplified synchronization the assumption must be made that the value will only be set once. This allows us to use a simple boolean that we will set directly without an interlocked exchange. This boolean is used in a technique called quick-checked locking, which is a way of reduce the overhead of checking or waiting for a synchronization object. This technique is similar to double-checked locking used in the Singleton Pattern, but is contra-indicated in some languages since order of execution is not always guaranteed. In this case I believe it is safe, and it speeds up the <code class="language-plaintext highlighter-rouge">GetValue</code> routine when the value has already been delivered</p>
<p>You will notice that the value is set without an interlocked exhange. I believe this is safe with the assumption that we can write only once and can read only after the TEvent is set. I also believe that setting booleans are safe and do not require an interlock exchange. If you believe I am in error please comment below or create a pull request on the source code for this blog post series.</p>
<p>You can download the <a href="https://github.com/schellingerhout/active-object-delphi">source code</a>. Please comment or contribute.</p>Jasper SchellingerhoutIn this blog post I will continue to describe the process to develop a solution in Delphi using the Active Object Design Pattern. Specifically, we will see how Future Values are deliveredActive Object Design Pattern in Delphi (Part 4): The Servant and the Proxy2017-11-25T00:00:00+00:002017-11-25T00:00:00+00:00https://schellingerhout.github.io/design%20patterns/activeobject-pattern4<p>In this blog post I will continue to describe the process to develop a solution in Delphi using the Active Object Design Pattern. Specifically, we will see how the Proxy composes and uses a Scheduler and a Servant</p>
<!--more-->
<p>If you missed <a href="/design%20patterns/activeobject-pattern1/">Part 1: Method Requests</a>, <a href="/design%20patterns/activeobject-pattern2/">Part 2: The Scheduler</a> or <a href="/design%20patterns/activeobject-pattern3/">Part 3: Futures</a>. I would recommend that you read those first before proceeding.</p>
<p>In the fourth part of the deep dive into the Active Object pattern I want to look at how we put the pieces together to create a Proxy for a Servant object. First a few guidelines given by the pattern:</p>
<ul>
<li>The Servant does not do synchronization, it is unaware of the Scheduler and the Proxy</li>
<li>The Client should be able to use the Proxy in the same or similar way to the Servant (except for Futures)</li>
<li>The Client and Proxy operate in the Client’s thread, the Scheduler and Servant operate in the Scheduler’s thread</li>
</ul>
<p>The Servant is an ordinary object that holds the implementation or functions of the work and data that need to grouped together. If we are using the Active Object Pattern to retrofit an existing system, the Servant would be your current non-threaded object that you would want to be running in its own thread.</p>
<p>The Proxy presents the same or a very similar interface to the Client as the Servant would have. This allows us to retrofit our code to use an object that runs in its own thread without any special synchronization code in the Client. The Proxy typically internally constructs the Servant as well as the Scheduler and would enqueue methods intended for the servant</p>
<p>Here is an example of a Servant that provides methods <code class="language-plaintext highlighter-rouge">put_i</code>, <code class="language-plaintext highlighter-rouge">get_i</code>, <code class="language-plaintext highlighter-rouge">empty_i</code> and <code class="language-plaintext highlighter-rouge">full_i</code>. The naming was chosen to match the sample provided in <a href="http://www.cs.wustl.edu/~schmidt/PDF/Act-Obj.pdf">Lavender and Schmidt’s Paper</a>. The message type (here called <code class="language-plaintext highlighter-rouge">TMessage</code>) is immaterial to the demonstration. Assume that it is communication that gets placed or read from the queue.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TServant</span> <span class="p">=</span> <span class="k">class</span>
<span class="k">private</span>
<span class="c1">// Internal Queue representation, e.g., a
</span> <span class="c1">// circular array or a linked list, etc.
</span>
<span class="k">public</span>
<span class="k">constructor</span> <span class="n">Create</span><span class="p">(</span><span class="n">mqsize</span><span class="p">:</span> <span class="kt">cardinal</span><span class="p">);</span>
<span class="c1">// Message queue implementation operations.
</span> <span class="k">procedure</span> <span class="n">put_i</span><span class="p">(</span><span class="k">const</span> <span class="n">msg</span><span class="p">:</span> <span class="n">TMessage</span><span class="p">);</span>
<span class="k">function</span> <span class="n">get_i</span><span class="p">:</span> <span class="n">TMessage</span><span class="p">;</span>
<span class="k">function</span> <span class="n">empty_i</span><span class="p">:</span> <span class="kt">boolean</span><span class="p">;</span>
<span class="k">function</span> <span class="n">full_i</span><span class="p">:</span> <span class="kt">boolean</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>I won’t cover <code class="language-plaintext highlighter-rouge">TServant</code>’s implementation because it simply enqueues and dequeues in a simple queue. It has no synchronization knowledge and does not need to have any special code to be thread safe or deal with concurrency.</p>
<p>The proxy will present a similar interface as the Servant. I assume that the sample in the paper used <code class="language-plaintext highlighter-rouge">*_i</code> for methods of the Servant to distinguish those in the Proxy that are named without the <code class="language-plaintext highlighter-rouge">_i</code> (perhaps meaning “internal”). However, in a real example the Proxy may actually follow the “Decorator” pattern or implement the same interface(s) as the Servant.</p>
<p>The Proxy composes the Servant and the Scheduler as follows</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="n">TProxy</span> <span class="p">=</span> <span class="k">class</span>
<span class="k">private</span>
<span class="k">protected</span>
<span class="c1">// The Servant that implements the
</span> <span class="c1">// Active Object methods.
</span> <span class="n">FServant</span><span class="p">:</span> <span class="n">TServant</span><span class="p">;</span>
<span class="c1">// A scheduler for the Message Queue.
</span> <span class="n">FScheduler</span><span class="p">:</span> <span class="n">TActivationScheduler</span><span class="p">;</span>
<span class="k">public</span>
<span class="k">constructor</span> <span class="n">Create</span><span class="p">;</span> <span class="c1">// Also creates composed FServant and FScheduler
</span> <span class="k">destructor</span> <span class="n">Destroy</span><span class="p">;</span> <span class="k">override</span><span class="p">;</span>
<span class="k">procedure</span> <span class="n">put</span><span class="p">(</span><span class="k">const</span> <span class="n">msg</span><span class="p">:</span> <span class="n">TMessage</span><span class="p">);</span>
<span class="k">function</span> <span class="n">get</span> <span class="p">:</span> <span class="n">IFuture</span><span class="p"><</span><span class="n">TMessage</span><span class="p">>;</span>
<span class="c1">// empty and full may also need implementation
</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>And the implementation would look like the sample we saw in <a href="/design%20patterns/activeobject-pattern1/">Part 1: Method Requests</a>. Our <code class="language-plaintext highlighter-rouge">put</code> and <code class="language-plaintext highlighter-rouge">get</code> are converted to Method Requests and enqueued with the Scheduler.</p>
<figure class="highlight"><pre><code class="language-pascal" data-lang="pascal"><span class="k">procedure</span> <span class="n">TProxy</span><span class="p">.</span><span class="n">put</span><span class="p">(</span><span class="k">const</span> <span class="n">msg</span><span class="p">:</span> <span class="n">TMessage</span><span class="p">);</span>
<span class="k">var</span>
<span class="n">LMsg</span><span class="p">:</span> <span class="n">TMessage</span><span class="p">;</span>
<span class="k">begin</span>
<span class="n">LMsg</span> <span class="p">:=</span> <span class="n">msg</span><span class="p">;</span>
<span class="n">FScheduler</span><span class="p">.</span><span class="n">Equeue</span><span class="p">(</span>
<span class="n">TMethodRequest</span><span class="p">.</span><span class="n">Create</span><span class="p">(</span>
<span class="c1">// Call
</span> <span class="k">procedure</span>
<span class="k">begin</span>
<span class="n">FServant</span><span class="p">.</span><span class="n">put_i</span><span class="p">(</span><span class="n">LMsg</span><span class="p">);</span>
<span class="k">end</span><span class="p">,</span>
<span class="c1">// Optional Guard
</span> <span class="k">function</span> <span class="p">:</span> <span class="kt">boolean</span>
<span class="k">begin</span>
<span class="n">result</span> <span class="p">:=</span> <span class="k">not</span> <span class="n">FServant</span><span class="p">.</span><span class="n">full_i</span><span class="p">;</span>
<span class="k">end</span>
<span class="p">)</span>
<span class="p">);</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">function</span> <span class="n">TProxy</span><span class="p">.</span><span class="n">get</span><span class="p">:</span> <span class="n">IFuture</span><span class="p"><</span><span class="n">TMessage</span><span class="p">>;</span>
<span class="k">var</span>
<span class="n">LActiveFuture</span><span class="p">:</span> <span class="n">TFuture</span><span class="p"><</span><span class="n">TMessage</span><span class="p">>;</span>
<span class="k">begin</span>
<span class="n">LActiveFuture</span> <span class="p">:=</span> <span class="n">TFutureValue</span><span class="p"><</span><span class="n">TMessage</span><span class="p">>.</span><span class="n">Create</span><span class="p">;</span>
<span class="n">result</span> <span class="p">:=</span> <span class="n">LActiveFuture</span><span class="p">;</span>
<span class="n">FScheduler</span><span class="p">.</span><span class="n">Enqueue</span><span class="p">(</span>
<span class="n">TMethodRequest</span><span class="p">.</span><span class="n">Create</span><span class="p">(</span>
<span class="c1">// Call
</span> <span class="k">procedure</span>
<span class="k">begin</span>
<span class="n">LActiveFuture</span><span class="p">.</span><span class="n">SetValue</span><span class="p">(</span><span class="n">FServant</span><span class="p">.</span><span class="n">get_i</span><span class="p">);</span> <span class="c1">// closure over the future and servant
</span> <span class="k">end</span><span class="p">,</span>
<span class="c1">// Optional Guard
</span> <span class="k">function</span> <span class="p">:</span> <span class="kt">boolean</span>
<span class="k">begin</span>
<span class="n">result</span> <span class="p">:=</span> <span class="k">not</span> <span class="n">FServant</span><span class="p">.</span><span class="n">empty_i</span><span class="p">;</span>
<span class="k">end</span>
<span class="p">)</span>
<span class="p">);</span>
<span class="k">end</span><span class="p">;</span></code></pre></figure>
<p>This concludes the series on the Active Object pattern. I hope you found it useful. You can download the <a href="https://github.com/schellingerhout/active-object-delphi">source code</a>. Please comment or contribute.</p>Jasper SchellingerhoutIn this blog post I will continue to describe the process to develop a solution in Delphi using the Active Object Design Pattern. Specifically, we will see how the Proxy composes and uses a Scheduler and a Servant