<feed xmlns="http://www.w3.org/2005/Atom">
    <title>Whatever crossed my mind</title>
    <subtitle>Notes on software, algorithms, performance, and security by Ian Qvist.</subtitle>
    <id>https://genbox.github.io/</id>
    <link href="https://genbox.github.io/" />
    <link href="https://genbox.github.io/feed.xml" rel="self" />
    <updated>2026-11-20T00:00:00.0000000+01:00</updated>
    <author>
        <name>Ian Qvist</name>
        <uri>https://github.com/Genbox</uri>
    </author>
    <entry>
    <title>Hash functions part 3 – performance</title>
    <id>https://genbox.github.io/2026/11/hash-functions-part-3-performance.html</id>
    <link href="https://genbox.github.io/2026/11/hash-functions-part-3-performance.html" />
    <published>2026-11-20T00:00:00.0000000+01:00</published>
    <updated>2026-11-20T00:00:00.0000000+01:00</updated>
    <summary>Exploring hash function performance, algorithmic complexity, and the challenges of benchmarking.</summary>
    <category term="Algorithms" />
<category term="Performance" />
    <content type="html" xml:base="https://genbox.github.io/2026/11/hash-functions-part-3-performance.html">&lt;p&gt;In part 1 we introduced hash functions, and in part 2 we took a look at what security means for
hash functions. In this part, we will take a look at the performance characteristics of hash
functions.&lt;/p&gt;
&lt;p&gt;One of the most important aspects of hash functions is the performance. We want to map arbitrary
data to an integer representation, and we want it to be fast since we are going to use it as an
integral part of high-performance data structures. However, hash functions are notoriously hard
to measure, as a fast hash function on one system, might be the slowest hash on another system.
We will clarify that in just a minute.&lt;/p&gt;
&lt;h2 id=&quot;algorithmic-complexity&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Algorithmic Complexity&lt;a class=&quot;hash&quot; href=&quot;#algorithmic-complexity&quot; aria-label=&quot;Link to Algorithmic Complexity&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Most hash functions can be grouped into a handful of different algorithmic complexity
categories, as we started out doing in part 1. The relationship between the complexity and
performance is tight, and we will always be in a situation where the simple hash functions will
be able to outperform the more complex algorithms. This is simply due to the fact they need to
do less work for each byte, compared to more complex algorithms.&lt;/p&gt;
&lt;p&gt;Some algorithms might have a simple main loop made up of add, subtract, xor and shift
operations, which are all fast on modern CPUs, but then finish up with a heavy mixer function
for &lt;a href=&quot;https://en.wikipedia.org/wiki/Avalanche_effect&quot;&gt;avalanching&lt;/a&gt;. Suffice to say, that simpler
hash functions that don&apos;t do avalanching are always going to be a tiny bit faster, but also
have worse security characteristics.&lt;/p&gt;
&lt;h2 id=&quot;code-speed-vs.instructions&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;Code Speed vs. Instructions&lt;a class=&quot;hash&quot; href=&quot;#code-speed-vs.instructions&quot; aria-label=&quot;Link to Code Speed vs. Instructions&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;One of the most misunderstood concepts in computer science is code speed. Usually we observe
that 10 lines of code are faster than 100 lines of code, but actually, nothing is further from
the truth. Programming languages give us a way to build an abstraction to represent what we
need, which is almost always easier to work with than machine language at the instruction level.
Compilers nowadays are really clever, and they can actually optimize 100 lines of code to fewer
instructions than those 10 lines of code.&lt;/p&gt;
&lt;h3 id=&quot;instruction-speed&quot;&gt;Instruction Speed&lt;a class=&quot;hash&quot; href=&quot;#instruction-speed&quot; aria-label=&quot;Link to Instruction Speed&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Some instructions are faster than others, and it is directly tied to the inner workings of the
CPU and because of that, the speed varies a lot between different architectures. In this blog
post, we work in orders of magnitude instead of specific timings. If you need specific timings,
Agner Fog has an &lt;a href=&quot;http://agner.org/optimize/instruction_tables.pdf&quot;&gt;amazing list&lt;/a&gt; of timings for
each CPU architecture on his &lt;a href=&quot;http://agner.org/optimize/&quot;&gt;website&lt;/a&gt;. If we group the speeds into
3 groups, ordered by descending speeds:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Bitwise / addition / subtraction / shifts&lt;/li&gt;
&lt;li&gt;Multiplication&lt;/li&gt;
&lt;li&gt;Division&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;For simplicity, let&apos;s assume that a multiplication is 10x slower than additions, and divisions
are 10x slower than multiplications. It is not quite correct, but close enough for our purposes.&lt;/p&gt;
&lt;h3 id=&quot;compiler-optimization-modulus&quot;&gt;Compiler Optimization: Modulus&lt;a class=&quot;hash&quot; href=&quot;#compiler-optimization-modulus&quot; aria-label=&quot;Link to Compiler Optimization: Modulus&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;The modulus operator, often denoted as &lt;code&gt;%&lt;/code&gt; in programming languages, is very common in hash
functions. It returns the remainder of a division, which is usually used to trim a value to be
within a range. Let&apos;s have a look at an example:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 1&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt; a = &lt;span class=&quot;m&quot;&gt;10&lt;/span&gt; % &lt;span class=&quot;m&quot;&gt;3&lt;/span&gt;; &lt;span class=&quot;c1&quot;&gt;// 1&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt; b = &lt;span class=&quot;m&quot;&gt;11&lt;/span&gt; % &lt;span class=&quot;m&quot;&gt;3&lt;/span&gt;; &lt;span class=&quot;c1&quot;&gt;// 2&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt; c = &lt;span class=&quot;m&quot;&gt;12&lt;/span&gt; % &lt;span class=&quot;m&quot;&gt;3&lt;/span&gt;; &lt;span class=&quot;c1&quot;&gt;// 0&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt; d = &lt;span class=&quot;m&quot;&gt;13&lt;/span&gt; % &lt;span class=&quot;m&quot;&gt;3&lt;/span&gt;; &lt;span class=&quot;c1&quot;&gt;// 1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;For nonnegative values on the left, the results are within [0..2].&lt;/p&gt;
&lt;p&gt;The modulus operator is actually quite slow. It&apos;s equivalent to:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 2&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;c = a – b * (a / b)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;As you can see, it is a subtraction, a multiplication and a division operation, where the division is the
really expensive operation. For this reason, compilers often try to optimize it. One really neat
trick compilers do is to look for modulus that are powers of 2, and replace them with an &lt;em&gt;and&lt;/em&gt;
(&lt;code&gt;&amp;amp;&lt;/code&gt;) operation instead. When &lt;code&gt;n&lt;/code&gt; is unsigned and &lt;code&gt;m&lt;/code&gt; is a nonzero power of 2, the following holds true:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 3&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;n % m == (n &amp;amp; (m - 1))
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;When the mask is known at compile time, this can reduce the calculation to a single AND instruction.&lt;/p&gt;
&lt;h2 id=&quot;putting-it-all-together&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;Putting It All Together&lt;a class=&quot;hash&quot; href=&quot;#putting-it-all-together&quot; aria-label=&quot;Link to Putting It All Together&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;In the phone directory, &lt;code&gt;Bob&lt;/code&gt; and &lt;code&gt;Dan&lt;/code&gt; both hash to 275. With eight buckets, &lt;code&gt;275 % 8&lt;/code&gt; and
&lt;code&gt;275 &amp;amp; 7&lt;/code&gt; both select bucket 3. The bit mask can make bucket selection cheaper, but the table
still has to compare names to return the right number, just as it did in
&lt;a href=&quot;/2026/11/hash-functions-part-1-introduction.html&quot;&gt;part 1&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;If anyone can submit names, a crowded bucket can make even a lookup for a missing name
expensive. A suitable keyed hash makes bucket placement harder to predict, and request limits
help bound the work a caller can trigger. Both matter for the networked directory in
&lt;a href=&quot;/2026/11/hash-functions-part-2-security.html&quot;&gt;part 2&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;When we benchmark the directory, we should measure the whole lookup: hashing the name,
selecting the bucket, and searching it. Try names that are present and names that are missing,
using key lengths and table sizes typical of the application. A cheaper hash or bucket
calculation only helps if that saving survives the rest of the lookup.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Qirk #2: Harmless enough</title>
    <id>https://genbox.github.io/2026/11/qirk-2.html</id>
    <link href="https://genbox.github.io/2026/11/qirk-2.html" />
    <published>2026-11-17T00:00:00.0000000+01:00</published>
    <updated>2026-11-17T00:00:00.0000000+01:00</updated>
    <summary>The second issue of Qirk, a comic about giving a robotic vacuum a little too much intelligence.</summary>
    
    <content type="html" xml:base="https://genbox.github.io/2026/11/qirk-2.html">&lt;p&gt;&lt;img src=&quot;/assets/images/posts/qirk-2.jpg?v=0cd261bf681cb2a9&quot; width=&quot;944&quot; height=&quot;1280&quot; decoding=&quot;async&quot; style=&quot;width: min(100%, 720px); display: block; margin-inline: auto;&quot; alt=&quot;Four-panel comic. A person says, &amp;quot;I just upgraded my robotic vacuum with AI.&amp;quot; Another person asks, &amp;quot;Are you not concerned about surveillance in your home?&amp;quot; The owner replies, &amp;quot;It seems harmless enough,&amp;quot; while the vacuum beeps. In the final panel, the vacuum climbs the wall, still beeping, and the owner says, &amp;quot;Okay, I might disable the AI.&amp;quot;&quot;&gt;&lt;/p&gt;
&lt;p style=&quot;text-align: center;&quot;&gt;&lt;em&gt;Taking home surveillance to a whole new level.&lt;/em&gt;&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Hash functions part 2 – security</title>
    <id>https://genbox.github.io/2026/11/hash-functions-part-2-security.html</id>
    <link href="https://genbox.github.io/2026/11/hash-functions-part-2-security.html" />
    <published>2026-11-13T00:00:00.0000000+01:00</published>
    <updated>2026-11-13T00:00:00.0000000+01:00</updated>
    <summary>How attackers turn predictable bucket collisions into hash flooding, and how secret keys and table design limit the damage.</summary>
    <category term="Algorithms" />
<category term="Security" />
    <content type="html" xml:base="https://genbox.github.io/2026/11/hash-functions-part-2-security.html">&lt;p&gt;In &lt;a href=&quot;/2026/11/hash-functions-part-1-introduction.html&quot;&gt;part 1&lt;/a&gt;, we built a phone directory using a
hash table. We used a deliberately weak hash function, resolved collisions by comparing names,
and saw why a matching hash does not prove equality.&lt;/p&gt;
&lt;p&gt;Now suppose anyone can submit names to our directory over the network. An attacker can choose
names specifically to make the table slow. The directory might still return the correct phone
number, but spend so much time searching that it cannot serve other users.&lt;/p&gt;
&lt;p&gt;This is &lt;strong&gt;hash flooding&lt;/strong&gt;. Understanding it requires separating two kinds of collision and
asking what the attacker can predict.&lt;/p&gt;
&lt;h2 id=&quot;hash-collisions-and-bucket-collisions&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Hash collisions and bucket collisions&lt;a class=&quot;hash&quot; href=&quot;#hash-collisions-and-bucket-collisions&quot; aria-label=&quot;Link to Hash collisions and bucket collisions&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A &lt;strong&gt;hash collision&lt;/strong&gt; occurs when two distinct inputs produce the same hash value:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 1&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;a != b
h(a) == h(b)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;For arbitrary-length inputs and a fixed-size output, collisions are unavoidable. There are more
possible inputs than outputs. A cryptographic hash aims to make finding collisions computationally
infeasible; it does not eliminate them.&lt;/p&gt;
&lt;p&gt;A &lt;strong&gt;bucket collision&lt;/strong&gt; occurs when two inputs map to the same bucket. Their full hash values can
be different. With eight buckets, our directory uses:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 2&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;bucket(name) = h(name) % 8
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;Using &lt;code&gt;SumCodeUnits&lt;/code&gt; from part 1, which adds the UTF-16 code units into an unsigned 32-bit value:&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 1&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;Name&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Hash value&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Bucket (% 8)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Bob&lt;/td&gt;
&lt;td&gt;66 + 111 + 98 = 275&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dan&lt;/td&gt;
&lt;td&gt;68 + 97 + 110 = 275&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Eve&lt;/td&gt;
&lt;td&gt;69 + 118 + 101 = 288&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ian&lt;/td&gt;
&lt;td&gt;73 + 97 + 110 = 280&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;p&gt;&lt;code&gt;Bob&lt;/code&gt; and &lt;code&gt;Dan&lt;/code&gt; collide in both their hash values and their buckets. &lt;code&gt;Eve&lt;/code&gt; and &lt;code&gt;Ian&lt;/code&gt; only collide
in their buckets. Hash flooding can exploit either: the attacker needs many entries to follow
the same expensive search path, not necessarily to have identical full hashes.&lt;/p&gt;
&lt;p&gt;Collisions are therefore not automatically vulnerabilities. A correctly implemented table
handles them. The security problem is allowing an attacker to concentrate enough of them to
exhaust the application&apos;s resources.&lt;/p&gt;
&lt;h2 id=&quot;how-hash-flooding-becomes-expensive&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;How hash flooding becomes expensive&lt;a class=&quot;hash&quot; href=&quot;#how-hash-flooding-becomes-expensive&quot; aria-label=&quot;Link to How hash flooding becomes expensive&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Consider a table that resolves bucket collisions with a linked list. If an attacker submits
&lt;code&gt;n&lt;/code&gt; distinct names that all land in one bucket, looking for a missing name in that bucket can
require checking all &lt;code&gt;n&lt;/code&gt; entries. That lookup costs O(n), rather than the expected O(1) under
well-distributed input and a controlled load factor.&lt;/p&gt;
&lt;p&gt;Insertion depends on what the table does. Adding an entry at the head of a list can be constant
time, but a dictionary normally needs to check whether the key already exists. If each insertion
searches the growing chain, building it requires roughly:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 3&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;0 + 1 + 2 + ... + (n - 1) = n * (n - 1) / 2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;That is O(n²) total work. Cached full hashes may avoid some key comparisons, but the chain still
has to be traversed. Other table designs use different collision strategies, so these particular
costs describe this chained-table example rather than every hash table.&lt;/p&gt;
&lt;p&gt;The danger is amplified when a server automatically builds dictionaries from request parameters
or other user-controlled fields. The attacker supplies data; the server pays the insertion cost.
&lt;a href=&quot;https://peps.python.org/pep-0456/&quot;&gt;Python&apos;s PEP 456&lt;/a&gt; documents why predictable hashing and earlier
randomization attempts were insufficient protection against this class of attack. Microsoft&apos;s
&lt;a href=&quot;https://www.microsoft.com/en-us/msrc/blog/2011/12/more-information-about-the-december-2011-asp-net-vulnerability&quot;&gt;December 2011 ASP.NET vulnerability write-up&lt;/a&gt;
describes how specially crafted form submissions could trigger expensive hash-table insertions
and cause a denial of service.&lt;/p&gt;
&lt;h2 id=&quot;predictable-hashing-lets-attackers-prepare-offline&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;Predictable hashing lets attackers prepare offline&lt;a class=&quot;hash&quot; href=&quot;#predictable-hashing-lets-attackers-prepare-offline&quot; aria-label=&quot;Link to Predictable hashing lets attackers prepare offline&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Our sum function is particularly easy to manipulate: rearranging the letters of a name always
preserves its hash. An attacker can generate many distinct permutations without searching for
collisions at all.&lt;/p&gt;
&lt;p&gt;Even without such a shortcut, a public deterministic hash lets an attacker calculate bucket
indices locally. For a fixed table with &lt;code&gt;m&lt;/code&gt; buckets and uniform output, about one candidate in
&lt;code&gt;m&lt;/code&gt; lands in a chosen bucket. The attacker can keep those candidates and submit only the ones
that collide. Resizing can change bucket placement, but it does not separate inputs with
identical full hashes.&lt;/p&gt;
&lt;p&gt;Replacing the sum with a public, unkeyed cryptographic hash does not remove this opportunity.
The attacker can still evaluate the hash and filter candidates by bucket.&lt;/p&gt;
&lt;h2 id=&quot;a-seed-is-not-necessarily-a-secret-key&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;A seed is not necessarily a secret key&lt;a class=&quot;hash&quot; href=&quot;#a-seed-is-not-necessarily-a-secret-key&quot; aria-label=&quot;Link to A seed is not necessarily a secret key&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A seeded function takes an extra parameter that changes its output. This can help vary bucket
placement, but the existence of a seed parameter says little about resistance to an attacker.
A fixed public seed can be included in an offline search. A secret seed may still be too small
to resist guessing, recoverable from observations, or mixed in a way that preserves collisions.&lt;/p&gt;
&lt;p&gt;For example, adding a seed after hashing cannot separate an existing full-hash collision:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 4&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;h(a) == h(b)
(h(a) + seed) mod 2^32 == (h(b) + seed) mod 2^32
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;Mixing a seed throughout the computation can avoid that particular weakness, but does not by
itself establish cryptographic security. The distinction matters in practice: the
&lt;a href=&quot;https://github.com/google/highwayhash#defending-against-hash-flooding&quot;&gt;HighwayHash authors&apos; discussion of hash flooding&lt;/a&gt;
links attacks that construct collisions independent of the seed for particular CityHash and
MurmurHash variants. Those findings concern specific algorithms and variants, not every function
with a seed parameter.&lt;/p&gt;
&lt;h2 id=&quot;keyed-hashing-makes-placement-unpredictable&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;05&lt;/span&gt;Keyed hashing makes placement unpredictable&lt;a class=&quot;hash&quot; href=&quot;#keyed-hashing-makes-placement-unpredictable&quot; aria-label=&quot;Link to Keyed hashing makes placement unpredictable&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A pseudorandom function (PRF) takes a secret key and an input. For someone who does not know the
key, its outputs should be computationally indistinguishable from those of a random function.
The same key and input still produce the same output, so the table can repeat its calculations.&lt;/p&gt;
&lt;p&gt;SipHash was designed for short inputs and uses a secret key. In the standard SipHash-2-4 variant,
the key is 128 bits and the output is 64 bits. The
&lt;a href=&quot;https://www.kernel.org/doc/html/latest/security/siphash.html&quot;&gt;Linux kernel&apos;s SipHash documentation&lt;/a&gt;
shows generating a key from a cryptographically secure random source and using the result to
select a hash-table bucket. Keep the key secret and stable for the table&apos;s lifetime; changing it
requires rebuilding the table&apos;s placement.&lt;/p&gt;
&lt;h2 id=&quot;why-reducing-the-hash-to-a-bucket-still-works&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;06&lt;/span&gt;Why reducing the hash to a bucket still works&lt;a class=&quot;hash&quot; href=&quot;#why-reducing-the-hash-to-a-bucket-still-works&quot; aria-label=&quot;Link to Why reducing the hash to a bucket still works&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A table must reduce the hash to its bucket range. Conceptually, using a SipHash implementation
that returns an unsigned 64-bit value:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 5&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;c1&quot;&gt;// Conceptual API: secretKey is generated once for this table.&lt;/span&gt;
&lt;span class=&quot;k&quot;&gt;ulong&lt;/span&gt; hash = SipHash24(secretKey, nameBytes);
&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt; bucketIndex = (&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt;)(hash % (&lt;span class=&quot;k&quot;&gt;ulong&lt;/span&gt;)buckets.Length);
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;With four buckets, the index is between 0 and 3. Collisions are common, regardless of the hash&apos;s
width. That does not make the secret key useless.&lt;/p&gt;
&lt;p&gt;With a public hash, an attacker can predict which names share a bucket before sending them.
With a suitable keyed function and an unknown random key, the same offline calculation is
unavailable. A collision set prepared for one independently keyed table does not reliably
concentrate entries in another.&lt;/p&gt;
&lt;p&gt;The benefit is unpredictable placement, not collision-free placement. If the attacker cannot
observe which bucket a candidate reaches, submitting more random names normally spreads them
across the table instead of selecting only the troublesome ones.&lt;/p&gt;
&lt;p&gt;There are limits. Keyed hashing raises the cost of preparing an attack; it does not guarantee constant-time
operations or prevent every denial of service. The
&lt;a href=&quot;https://github.com/google/highwayhash#defending-against-hash-flooding&quot;&gt;HighwayHash discussion&lt;/a&gt;
also examines why the data structure and information exposed to the attacker matter.&lt;/p&gt;
&lt;h2 id=&quot;protecting-the-whole-operation&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;07&lt;/span&gt;Protecting the whole operation&lt;a class=&quot;hash&quot; href=&quot;#protecting-the-whole-operation&quot; aria-label=&quot;Link to Protecting the whole operation&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;For our networked directory, the hash function is one part of the defense:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Use a table implementation with documented protection for attacker-controlled keys. Any custom
hash or equality implementation must preserve the assumptions that protection relies on.&lt;/li&gt;
&lt;li&gt;When keyed hashing is used, generate its key securely and keep it secret. A predictable seed
or a public unkeyed digest does not provide the same protection.&lt;/li&gt;
&lt;li&gt;Bound the work a request can trigger: the number and length of names, the total input size,
and the amount of memory the directory can consume.&lt;/li&gt;
&lt;li&gt;Consider the collision strategy. A balanced tree within an overloaded bucket can bound search
costs when keys support a suitable ordering; resizing alone cannot fix identical full hashes.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Our table still needs to compare keys for equality, just as it did in part 1. Security adds a
further requirement: handling attacker-chosen inputs without allowing a small request to trigger
an unreasonable amount of work.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>When the .NET WebAssembly interpreter rotates by the wrong number of bits</title>
    <id>https://genbox.github.io/2026/11/mono-interpreter-conditional-rotation.html</id>
    <link href="https://genbox.github.io/2026/11/mono-interpreter-conditional-rotation.html" />
    <published>2026-11-10T20:00:00.0000000+01:00</published>
    <updated>2026-11-10T20:00:00.0000000+01:00</updated>
    <summary>Tracing a conditional rotation through C# IL and Mono to find a missing basic-block check, with a verified patch tested across 20 cases.</summary>
    <category term="C#" />
<category term="Algorithms" />
    <content type="html" xml:base="https://genbox.github.io/2026/11/mono-interpreter-conditional-rotation.html">&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; I reported this issue in &lt;a href=&quot;https://github.com/dotnet/runtime/issues/137279&quot;&gt;dotnet/runtime#137279&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;A conditional argument to &lt;code&gt;BitOperations.RotateLeft&lt;/code&gt; can produce an erroneous result in the .NET WebAssembly interpreter:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 1&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;ulong&lt;/span&gt; Rotate(&lt;span class=&quot;k&quot;&gt;ulong&lt;/span&gt; value, &lt;span class=&quot;k&quot;&gt;bool&lt;/span&gt; flag) =&amp;gt; BitOperations.RotateLeft(value, flag ? &lt;span class=&quot;m&quot;&gt;49&lt;/span&gt; : &lt;span class=&quot;m&quot;&gt;24&lt;/span&gt;);
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;With &lt;code&gt;flag&lt;/code&gt; set to &lt;code&gt;false&lt;/code&gt;, it should rotate by 24 bits. However, calling &lt;code&gt;Rotate(0x123456789ABCDEF0UL, false)&lt;/code&gt; gave these results:&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 1&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;Runtime&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Result&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Rotation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Desktop CoreCLR 10.0.14&lt;/td&gt;
&lt;td&gt;&lt;code&gt;789ABCDEF0123456&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;24 bits, correct&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Browser Mono interpreter 10.0.14&lt;/td&gt;
&lt;td&gt;&lt;code&gt;BDE02468ACF13579&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;49 bits, incorrect&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;p&gt;I also reproduced the issue with &lt;code&gt;RotateRight(ulong)&lt;/code&gt; and &lt;code&gt;RotateLeft(uint)&lt;/code&gt;: when a conditional expression supplied two
constant counts directly to the call, the interpreter used the true branch&apos;s constant for both branches. Woops!&lt;/p&gt;
&lt;p&gt;I built 20 test cases. The results were surprising:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;14/20 passed in the browser&lt;/li&gt;
&lt;li&gt;20/20 passed on desktop CoreCLR&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Disabling the jiterpreter, Mono&apos;s partial JIT for WebAssembly, did not help. Neither did &lt;code&gt;--optimize=-all&lt;/code&gt;.
The next step was to find where the choice between 24 and 49 disappeared.&lt;/p&gt;
&lt;h2 id=&quot;the-compiler-preserved-the-choice&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;The compiler preserved the choice&lt;a class=&quot;hash&quot; href=&quot;#the-compiler-preserved-the-choice&quot; aria-label=&quot;Link to The compiler preserved the choice&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The Release IL contained exactly the branches I expected:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 2&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;IL_0000: ldarg.0            // Push value
IL_0001: ldarg.1            // Push flag
IL_0002: brtrue.s IL_0008   // Pop flag; branch if true
IL_0004: ldc.i4.s 24        // False path: push count 24
IL_0006: br.s IL_000a       // Skip the load of 49; stack is [value, 24]
IL_0008: ldc.i4.s 49        // True path: push count 49; stack is [value, 49]
IL_000a: call uint64 System.Numerics.BitOperations::RotateLeft(uint64, int32)
IL_000f: ret                // Return the rotation result
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;The false path loads 24 and jumps to the call. The true path loads 49 and reaches the same call. Looks correct.&lt;/p&gt;
&lt;p&gt;To see what Mono did with this IL, I enabled &lt;code&gt;MONO_VERBOSE_METHOD=Repro:Conditional&lt;/code&gt; and ran the WebAssembly.
The method trace showed the mistake in Mono&apos;s intermediate representation:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 3&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;BB4:
IL_000a: rol.i8.imm [6 &amp;lt;- 2], 49
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;&lt;code&gt;rol.i8.imm&lt;/code&gt; rotates a 64-bit value by a count embedded in the instruction. Here, that count is 49, regardless of which branch ran. The choice was present in the IL, but Mono had replaced it with a constant while translating.&lt;/p&gt;
&lt;h2 id=&quot;an-optimization-gone-wrong&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;An optimization gone wrong&lt;a class=&quot;hash&quot; href=&quot;#an-optimization-gone-wrong&quot; aria-label=&quot;Link to An optimization gone wrong&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Mono&apos;s &lt;a href=&quot;https://github.com/dotnet/runtime/blob/v10.0.14/src/mono/mono/mini/interp/transform.c#L2720-L2742&quot;&gt;rotation intrinsic handling in &lt;code&gt;transform.c&lt;/code&gt;&lt;/a&gt;
looks at &lt;code&gt;td-&amp;gt;last_ins&lt;/code&gt;, the last instruction it emitted. If that instruction loads an integer constant, Mono can use an immediate rotation instead of a normal method call. It does not check whether that instruction belonged to the current basic block.&lt;/p&gt;
&lt;p&gt;When Mono processed the join, its global last-instruction pointer still referred to the load of 49 in the preceding block. The intrinsic handling treated that constant as the count for the call, even though another path could arrive with 24.&lt;/p&gt;
&lt;p&gt;The defect dates back to
&lt;a href=&quot;https://github.com/dotnet/runtime/commit/6ddb03f8e8c1876fa14f0289a42c128dd95e08a4&quot;&gt;&lt;code&gt;6ddb03f8e8&lt;/code&gt;&lt;/a&gt;, &lt;strong&gt;Add intrinsics for BitOperations&lt;/strong&gt;, merged on &lt;strong&gt;14 March 2023&lt;/strong&gt; through
&lt;a href=&quot;https://github.com/dotnet/runtime/pull/82827&quot;&gt;PR #82827&lt;/a&gt;.
That commit introduced the immediate rotation instructions and the unchecked use of &lt;code&gt;td-&amp;gt;last_ins&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id=&quot;the-missing-check&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;The missing check&lt;a class=&quot;hash&quot; href=&quot;#the-missing-check&quot; aria-label=&quot;Link to The missing check&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The fix is simple. Add one condition before Mono accepts a constant rotation count:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #7CB7FF&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C code example 4&quot;&gt;&lt;code class=&quot;language-c&quot;&gt;&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; (td-&amp;gt;last_ins == td-&amp;gt;cbb-&amp;gt;last_ins) {
    &lt;span class=&quot;c1&quot;&gt;// Code here&lt;/span&gt;
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;It checks if the last emitted instruction is also the last instruction in the current basic block. Nearby &lt;a href=&quot;https://github.com/dotnet/runtime/blob/v10.0.14/src/mono/mono/mini/interp/transform.c#L6577-L6590&quot;&gt;conversion optimizations use the same guard&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The guard covers both rotation directions (Left/Right) and both integer widths (uint/ulong). I patched mono with the fix, and my 20 tests all passed.&lt;/p&gt;
&lt;p&gt;A simple workaround until the bug is fixed, is to simply pull out the branching before the call:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 5&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;static&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;ulong&lt;/span&gt; Rotate(&lt;span class=&quot;k&quot;&gt;ulong&lt;/span&gt; value, &lt;span class=&quot;k&quot;&gt;bool&lt;/span&gt; flag)
{
    &lt;span class=&quot;k&quot;&gt;int&lt;/span&gt; count = flag ? &lt;span class=&quot;m&quot;&gt;49&lt;/span&gt; : &lt;span class=&quot;m&quot;&gt;24&lt;/span&gt;;
    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; BitOperations.RotateLeft(value, count);
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
</content>
</entry>
<entry>
    <title>Qirk #1: Problem solved</title>
    <id>https://genbox.github.io/2026/11/qirk-1.html</id>
    <link href="https://genbox.github.io/2026/11/qirk-1.html" />
    <published>2026-11-09T00:00:00.0000000+01:00</published>
    <updated>2026-11-09T00:00:00.0000000+01:00</updated>
    <summary>The first issue of Qirk, a comic about fixing the problem in front of you.</summary>
    
    <content type="html" xml:base="https://genbox.github.io/2026/11/qirk-1.html">&lt;p&gt;&lt;img src=&quot;/assets/images/posts/qirk-1.png?v=aa98b3b87ad4b277&quot; width=&quot;1024&quot; height=&quot;1123&quot; decoding=&quot;async&quot; style=&quot;width: min(100%, 720px); display: block; margin-inline: auto;&quot; alt=&quot;Four-panel comic. A person notices rot at the base of a fence post and says, &amp;quot;Oh no, I better take care of this.&amp;quot; They check their phone, then admire the repaired post: &amp;quot;Good thing I noticed the problem!&amp;quot; A wider view reveals that many of the other fence posts are still rotting.&quot;&gt;&lt;/p&gt;
&lt;p style=&quot;text-align: center;&quot;&gt;&lt;em&gt;One problem solved. Perspective pending.&lt;/em&gt;&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Hash functions part 1 – introduction</title>
    <id>https://genbox.github.io/2026/11/hash-functions-part-1-introduction.html</id>
    <link href="https://genbox.github.io/2026/11/hash-functions-part-1-introduction.html" />
    <published>2026-11-03T00:00:00.0000000+01:00</published>
    <updated>2026-11-03T00:00:00.0000000+01:00</updated>
    <summary>How hash functions turn keys into hash values, how hash tables use them, and why a matching hash does not prove equality.</summary>
    <category term="Algorithms" />
    <content type="html" xml:base="https://genbox.github.io/2026/11/hash-functions-part-1-introduction.html">&lt;p&gt;Suppose we are building a phone directory. Given a name, we want to find its phone number. We
could store names and numbers in a list and check each name until we find a match. That works,
but as the directory grows, so does the amount of searching.&lt;/p&gt;
&lt;p&gt;A hash table gives us another option: use the name to calculate where to start looking. The
calculation is performed by a &lt;strong&gt;hash function&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Hash functions also appear in file verification, caches, Bloom filters, and cryptography. Those
applications have different requirements, but before choosing an algorithm, we need to understand
what a hash value tells us, and what it does not.&lt;/p&gt;
&lt;p&gt;In this first part, we will use a small phone directory and a deliberately weak hash function.
In &lt;a href=&quot;/2026/11/hash-functions-part-2-security.html&quot;&gt;part 2&lt;/a&gt;, we will examine what changes when an attacker chooses the inputs.&lt;/p&gt;
&lt;h2 id=&quot;from-an-input-to-a-hash-value&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;From an input to a hash value&lt;a class=&quot;hash&quot; href=&quot;#from-an-input-to-a-hash-value&quot; aria-label=&quot;Link to From an input to a hash value&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A hash function maps an input to a value in an output space. For the fixed-width hashes we will
focus on here, that output is often a 32-bit, 64-bit, or larger value, regardless of the input&apos;s
length. We call it the &lt;strong&gt;hash value&lt;/strong&gt;, or simply the &lt;strong&gt;hash&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;We can write the operation as &lt;code&gt;h(input) = hash&lt;/code&gt;. The input might be a name, a sequence of bytes,
or the fields of an object. The important property is determinism: with the same input and the
same algorithm and parameters, we get the same result.&lt;/p&gt;
&lt;p&gt;That qualification about parameters matters. A hash function can take a seed or a key, and
changing it can change the result. Determinism does not mean every application must use the same
hash for the same name.&lt;/p&gt;
&lt;p&gt;A hash is also not necessarily an identifier. If two inputs have the same hash, we cannot assume
they are the same input. We will see why shortly.&lt;/p&gt;
&lt;h2 id=&quot;a-hash-function-we-can-calculate-by-hand&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;A hash function we can calculate by hand&lt;a class=&quot;hash&quot; href=&quot;#a-hash-function-we-can-calculate-by-hand&quot; aria-label=&quot;Link to A hash function we can calculate by hand&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;For our directory, let&apos;s add up the numeric values in each name:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 1&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;static&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;uint&lt;/span&gt; SumCodeUnits(&lt;span class=&quot;k&quot;&gt;string&lt;/span&gt; value)
{
    &lt;span class=&quot;k&quot;&gt;uint&lt;/span&gt; hash = &lt;span class=&quot;m&quot;&gt;0&lt;/span&gt;;

    &lt;span class=&quot;k&quot;&gt;foreach&lt;/span&gt; (&lt;span class=&quot;k&quot;&gt;char&lt;/span&gt; c &lt;span class=&quot;k&quot;&gt;in&lt;/span&gt; value)
    {
        hash = &lt;span class=&quot;k&quot;&gt;unchecked&lt;/span&gt;(hash + c);
    }

    &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; hash;
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;In C#, a &lt;a href=&quot;https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/builtin-types/char&quot;&gt;&lt;code&gt;char&lt;/code&gt; is a UTF-16 code unit&lt;/a&gt;.
It does not always represent a complete character. Our examples use simple Latin letters, so we
can calculate their sums without dealing with surrogate pairs. &lt;code&gt;unchecked&lt;/code&gt; makes overflow wrap
around within the 32-bit unsigned result, even if overflow checking is enabled elsewhere.&lt;/p&gt;
&lt;p&gt;For the name &lt;code&gt;Ada&lt;/code&gt;, the calculation is:&lt;/p&gt;
&lt;figure class=&quot;code&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Text&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Text code example 2&quot;&gt;&lt;code class=&quot;language-text&quot;&gt;&apos;A&apos; = 65
&apos;d&apos; = 100
&apos;a&apos; = 97

h(&quot;Ada&quot;) = 65 + 100 + 97 = 262
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;This is a hash function, but it is a poor choice for a real hash table. Rearranging the same
letters never changes the sum: &lt;code&gt;Ada&lt;/code&gt; and &lt;code&gt;Aad&lt;/code&gt; both produce 262. We are using it because every
step is visible. A production hash function needs to distribute the inputs we expect much more
effectively.&lt;/p&gt;
&lt;h2 id=&quot;from-a-hash-value-to-a-bucket&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;From a hash value to a bucket&lt;a class=&quot;hash&quot; href=&quot;#from-a-hash-value-to-a-bucket&quot; aria-label=&quot;Link to From a hash value to a bucket&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Our function returns a &lt;code&gt;uint&lt;/code&gt;, but we do not want an array with over four billion entries.
Let&apos;s start with just eight &lt;strong&gt;buckets&lt;/strong&gt; instead. A bucket is a place where the table can store
entries. To select one, we take the remainder after dividing the hash value by eight:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 3&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;const&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;uint&lt;/span&gt; bucketCount = &lt;span class=&quot;m&quot;&gt;8&lt;/span&gt;;
&lt;span class=&quot;k&quot;&gt;uint&lt;/span&gt; hash = SumCodeUnits(&lt;span class=&quot;s&quot;&gt;&quot;Ada&quot;&lt;/span&gt;);
&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt; bucketIndex = (&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt;)(hash % bucketCount);

Console.WriteLine(bucketIndex); &lt;span class=&quot;c1&quot;&gt;// 6&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;The remainder is between 0 and 7, so it is a valid array index. Here is what happens to five names:&lt;/p&gt;
&lt;figure class=&quot;diagram diagram-panel hash-example&quot; role=&quot;img&quot; aria-label=&quot;Names are hashed and then reduced to eight buckets: Ada has hash 262 and bucket 6; Bob has hash 275 and bucket 3; Dan has hash 275 and bucket 3; Eve has hash 288 and bucket 0; Ian has hash 280 and bucket 0.&quot;&gt;
    &lt;div class=&quot;hash-example-grid&quot; aria-hidden=&quot;true&quot;&gt;
        &lt;span class=&quot;hash-example-heading&quot;&gt;Name&lt;/span&gt;
        &lt;span class=&quot;hash-example-heading&quot;&gt;Hash value&lt;/span&gt;
        &lt;span class=&quot;hash-example-heading&quot;&gt;Bucket (% 8)&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;Ada&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;262&lt;/span&gt;
        &lt;span class=&quot;hash-example-value hash-example-bucket&quot;&gt;6&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;Bob&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;275&lt;/span&gt;
        &lt;span class=&quot;hash-example-value hash-example-bucket&quot;&gt;3&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;Dan&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;275&lt;/span&gt;
        &lt;span class=&quot;hash-example-value hash-example-bucket&quot;&gt;3&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;Eve&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;288&lt;/span&gt;
        &lt;span class=&quot;hash-example-value hash-example-bucket&quot;&gt;0&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;Ian&lt;/span&gt;
        &lt;span class=&quot;hash-example-value&quot;&gt;280&lt;/span&gt;
        &lt;span class=&quot;hash-example-value hash-example-bucket&quot;&gt;0&lt;/span&gt;
    &lt;/div&gt;
    &lt;figcaption aria-hidden=&quot;true&quot;&gt;Hashing produces a value. Reducing that value selects a bucket. These are two separate steps.&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Notice the two kinds of overlap. &lt;code&gt;Bob&lt;/code&gt; and &lt;code&gt;Dan&lt;/code&gt; have the same hash value: this is a &lt;strong&gt;hash
collision&lt;/strong&gt;. &lt;code&gt;Eve&lt;/code&gt; and &lt;code&gt;Ian&lt;/code&gt; have different hashes, but both land in bucket 0: this is a &lt;strong&gt;bucket
collision&lt;/strong&gt;. Making the array larger can separate some bucket collisions, but it cannot separate
identical hash values using the same remainder operation.&lt;/p&gt;
&lt;p&gt;The bucket index belongs to the table&apos;s layout. If we resize the table, an entry may need a new
bucket even though its hash has not changed.&lt;/p&gt;
&lt;h2 id=&quot;a-collision-does-not-mean-we-lost-an-entry&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;A collision does not mean we lost an entry&lt;a class=&quot;hash&quot; href=&quot;#a-collision-does-not-mean-we-lost-an-entry&quot; aria-label=&quot;Link to A collision does not mean we lost an entry&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Our table must account for multiple keys reaching the same bucket. One approach is &lt;strong&gt;chaining&lt;/strong&gt;:
each bucket holds a collection of entries. The occupied buckets might look like this:&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 1&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;Bucket&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Stored entries: key → value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;Eve → 555-0400; Ian → 555-0500&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Bob → 555-0200; Dan → 555-0300&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Ada → 555-0100&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;p&gt;The table stores the original keys as well as the phone numbers. To look up &lt;code&gt;Dan&lt;/code&gt;, we:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Calculate &lt;code&gt;h(&quot;Dan&quot;) = 275&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Calculate &lt;code&gt;275 % 8 = 3&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Inspect the entries in bucket 3.&lt;/li&gt;
&lt;li&gt;Compare the stored names with &lt;code&gt;Dan&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Return &lt;code&gt;555-0300&lt;/code&gt; when the key matches.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The hash narrows the search. &lt;strong&gt;The key comparison decides whether we found the right entry.&lt;/strong&gt;
Returning the first phone number in the bucket would give us Bob&apos;s number instead.&lt;/p&gt;
&lt;p&gt;Another family of techniques, called &lt;strong&gt;open addressing&lt;/strong&gt;, keeps entries in the table&apos;s array
and probes other positions when a position is occupied. The details differ, but the table still
needs a collision strategy and an equality check.&lt;/p&gt;
&lt;p&gt;In application code, we can let a collection handle these details:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 4&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;var&lt;/span&gt; directory = &lt;span class=&quot;k&quot;&gt;new&lt;/span&gt; Dictionary&amp;lt;&lt;span class=&quot;k&quot;&gt;string&lt;/span&gt;, &lt;span class=&quot;k&quot;&gt;string&lt;/span&gt;&amp;gt;(StringComparer.Ordinal)
{
    [&lt;span class=&quot;s&quot;&gt;&quot;Ada&quot;&lt;/span&gt;] = &lt;span class=&quot;s&quot;&gt;&quot;555-0100&quot;&lt;/span&gt;,
    [&lt;span class=&quot;s&quot;&gt;&quot;Bob&quot;&lt;/span&gt;] = &lt;span class=&quot;s&quot;&gt;&quot;555-0200&quot;&lt;/span&gt;,
    [&lt;span class=&quot;s&quot;&gt;&quot;Dan&quot;&lt;/span&gt;] = &lt;span class=&quot;s&quot;&gt;&quot;555-0300&quot;&lt;/span&gt;,
    [&lt;span class=&quot;s&quot;&gt;&quot;Eve&quot;&lt;/span&gt;] = &lt;span class=&quot;s&quot;&gt;&quot;555-0400&quot;&lt;/span&gt;,
    [&lt;span class=&quot;s&quot;&gt;&quot;Ian&quot;&lt;/span&gt;] = &lt;span class=&quot;s&quot;&gt;&quot;555-0500&quot;&lt;/span&gt;
};

&lt;span class=&quot;k&quot;&gt;if&lt;/span&gt; (directory.TryGetValue(&lt;span class=&quot;s&quot;&gt;&quot;Dan&quot;&lt;/span&gt;, &lt;span class=&quot;k&quot;&gt;out&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;string&lt;/span&gt;? phone))
{
    Console.WriteLine(phone); &lt;span class=&quot;c1&quot;&gt;// 555-0300&lt;/span&gt;
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;This dictionary uses its own hashing and storage implementation, not our sum function or our
eight-bucket layout. &lt;code&gt;StringComparer.Ordinal&lt;/code&gt; makes our intended case-sensitive string comparison
explicit. Microsoft&apos;s &lt;a href=&quot;https://learn.microsoft.com/en-us/dotnet/api/system.collections.generic.dictionary-2&quot;&gt;Dictionary documentation&lt;/a&gt;
describes the collection and its comparer support.&lt;/p&gt;
&lt;h2 id=&quot;why-collisions-are-unavoidable&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;05&lt;/span&gt;Why collisions are unavoidable&lt;a class=&quot;hash&quot; href=&quot;#why-collisions-are-unavoidable&quot; aria-label=&quot;Link to Why collisions are unavoidable&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A 32-bit hash has exactly 2³² possible output values. There are more than 2³² possible strings.
If we map all those strings into that output space, some different strings must share an output.
This follows from the pigeonhole principle: we have more inputs than places to put them.&lt;/p&gt;
&lt;p&gt;A larger output gives us more possible values, but it does not make a fixed-width hash unique
for every possible input. Even a cryptographic hash has collisions. Its security goals include
making certain kinds of attack computationally infeasible, rather than making collisions
mathematically impossible.&lt;/p&gt;
&lt;p&gt;For a hash table, collisions are an expected part of operation. What matters is how often they
occur for our workload and how the table handles them.&lt;/p&gt;
&lt;h2 id=&quot;what-makes-lookup-fast&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;06&lt;/span&gt;What makes lookup fast?&lt;a class=&quot;hash&quot; href=&quot;#what-makes-lookup-fast&quot; aria-label=&quot;Link to What makes lookup fast?&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;If the keys are spread reasonably evenly and the table grows as we add entries, each lookup
only needs to inspect a small number of candidates. Under suitable assumptions about hashing
and table occupancy, this gives expected O(1) lookup with respect to the number of stored entries.&lt;/p&gt;
&lt;p&gt;The qualification is important. If all keys land in the same bucket of a chained table, we can
end up scanning that bucket&apos;s entire collection: O(n) work. The table can still return correct
answers, but it loses the performance benefit we wanted.&lt;/p&gt;
&lt;p&gt;There is also a cost before we reach the bucket. Our sum function reads every UTF-16 code unit,
so hashing a name of length L takes O(L) time. Comparing strings can also depend on their
length. Saying that hash-table lookup is O(1) does not mean looking up a million-character key
costs the same as looking up &lt;code&gt;Ada&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;These are separate concerns: the cost of hashing a key, the distribution of keys among buckets,
and the work required to resolve collisions. We will need to measure all of them when we get
to performance.&lt;/p&gt;
&lt;h2 id=&quot;equality-comes-first&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;07&lt;/span&gt;Equality comes first&lt;a class=&quot;hash&quot; href=&quot;#equality-comes-first&quot; aria-label=&quot;Link to Equality comes first&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Hashing and equality must agree. If two keys are considered equal, their hashes must also be
equal under the comparer used by the table. The reverse is not required.&lt;/p&gt;
&lt;p&gt;Suppose we decide that &lt;code&gt;Ada&lt;/code&gt; and &lt;code&gt;ADA&lt;/code&gt; should refer to the same entry. Our sum function gives
them different values, so pairing it with case-insensitive equality would be a mistake. In C#,
we can choose &lt;code&gt;StringComparer.OrdinalIgnoreCase&lt;/code&gt; for the dictionary instead; the comparer
supplies compatible equality and hashing behavior.&lt;/p&gt;
&lt;p&gt;For the same reason, changing fields that participate in a key&apos;s hash or equality while that
key is stored in a hash table can make lookups fail. The entry is still located according to
the old key. Immutable keys make this much easier to reason about.&lt;/p&gt;
&lt;p&gt;An in-memory hash code is not automatically a persistent fingerprint, either. .NET does not
promise that a string&apos;s &lt;code&gt;GetHashCode()&lt;/code&gt; result remains stable across processes or runtime versions.
Do not save it as a durable identifier. See the &lt;a href=&quot;https://learn.microsoft.com/en-us/dotnet/api/system.string.gethashcode&quot;&gt;String.GetHashCode documentation&lt;/a&gt;
for the lifetime and equality guarantees.&lt;/p&gt;
&lt;h2 id=&quot;different-uses-different-requirements&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;08&lt;/span&gt;Different uses, different requirements&lt;a class=&quot;hash&quot; href=&quot;#different-uses-different-requirements&quot; aria-label=&quot;Link to Different uses, different requirements&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Our directory needs fast hashing, useful distribution, and a way to resolve collisions. Other
applications ask different questions:&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 2&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;Application&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;What the hash helps us do&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hash table&lt;/td&gt;
&lt;td&gt;Find candidate entries quickly, then compare their keys.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;File deduplication&lt;/td&gt;
&lt;td&gt;Identify candidate duplicates; compare file contents when exact equality is required.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bloom filter&lt;/td&gt;
&lt;td&gt;Select bit positions for approximate membership queries.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cryptographic hashing&lt;/td&gt;
&lt;td&gt;Produce a digest with defined resistance to attacks such as finding collisions or preimages.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;p&gt;Good distribution alone does not establish cryptographic security. A hash can work well on
ordinary data while being easy for an attacker to manipulate. Likewise, password storage has
additional requirements and uses dedicated password-hashing constructions; our directory example
does not tell us how to store passwords safely.&lt;/p&gt;
&lt;p&gt;For now, the useful distinction is between the hash function and the application using it.
The function produces a hash value. The application determines how to interpret that value,
what to do about collisions, and which guarantees it needs.&lt;/p&gt;
&lt;p&gt;In &lt;a href=&quot;/2026/11/hash-functions-part-2-security.html&quot;&gt;part 2&lt;/a&gt;, we will look at how attackers exploit predictable bucket placement and how keyed hashing helps protect a table.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>The history of gperf</title>
    <id>https://genbox.github.io/2025/03/the-history-of-gperf.html</id>
    <link href="https://genbox.github.io/2025/03/the-history-of-gperf.html" />
    <published>2025-03-18T21:14:00.0030000+00:00</published>
    <updated>2025-03-18T21:32:26.5680000+00:00</updated>
    <summary>Tracing the origins of gperf through historical documents and source code, from Perfect and its early authors to the GNU tool.</summary>
    <category term="Algorithms" />
<category term="Performance" />
    <content type="html" xml:base="https://genbox.github.io/2025/03/the-history-of-gperf.html">&lt;p&gt;In this post I&apos;ll document my research into the history of gperf. A copy of the historical documents
and source code referenced in this post can be &lt;a href=&quot;https://github.com/Genbox/gperf-archive&quot;&gt;found
here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;If you have worked with hashing you have probably used gperf. It is a tool you give a list of
keywords and outputs C/C++ code to look up if a keyword is contained within the list. The lookup
function is incredibly fast and looks like black magic.&lt;/p&gt;
&lt;p&gt;In 1979 Richard J. Cichelli wrote an article titled &lt;a href=&quot;https://dl.acm.org/doi/10.1145/358808.358813&quot;&gt;Minimal Perfect Hash Functions Made
Simple&lt;/a&gt;. In January 1980 it was published in
Communications of the ACM, Volume 23, Issue 1. Back then Richard worked as a Research Manager for
American Newspaper Publishers Association Research Institute, but was doing this as part of his own
company &lt;a href=&quot;https://www.newspapersystems.com/about-the-founders.html&quot;&gt;Software Consulting Services (SCS)&lt;/a&gt; which he ran with his wife Martha
Cichelli until 2021.&lt;/p&gt;
&lt;p&gt;Today, Richard&apos;s article is known as &lt;em&gt;Cichelli&apos;s Method&lt;/em&gt; and works by computing a hash based on the
first and last character of the input, as well as the length of the input, then using that hash with
an association table to generate a nearly-minimal perfect hash function. Cichelli describes two
disadvantages in his article:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;It requires that no two keys share length and the first and last characters&lt;/li&gt;
&lt;li&gt;For lists with more than about 45 items, segmentation into sublists may be necessary&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;These limitations will become important in the next paragraph.&lt;/p&gt;
&lt;p&gt;In December 1980, Gerhard Jaeschke and G. Osterburg from IBM Scientific Center in Heidelberg, West
Germany published an article in Communications of the ACM, Volume 23, Issue 12 titled &lt;a href=&quot;https://dl.acm.org/doi/10.1145/359038.3084011&quot;&gt;Technical
Response to: On Cichelli&apos;s Minimal Perfect Hash Functions
Method&lt;/a&gt;. In the article, they determine that there
exists an infinite number of keywords, on which Cichelli&apos;s perfect hash method (noted as Cichelli&apos;s
Condition or CD) does not work. As a response to this, Richard adds a third disadvantage:&lt;/p&gt;
&lt;ol start=&quot;3&quot;&gt;
&lt;li&gt;For certain contrived lists of keys, it may be impossible to determine in advance whether this
method will yield a minimal solution.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Note: Gerhard Jaeschke has himself published several articles pertaining to minimal perfect hash
functions which are quite interesting, but let&apos;s stay on the subject of gperf.&lt;/p&gt;
&lt;p&gt;In 1982 Charlie D. Havener wrote a tool called Perfect and in 1983 &lt;a href=&quot;https://www.tuhs.org/Usenet/comp.sources.unix/1983-October/003584.html&quot;&gt;posted
it&lt;/a&gt; in the Usenet
newsgroup comp.sources.unix. It is based on Cichelli&apos;s method, and is the first public
implementation, as the Pascal version written by Richard himself was never published. Charlie notes
that Perfect was used to write a Lexical Analyzer Data Structure and posted in Appendix B in the
article More on Minimal Perfect Hash Tables by Cook, Curtis R. and Oldehoeft, R. R.&lt;/p&gt;
&lt;p&gt;Back then Charlie worked with hardware and software design as a &lt;a href=&quot;https://ecommons.cornell.edu/bitstream/1813/28000/1/086_08.pdf&quot;&gt;senior principal
engineer&lt;/a&gt; at GenRad Inc. in the
component test division in Bolton, Massachusetts. Later in 1988 he &lt;a href=&quot;https://ecommons.cornell.edu/bitstream/1813/28039/1/090_07.pdf&quot;&gt;taught computer
science&lt;/a&gt; at Boston University.
Unfortunately, Charlie suffered from myelofibrosis (a rare blood cancer) and
&lt;a href=&quot;https://web.archive.org/web/20250318184140/https://www.legacy.com/us/obituaries/name/charles-havener-obituary?pid=178921677&quot;&gt;died&lt;/a&gt; November
10, 2012.&lt;/p&gt;
&lt;p&gt;In 1984 Keith
Bostic &lt;a href=&quot;https://www.usenetarchives.com/view.php?id=net.sources&amp;amp;mid=PDQyMzRAc2Vpc21vLlVVQ1A%2B&quot;&gt;posted&lt;/a&gt; an
improved version of Charlie&apos;s Perfect in the Usenet newsgroup net.sources. He highlights the
limitations of the original implementation as mentioned above. Keith needed to hash more keywords
than what was supported by Perfect, and therefore, rewrote Perfect to support his use case. He notes
the following improvements:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;No limit on the number of keywords&lt;/li&gt;
&lt;li&gt;Much faster&lt;/li&gt;
&lt;li&gt;No need to reorder the keywords&lt;/li&gt;
&lt;li&gt;The ability to specify which keys to hash&lt;/li&gt;
&lt;li&gt;It handles keywords that hash to identical values&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Instead of just taking the first and last characters, Keith&apos;s version allows the user to specify
which characters to hash.&lt;/p&gt;
&lt;p&gt;In January 1989 Douglas C. Schmidt wrote version 1.1 of gperf in C++ while he was at University of
California, Irvine, USA. As mentioned in the
&lt;a href=&quot;https://www.gnu.org/software/gperf/manual/gperf.html#Contributors&quot;&gt;manual&lt;/a&gt;, it was based on Keith&apos;s
version of Perfect. In May 1989 Douglas made a C version which was published on comp.sources.unix.&lt;/p&gt;
&lt;p&gt;For many years, gperf was part of the libg++ library and the sources can still be found in &lt;a href=&quot;https://ftp.tsukuba.wide.ad.jp/software/gcc/old-releases/libg++/&quot;&gt;old
releases&lt;/a&gt;. The earliest release I
could find was libg++-1.35.1. In August 2000, with the release of version 2.7, it was moved to be a
separate project.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Escape the crab mentality</title>
    <id>https://genbox.github.io/2019/10/escape-crab-mentality.html</id>
    <link href="https://genbox.github.io/2019/10/escape-crab-mentality.html" />
    <published>2019-10-08T19:44:00.0020000+00:00</published>
    <updated>2021-05-02T13:06:00.6210000+00:00</updated>
    <summary>Recognizing crab mentality in teams and communities, and considering how to escape patterns that hold others back.</summary>
    
    <content type="html" xml:base="https://genbox.github.io/2019/10/escape-crab-mentality.html">&lt;p&gt;There is a well-known concept within psychology called &lt;a href=&quot;https://en.wikipedia.org/wiki/Crab_mentality&quot;&gt;crab mentality&lt;/a&gt;, which is a way of thinking
best described by &quot;If I can&apos;t have it, neither can you&quot;. It is a mentality shared by millions around
the world, everywhere from small teams to local communities and all the way up to large enterprises
and countries.&lt;/p&gt;
&lt;p&gt;It is a way to survive when resources are scarce, as any single entity in the group who consumes too
much must be forced to live under the same constraints as everyone else. However, the crab
survivalist way of thinking can often transcend constraints on resources, and factors such as envy
can exacerbate the mentality into a selfish ego-centric paradigm, which limits other people&apos;s
potential to accelerate beyond their peers and become better.&lt;/p&gt;
&lt;p&gt;It is from this mentality we hear stories like, &quot;I want to leave my local community to attend
university, but our priest says I have to stay and help my parents&quot;. What we see here is that our
protagonist is not allowed to leave, as the priest - a local community authority figure - would
rather have them stay and help their parents than attend a university.&lt;/p&gt;
&lt;p&gt;The fact that the priest interferes in the affairs of our protagonist is rather odd, and from the
simplified description we started out with in the first paragraph, Crab mentality means &quot;the priest
can&apos;t go to the university, so he doesn&apos;t want anyone to go&quot;. That is not the case, but it is close!
The difference between envy and Crab mentality is that in the latter, our priest is not thinking of
himself; he is thinking of himself and on behalf of the whole community. He does not want our
protagonist to leave.&lt;/p&gt;
&lt;p&gt;You see, when someone wants to better themselves, they often realize that they can&apos;t become better
while constrained by their current situation, and the only solution is to look elsewhere for an open
mind and acceptance. However, at the same time, the local community will lose a member, which is
good at something, because they better themselves. It is kind of like a catch-22 situation where the
community doesn&apos;t want to lose somebody that is good at something, but the person that is good at it
feels like they are constrained by their community and therefore want to leave.&lt;/p&gt;
&lt;p&gt;If you think about it, you might know someone who is caught by this mentality, or you might be in
this situation yourself. Some societies even have this mentality &lt;a href=&quot;https://en.wikipedia.org/wiki/%C7%83Kung_people#Hunting_rituals&quot;&gt;ingrained into their
culture&lt;/a&gt; , and others have &lt;a href=&quot;https://en.wikipedia.org/wiki/Harrison_Bergeron#Plot&quot;&gt;written
satirical books&lt;/a&gt; to illustrate just how
ridiculous this behavior is.&lt;/p&gt;
&lt;p&gt;So if you are reading this: Don&apos;t be limited by your peers and don&apos;t accept your current constraints
if you can do better. Read what you can, research what you need, and learn to become a better person
through knowledge.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Strings from a security perspective</title>
    <id>https://genbox.github.io/2019/06/strings-from-security-perspective.html</id>
    <link href="https://genbox.github.io/2019/06/strings-from-security-perspective.html" />
    <published>2019-06-09T14:30:00.0010000+00:00</published>
    <updated>2026-10-31T16:14:26.0000000+00:00</updated>
    <summary>How .NET strings can expose secrets in memory, and the tradeoffs of SecureString, unmanaged memory, and the Data Protection API.</summary>
    <category term="Security" />
<category term="C#" />
    <content type="html" xml:base="https://genbox.github.io/2019/06/strings-from-security-perspective.html">&lt;p&gt;There are times when you have to store secrets in your application. You wish to do it securely, but
how exactly would you do that? It is more problematic than you think, and if you don&apos;t design your
application around certain concepts, you will end up with a security nightmare that is nearly
impossible to solve.&lt;/p&gt;
&lt;p&gt;Let&apos;s take a look at an example. Here we are using the popular S3 client from Amazon&apos;s official AWS
SDK for .NET:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 1&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;AWSCredentials cred = &lt;span class=&quot;k&quot;&gt;new&lt;/span&gt; BasicAWSCredentials(&lt;span class=&quot;s&quot;&gt;&quot;keyId&quot;&lt;/span&gt;, &lt;span class=&quot;s&quot;&gt;&quot;secretAccessKey&quot;&lt;/span&gt;);
&lt;span class=&quot;k&quot;&gt;using&lt;/span&gt; (AmazonS3Client client = &lt;span class=&quot;k&quot;&gt;new&lt;/span&gt; AmazonS3Client(cred))
{
    &lt;span class=&quot;c1&quot;&gt;//do something with client&lt;/span&gt;
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;The problem here is that you are forced to enter the secret access key as a string, and strings are
not designed around security best practices. In .NET, strings occupy garbage-collected storage.
Value types such as integers and DateTime can also reside inside managed objects and arrays. It is not the only problem
with strings, but let&apos;s take one issue at a time.&lt;/p&gt;
&lt;h2 id=&quot;strings-are-garbage-collected&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Strings are garbage (collected)&lt;a class=&quot;hash&quot; href=&quot;#strings-are-garbage-collected&quot; aria-label=&quot;Link to Strings are garbage (collected)&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;When you initialize a string like string myString = &quot;some value&quot; you are actually creating an
object, which is tracked by the garbage collector. This is a nifty feature as you don&apos;t have to
free() the string after use. However, it is a double-edged sword when it comes to security. The
string might be kept in memory for a &lt;a href=&quot;https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/fundamentals#what-happens-during-a-garbage-collection&quot;&gt;very long
time,&lt;/a&gt;
even after it has been used and is not required anymore.&lt;/p&gt;
&lt;p&gt;In the example above with S3, if a malicious actor has the ability to look into your application
memory, then he will get hold of the secret access key, and therefore direct access to download
files from your S3 bucket. This is a reoccurring issue for websites that utilize cloud services in
the backend, as access is governed by access control (access keys) alone.&lt;/p&gt;
&lt;p&gt;Security bugs that result in remote information disclosure are misused by attackers on a daily
basis, and sometimes a bug like &lt;a href=&quot;https://heartbleed.com/&quot;&gt;HeartBleed&lt;/a&gt; is discovered, that when
attackers exploit it, millions of applications are suddenly vulnerable to remote memory disclosure.&lt;/p&gt;
&lt;p&gt;Languages like Java, C#, VB.net and Go all have a garbage collector, so the issue described in this
section pertains to applications written in any of them.&lt;/p&gt;
&lt;h2 id=&quot;it-gets-worse&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;It gets worse...&lt;a class=&quot;hash&quot; href=&quot;#it-gets-worse&quot; aria-label=&quot;Link to It gets worse...&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The quick-thinking developer might say: &quot;&lt;em&gt;Just overwrite the string with another value to clear the
secret from memory!&lt;/em&gt;&quot;&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 2&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;string&lt;/span&gt; myString = &lt;span class=&quot;k&quot;&gt;null&lt;/span&gt;;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;A fallacious thought, which does not work.&lt;/p&gt;
&lt;p&gt;In C#, string literals are typically
&lt;a href=&quot;https://en.wikipedia.org/wiki/String_interning&quot;&gt;interned&lt;/a&gt;; runtime-created strings are generally
separate objects unless explicitly interned. String Interning is great for
performance and reducing memory usage, but it is a nightmare for security. In order to intern
strings, you have to design the string as an &lt;a href=&quot;https://en.wikipedia.org/wiki/Immutable_object&quot;&gt;immutable
object&lt;/a&gt;. Assigning another value or &lt;code&gt;null&lt;/code&gt; to a
string variable changes the reference; the original object’s characters remain in memory until
that storage is overwritten.&lt;/p&gt;
&lt;p&gt;The interned string pool retains references to shared strings, which can keep their contents alive
for the lifetime of the process. Other strings become eligible for collection once unreachable.&lt;/p&gt;
&lt;h2 id=&quot;before-it-gets-better&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;... Before it gets better&lt;a class=&quot;hash&quot; href=&quot;#before-it-gets-better&quot; aria-label=&quot;Link to ... Before it gets better&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;So what should we do?&lt;/p&gt;
&lt;p&gt;That&apos;s simple: &lt;strong&gt;never use strings for secrets&lt;/strong&gt;. At least not in garbage collected and string
interned programming languages.&lt;/p&gt;
&lt;p&gt;I&apos;ve seen many different solutions to the problem. Everything from pinning the value in memory and
overwriting it using unmanaged code to scanning the process memory for the string and nuking it.
While some of them might work, it is a lot of hacks and workarounds to a problem that shouldn&apos;t
exist in the first place.&lt;/p&gt;
&lt;p&gt;Simply do not use strings for secrets, and instead utilize the right tools for the right job. .NET
already has 2 solutions to help us out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://learn.microsoft.com/en-us/dotnet/api/system.security.securestring?view=netframework-4.8&quot;&gt;SecureString&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/introduction?view=aspnetcore-2.2&quot;&gt;Data Protection
framework&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I have not added AES encryption to the list as you should rely on existing robust data protection
frameworks instead. I&apos;ll go into detail on each solution below.&lt;/p&gt;
&lt;h2 id=&quot;securestring&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;SecureString&lt;a class=&quot;hash&quot; href=&quot;#securestring&quot; aria-label=&quot;Link to SecureString&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;SecureString is the most underutilized feature of .NET. It can provide in-memory encryption of your
application secrets on Windows; protection depends on the platform. It provides low-level
functionality and requires careful handling and disposal. Here is how you create a SecureString:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 3&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;byte&lt;/span&gt;[] key = { &lt;span class=&quot;m&quot;&gt;115&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;101&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;99&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;114&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;101&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;116&lt;/span&gt; }; &lt;span class=&quot;c1&quot;&gt;//&quot;secret&quot;&lt;/span&gt;
SecureString secure = &lt;span class=&quot;k&quot;&gt;new&lt;/span&gt; SecureString();
&lt;span class=&quot;k&quot;&gt;foreach&lt;/span&gt; (&lt;span class=&quot;k&quot;&gt;byte&lt;/span&gt; b &lt;span class=&quot;k&quot;&gt;in&lt;/span&gt; key)
{
    secure.AppendChar((&lt;span class=&quot;k&quot;&gt;char&lt;/span&gt;)b);
}

secure.MakeReadOnly();
Array.Clear(key, &lt;span class=&quot;m&quot;&gt;0&lt;/span&gt;, key.Length);
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;No strings in sight! Instead, we use a &lt;code&gt;byte&lt;/code&gt; array with the content &quot;secret&quot; instead. Since &lt;code&gt;byte&lt;/code&gt;
arrays are reference types, they are garbage collected and we have to clear the content after use.
They are not interned like strings, so we don&apos;t have to worry about any side effects of clearing it.&lt;/p&gt;
&lt;p&gt;The call to MakeReadOnly() ensures the SecureString can&apos;t be written to after the initialization -
another small security perk we get for free.&lt;/p&gt;
&lt;p&gt;Reading a SecureString exposes plaintext temporarily, so keep its lifetime short and release
unmanaged buffers after use. For example:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 4&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;private&lt;/span&gt; IEnumerable&amp;lt;&lt;span class=&quot;k&quot;&gt;char&lt;/span&gt;&amp;gt; ReadSecureString(SecureString secure)
{
    IntPtr ptr = IntPtr.Zero;
    &lt;span class=&quot;k&quot;&gt;try&lt;/span&gt;
    {
        ptr = Marshal.SecureStringToGlobalAllocUnicode(secure);
        &lt;span class=&quot;k&quot;&gt;for&lt;/span&gt; (&lt;span class=&quot;k&quot;&gt;int&lt;/span&gt; i = &lt;span class=&quot;m&quot;&gt;0&lt;/span&gt;; i &amp;lt; secure.Length; i++)
        {
            &lt;span class=&quot;k&quot;&gt;yield&lt;/span&gt; &lt;span class=&quot;k&quot;&gt;return&lt;/span&gt; (&lt;span class=&quot;k&quot;&gt;char&lt;/span&gt;)Marshal.ReadInt16(ptr, i * &lt;span class=&quot;m&quot;&gt;2&lt;/span&gt;);
        }
    }
    &lt;span class=&quot;k&quot;&gt;finally&lt;/span&gt;
    {
        Marshal.ZeroFreeGlobalAllocUnicode(ptr);
    }
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;The iterator exposes plaintext characters and retains an unmanaged plaintext buffer until it is
finished or disposed. Dispose the SecureString itself when you are done with it.&lt;/p&gt;
&lt;p&gt;On Windows, SecureString uses operating-system memory protection. Code running in the process can
request the plaintext, so its protection depends on the process and operating-system security.&lt;/p&gt;
&lt;h2 id=&quot;the-data-protection-framework&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;05&lt;/span&gt;The Data Protection framework&lt;a class=&quot;hash&quot; href=&quot;#the-data-protection-framework&quot; aria-label=&quot;Link to The Data Protection framework&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Microsoft also provides a NuGet package
called &lt;a href=&quot;https://www.nuget.org/packages/Microsoft.AspNetCore.DataProtection/&quot;&gt;Microsoft.AspNetCore.DataProtection&lt;/a&gt;.
Don&apos;t let the AspNetCore section of the name fool you; it can be used by any type of application,
not just ASP.NET Core.&lt;/p&gt;
&lt;p&gt;With this framework, you get the ability to protect and unprotect blobs of data, and much like
SecureString above, we can also &lt;a href=&quot;https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/implementation/key-encryption-at-rest?view=aspnetcore-2.2#windows-dpapi&quot;&gt;use
DPAPI&lt;/a&gt;
as part of the encryption process. However, in this example, I&apos;m going to use plain old key files
instead. Protect access to this directory and configure key encryption at rest for production use.&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #9B7CF2&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;C#&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;C# code example 5&quot;&gt;&lt;code class=&quot;language-csharp&quot;&gt;&lt;span class=&quot;k&quot;&gt;byte&lt;/span&gt;[] key = { &lt;span class=&quot;m&quot;&gt;115&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;101&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;99&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;114&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;101&lt;/span&gt;, &lt;span class=&quot;m&quot;&gt;116&lt;/span&gt; }; &lt;span class=&quot;c1&quot;&gt;//&quot;secret&quot;&lt;/span&gt;
IDataProtectionProvider dpp = DataProtectionProvider.Create(&lt;span class=&quot;k&quot;&gt;new&lt;/span&gt; DirectoryInfo(&lt;span class=&quot;s&quot;&gt;@&quot;C:\Keys\&quot;&lt;/span&gt;));
IDataProtector protector = dpp.CreateProtector(&lt;span class=&quot;s&quot;&gt;&quot;SecretsProtector&quot;&lt;/span&gt;);
&lt;span class=&quot;k&quot;&gt;byte&lt;/span&gt;[] encrypted = protector.Protect(key);
Array.Clear(key, &lt;span class=&quot;m&quot;&gt;0&lt;/span&gt;, key.Length);
&lt;span class=&quot;k&quot;&gt;byte&lt;/span&gt;[] notEncrypted = protector.Unprotect(encrypted);
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;The Data Protection framework generates a key file and puts it into
C:\Keys\, it then uses that key file to encrypt our key,
which we can then decrypt using the same protector. Since we are not using strings and we are clearing
the original key after encryption, we reduce its plaintext lifetime. Clear &lt;code&gt;notEncrypted&lt;/code&gt; after use
as well. The persisted key files also need protection through access controls and key encryption.&lt;/p&gt;
&lt;p&gt;Microsoft&apos;s Data Protection framework has a ton of nice features, including rotating keys,
configurable cryptographic algorithms, Azure Key Vault storage and many more. FIPS compliance
depends on the configured algorithms, providers and deployment. Instead of trying to
build your own secure storage based on AES, use something like Data Protection as a foundation.&lt;/p&gt;
&lt;h2 id=&quot;securestring-vs.data-protection&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;06&lt;/span&gt;SecureString vs. Data Protection&lt;a class=&quot;hash&quot; href=&quot;#securestring-vs.data-protection&quot; aria-label=&quot;Link to SecureString vs. Data Protection&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;So when should we use one over the other? That&apos;s a hard question, and it really comes down to your
requirements, but I can provide some examples from my own experience.&lt;/p&gt;
&lt;p&gt;Use SecureString when...&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;You are building a library that accepts SecureString and can handle its plaintext lifetime carefully&lt;/li&gt;
&lt;li&gt;You don&apos;t require persistent storage of the secret in a secure format&lt;/li&gt;
&lt;li&gt;You don&apos;t need to transmit the secret to someone else&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Update 2019-08-04&lt;/strong&gt;: Microsoft recommends alternative approaches for new development because
SecureString’s protection varies across platforms.
See the remark in &lt;a href=&quot;https://learn.microsoft.com/en-us/dotnet/api/system.security.securestring?view=netframework-4.8&quot;&gt;the
documentation&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Use Data Protection when...&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;You are in full control of the application&lt;/li&gt;
&lt;li&gt;You need to store or transmit the encryption keys&lt;/li&gt;
&lt;li&gt;You need configurable encryption that can be evaluated against requirements such as FIPS&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;SecureString is really just a primitive data type while Data Protection is a full framework, so in
most cases you would probably want to utilize Data Protection for storing secrets securely at-rest (such as an encrypted config file), and then get a value from that config file as a SecureString in
order to send it to a third-party library. Be sure to use the right tool for the right job.&lt;/p&gt;
&lt;h2 id=&quot;but.librarynamehere-requires-a-string&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;07&lt;/span&gt;But... [LibraryNameHere] requires a string&lt;a class=&quot;hash&quot; href=&quot;#but.librarynamehere-requires-a-string&quot; aria-label=&quot;Link to But... [LibraryNameHere] requires a string&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Yep, and that&apos;s the case with almost all libraries, including Amazon’s own S3 client we used as an
example in the beginning. They just don&apos;t know how to do it securely, so send them a link to this
article.&lt;/p&gt;
&lt;p&gt;The target here is to ensure we only keep the secrets in memory when needed, but always have a way
to get the original secrets if/when needed - just &lt;strong&gt;not&lt;/strong&gt; as a string.&lt;/p&gt;
&lt;p&gt;If a library requires a secret key as a string, then you have no choice. You have to give it a
string. But please do your part and report it back to the developer.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>FindDupes 1.1 released</title>
    <id>https://genbox.github.io/2019/05/finddupes-11-released.html</id>
    <link href="https://genbox.github.io/2019/05/finddupes-11-released.html" />
    <published>2019-05-18T18:05:00.0000000+00:00</published>
    <updated>2019-05-18T18:05:25.4330000+00:00</updated>
    <summary>FindDupes 1.1 speeds up duplicate-file detection by grouping files by size before hashing, with installation instructions.</summary>
    <category term="C#" />
    <content type="html" xml:base="https://genbox.github.io/2019/05/finddupes-11-released.html">&lt;p&gt;I&apos;ve released a new version of FindDupes. It now groups files by size before hashing, thereby
dramatically speeding up the deduplication process.&lt;br&gt;
If you have .NET Core installed, you can install the tool by running:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #6EE7B7&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Bash&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Bash code example 1&quot;&gt;&lt;code class=&quot;language-bash&quot;&gt;dotnet tool install --global FindDupes
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;If you want to look through the code or report any issues, head over
to &lt;a href=&quot;https://github.com/Genbox/FindDupes&quot;&gt;https://github.com/Genbox/FindDupes&lt;/a&gt;&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Beware of BrotliStream in .NET Core 2.1</title>
    <id>https://genbox.github.io/2018/05/beware-of-brotlistream-in-net-core-21.html</id>
    <link href="https://genbox.github.io/2018/05/beware-of-brotlistream-in-net-core-21.html" />
    <published>2018-05-12T21:35:00.0030000+00:00</published>
    <updated>2026-10-31T16:14:26.0000000+00:00</updated>
    <summary>Investigating the compression defaults of BrotliStream in .NET Core 2.1, their performance cost, and subsequent changes in .NET.</summary>
    <category term="C#" />
<category term="Performance" />
    <content type="html" xml:base="https://genbox.github.io/2018/05/beware-of-brotlistream-in-net-core-21.html">&lt;p&gt;I love compression algorithms and have built a fair share of my own for fun and profit. It is a
difficult task to build something that is efficient when it comes to the compromise between speed
and compression ratio, so when I saw Google&apos;s Brotli algorithm mentioned in the  &lt;a href=&quot;https://devblogs.microsoft.com/dotnet/announcing-net-core-2-1-rc-1/&quot;&gt;.NET Core 2.1 RC1
announcement&lt;/a&gt;, I
was ecstatic!&lt;/p&gt;
&lt;p&gt;It lasted about 15 minutes until I put the BrotliStream to a test, only to find out that it took
140x longer to compress a 518 MB file!&lt;/p&gt;
&lt;p&gt;Curious if this was a bug, I delved into the code and realized that the default compression level
&lt;a href=&quot;https://github.com/dotnet/corefx/blob/8f9cfa462dfc2ec5bdbbe2462693036861fb0199/src/System.IO.Compression.Brotli/src/System/IO/Compression/BrotliUtils.cs#L13&quot;&gt;is set to
11&lt;/a&gt;.
For those not familiar with Brotli, that is the highest level available, that prioritizes compression
ratio above speed.&lt;/p&gt;
&lt;p&gt;I &lt;a href=&quot;https://github.com/dotnet/runtime/issues/26097&quot;&gt;reported it as an issue&lt;/a&gt;, but Microsoft seems to
think that having the same configuration as Google&apos;s Brotli library is more important than a sane
default, so it is only a matter of time before tons of StackOverflow posts pop up complaining about
the speed of BrotliStream.&lt;/p&gt;
&lt;p&gt;If you want to use Brotli in your application (a wise choice! It is a good algorithm), be sure to
use an explicit quality setting with the .NET Core 2.1 implementation discussed here. It accepts
an integer cast to CompressionLevel (I would recommend a value of 6) like this:&lt;/p&gt;
&lt;p&gt;BrotliStream stream = new BrotliStream(sm, (CompressionLevel)6);&lt;/p&gt;
&lt;p&gt;That is a good compromise between speed and compression ratio for most data. This cast relies on
the .NET Core 2.1 implementation and should be checked when changing runtime versions.&lt;/p&gt;
&lt;p&gt;Update 2021-11-01: Microsoft decided to block user settings &lt;a href=&quot;https://github.com/dotnet/runtime/pull/57561&quot;&gt;in this merge
request&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Update 2022-05-01: Microsoft now &lt;a href=&quot;https://github.com/dotnet/runtime/pull/72266&quot;&gt;defaults to quality
4&lt;/a&gt;. It will be part of .NET 7.&lt;/p&gt;
&lt;p&gt;Update 2022-09-01: &lt;a href=&quot;https://devblogs.microsoft.com/dotnet/performance_improvements_in_net_7/#compression&quot;&gt;Microsoft acknowledges the
problem&lt;/a&gt;, but
stays with the decision of setting quality to 4.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Performance measurement mistakes in academic publications</title>
    <id>https://genbox.github.io/2018/01/performance-measurement-mistakes-in.html</id>
    <link href="https://genbox.github.io/2018/01/performance-measurement-mistakes-in.html" />
    <published>2018-01-07T16:02:00.0020000+00:00</published>
    <updated>2026-10-31T16:14:26.0000000+00:00</updated>
    <summary>Common mistakes in academic performance measurements, including hardware comparisons, parallelization, and asymptotic complexity.</summary>
    <category term="Performance" />
    <content type="html" xml:base="https://genbox.github.io/2018/01/performance-measurement-mistakes-in.html">&lt;p&gt;One of the fastest ways to get me to stop reading a paper is to make incorrect assumptions in the
hypothesis or rely on previous work that is not completely solid. There is no doubt that research in
the academic context is hard, and writing a paper about it is even harder, but the value of all this
hard work is diminished if you make a mistake.&lt;/p&gt;
&lt;p&gt;This time, I will focus on performance measurement mistakes in computer science papers. They come in
different flavours and are sometimes very subtle. Performance is paramount in algorithmics and some
researchers don&apos;t do their due diligence when trying to prove theirs is faster. Here are the 3 most
often occurring mistakes I see, in no particular order.&lt;/p&gt;
&lt;h2 id=&quot;performance-measurements-on-different-hardware&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Performance measurements on different hardware&lt;a class=&quot;hash&quot; href=&quot;#performance-measurements-on-different-hardware&quot; aria-label=&quot;Link to Performance measurements on different hardware&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;When measuring performance, it is paramount to do it under the right circumstances. Every so often I
come across a paper that states algorithm X is 4x faster than algorithm Y, but they measured it
using absolute numbers between two very different hardware configurations. Of course, if algorithm X
takes 10 ms to run, and Y takes 20 ms to run &lt;strong&gt;on the same hardware&lt;/strong&gt;, it might be faster. I see
this problem occurring a lot in the fields of cryptocurrency (GPU acceleration) or Machine Learning (GPU/TPU acceleration).&lt;/p&gt;
&lt;p&gt;Researchers should use asymptotic notation - also called Big O notation - to indicate the
algorithmic complexity of their algorithm. If your algorithm is asymptotically slower than another,
such as O(n^2) vs. O(n), then it is up to you to demonstrate under which conditions your algorithm
is actually faster. Whether it be parallelization, specialization on specific architectures or even
favoured by certain compiler optimizations, it is your responsibility to make sure it is illuminated
by the paper.&lt;/p&gt;
&lt;p&gt;If you want to demonstrate any of those specialized cases, you are measuring efficiency, which is
measured in cycles per operation. Calculate how fast your algorithm performs compared to others in
that and you will be much better off.&lt;/p&gt;
&lt;h2 id=&quot;parallelization-as-an-excuse-for-speed&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;Parallelization as an excuse for speed&lt;a class=&quot;hash&quot; href=&quot;#parallelization-as-an-excuse-for-speed&quot; aria-label=&quot;Link to Parallelization as an excuse for speed&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;As improvements in single-core performance slow, increasing core counts make parallelization
increasingly important.
Researchers today take advantage of the many cores in computers, which often leads them to make the
mistake of reporting speedups without accounting for the extra resources used.&lt;/p&gt;
&lt;p&gt;Speed is not the same as throughput. When we measure an algorithm, we measure the efficiency of the
execution of some data. Using two cores can increase throughput or reduce a single job’s elapsed
time; efficiency also depends on the resources used. I often see the same mistake when measuring
network bandwidth. For example:&lt;/p&gt;
&lt;p&gt;You have a 100 megabit/second network connection. That means 100.000.000 bits can flow through your
network card per second, and therefore your throughput is 100 megabits/second.&lt;/p&gt;
&lt;p&gt;However, in order for that to make sense, you need to send it somewhere. How fast your network card
can send the data out is the speed. Let&apos;s say you send 3 packets of 5 bits each, and it takes 1 millisecond between each packet:&lt;/p&gt;
&lt;p&gt;[data].[data].[data]&lt;/p&gt;
&lt;p&gt;If packets arrive once per millisecond, the resulting throughput is 5 bits/ms; packet latency is
a separate measurement. You decide
to add another network card so you can send data in parallel:&lt;/p&gt;
&lt;p&gt;[data].[data].[data]&lt;/p&gt;
&lt;p&gt;[data].[data].[data]&lt;/p&gt;
&lt;p&gt;You just doubled the aggregate throughput to 10 bits/ms, while each card still sends 5 bits/ms!&lt;/p&gt;
&lt;p&gt;In this analogy, network cards are CPU cores. When reporting a speedup from using more cores,
state the core count and compare elapsed time, throughput and resource efficiency separately.&lt;/p&gt;
&lt;h2 id=&quot;reduction-of-asymptotic-notation&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;Reduction of asymptotic notation&lt;a class=&quot;hash&quot; href=&quot;#reduction-of-asymptotic-notation&quot; aria-label=&quot;Link to Reduction of asymptotic notation&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;I&apos;ve read papers that state their algorithms are constant (O(1)) in running time. Their argument is
that they split the problem into a smaller set of problems, and each problem takes 1/n to solve, and
therefore the same as O(1).&lt;/p&gt;
&lt;p&gt;No. That is not how it works.&lt;/p&gt;
&lt;p&gt;Algorithmic complexity describes how resource use grows with input size under a specified model.
An algorithm can have different best-case, average-case and worst-case complexity.&lt;/p&gt;
&lt;p&gt;What you are really measuring is the algorithm’s ability to solve a particular problem, and even then
a timing for smaller inputs describes a different workload. Splitting a sorting problem into
smaller chunks still requires accounting for all chunks and any merging work. General sorting
requires at least linear work to process all elements in the usual sequential model.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Reducing the size of self-contained .NET Core applications</title>
    <id>https://genbox.github.io/2018/01/reducing-size-of-self-contained-net.html</id>
    <link href="https://genbox.github.io/2018/01/reducing-size-of-self-contained-net.html" />
    <published>2018-01-01T17:47:00.0000000+00:00</published>
    <updated>2018-01-01T18:08:19.5420000+00:00</updated>
    <summary>Reducing self-contained .NET Core application size using trimming, linking, and native compilation, with size comparisons.</summary>
    <category term="C#" />
<category term="Performance" />
    <content type="html" xml:base="https://genbox.github.io/2018/01/reducing-size-of-self-contained-net.html">&lt;p&gt;Just for note keeping, I&apos;ve written down some methods of reducing the size of a .NET Core
application. I thought others could use it as well, so here you go.&lt;/p&gt;
&lt;h2 id=&quot;original-size&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Original size&lt;a class=&quot;hash&quot; href=&quot;#original-size&quot; aria-label=&quot;Link to Original size&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;First off, let&apos;s see how much disk space a
&lt;a href=&quot;https://learn.microsoft.com/en-us/dotnet/core/deploying/&quot;&gt;self-contained&lt;/a&gt; &apos;hello world&apos; application
takes up.&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet new console&lt;br&gt;
&amp;gt; dotnet publish -r win-x86 -c release&lt;/p&gt;
&lt;p&gt;Size: 53.9 MB - &lt;strong&gt;yuck!&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id=&quot;trimming&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;Trimming&lt;a class=&quot;hash&quot; href=&quot;#trimming&quot; aria-label=&quot;Link to Trimming&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Microsoft has built &lt;a href=&quot;https://github.com/dotnet/standard/blob/release/2.0.0/Microsoft.Packaging.Tools.Trimming/docs/trimming.md&quot;&gt;a
tool&lt;/a&gt;
that finds unused assemblies and removes them from the distribution package. This is much needed
since the &apos;netcoreapp2.0&apos; profile is basically .NET Framework all over again, and it contains a
truckload of assemblies our little &apos;hello world&apos; application doesn&apos;t use.&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet new console&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet add package Microsoft.Packaging.Tools.Trimming -v 1.1.0-preview1-25818-01&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet publish -r win-x86 -c release /p:TrimUnusedDependencies=true&lt;/p&gt;
&lt;p&gt;Size: 15.8 MB&lt;/p&gt;
&lt;p&gt;Much better! Although, we have only removed unused assemblies - what about unused code?&lt;/p&gt;
&lt;h2 id=&quot;linking&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;Linking&lt;a class=&quot;hash&quot; href=&quot;#linking&quot; aria-label=&quot;Link to Linking&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The Mono team over at Xamarin has &lt;a href=&quot;https://github.com/dotnet/core/blob/f99c03dc245c7d903047da8149d4e81350bb27b8/samples/linker-instructions.md&quot;&gt;built a
linker&lt;/a&gt; that can remove
unused code from assemblies. If you only use List&amp;lt;T&amp;gt; from System.Collections.Generic.dll, why have
HashSet&amp;lt;T&amp;gt; and all the other classes? This has a huge impact on reflection obviously, so use with
care.&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet new console&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet new nuget&lt;/p&gt;
&lt;p&gt;Open nuget.config and add &amp;lt;add key=&quot;dotnet-core&quot;
value=&quot;https://dotnet.myget.org/F/dotnet-core/api/v3/index.json&quot; /&amp;gt;&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet add package ILLink.Tasks -v 0.1.4-preview-981901&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet publish -r win-x86 -c release&lt;/p&gt;
&lt;p&gt;Size: 22.9 MB&lt;/p&gt;
&lt;p&gt;Not as good as trimming it seems, but definitely better than doing nothing.&lt;/p&gt;
&lt;h2 id=&quot;net-native&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;.NET Native&lt;a class=&quot;hash&quot; href=&quot;#net-native&quot; aria-label=&quot;Link to .NET Native&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;In the world of C, we have a thing called &apos;static linking&apos; which means all assemblies get linked
into a single assembly. This is very nice for small applications or deployments that can&apos;t rely on
existing assemblies on the system. C# does not have this feature, but with .NET Native we get
something very close.&lt;/p&gt;
&lt;p&gt;.NET Native is actually a refactored runtime called CoreRT, which uses RyuJIT to create native
assemblies. Getting it to work is &lt;a href=&quot;https://github.com/dotnet/corert/blob/master/Documentation/how-to-build-and-run-ilcompiler-in-console-shell-prompt.md&quot;&gt;not as
easy&lt;/a&gt;
as trimming or linking since unfortunately Microsoft only supports UWP as a native platform. .NET
Core is therefore left out in the cold for now.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Download &lt;a href=&quot;https://cmake.org/download/&quot;&gt;CMake&lt;/a&gt; 3.8 or later. I
choose &lt;a href=&quot;https://cmake.org/files/v3.10/cmake-3.10.1-win64-x64.msi&quot;&gt;cmake-3.10.1-win64-x64.msi&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Make sure you have Visual Studio 2017 with C++ support installed&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Download the latest successful build. Check &lt;a href=&quot;https://ci.dot.net/job/dotnet_corert/job/master/job/debug_windows_nt/lastStableBuild/&quot;&gt;their
Jenkins&lt;/a&gt; for
commit hash.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Unpack the content to &lt;code&gt;c:\corert\&lt;/code&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Open &apos;x64 Native Tools Command Prompt for VS 2017&apos;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Now you should compile .NET Native:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #6EE7B7&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Batch&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Batch code example 1&quot;&gt;&lt;code class=&quot;language-bat&quot;&gt;cd c:\corert
build.bat release
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;Note that it takes a long time to download missing packages and compile the whole project.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Create a new .NET Core project and add .NET Native msbuild targets:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #6EE7B7&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Batch&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Batch code example 2&quot;&gt;&lt;code class=&quot;language-bat&quot;&gt;mkdir c:\test
cd c:\test
dotnet new console
set IlcPath=c:\corert\bin\Windows_NT.x64.Release
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Edit test.csproj and add the two lines just before &lt;code&gt;&amp;lt;/Project&amp;gt;&lt;/code&gt;:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #F78C6C&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;XML&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;XML code example 3&quot;&gt;&lt;code class=&quot;language-xml&quot;&gt;&lt;span class=&quot;o&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Import&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Project&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;na&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;$(MSBuildSDKsPath)\Microsoft.NET.Sdk\Sdk\Sdk.targets&lt;/span&gt;&lt;span class=&quot;na&quot;&gt;&quot;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&amp;gt;&lt;/span&gt;
&lt;span class=&quot;o&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span class=&quot;n&quot;&gt;Import&lt;/span&gt; &lt;span class=&quot;na&quot;&gt;Project&lt;/span&gt;&lt;span class=&quot;o&quot;&gt;=&lt;/span&gt;&lt;span class=&quot;na&quot;&gt;&quot;&lt;/span&gt;&lt;span class=&quot;s&quot;&gt;$(IlcPath)\build\Microsoft.NETCore.Native.targets&lt;/span&gt;&lt;span class=&quot;na&quot;&gt;&quot;&lt;/span&gt; &lt;span class=&quot;o&quot;&gt;/&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Then publish the project:&lt;/p&gt;
&lt;figure class=&quot;code&quot; style=&quot;--dot: #6EE7B7&quot;&gt;&lt;figcaption class=&quot;code-bar&quot;&gt;&lt;span class=&quot;code-lang&quot;&gt;Batch&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;highlight&quot;&gt;&lt;pre tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Batch code example 4&quot;&gt;&lt;code class=&quot;language-bat&quot;&gt;dotnet publish -r win-x64 -c release
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/figure&gt;
&lt;p&gt;Size: 3.95 MB&lt;/p&gt;
&lt;p&gt;Wow! Unused assemblies and unused code have been removed and everything is statically compiled into an
executable. Note that I used x64 and not x86 in the .NET Native build, as I&apos;m not sure .NET Native
supports x86 fully yet.&lt;/p&gt;
&lt;h2 id=&quot;bonus-trim-and-link&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;05&lt;/span&gt;Bonus: trim and link&lt;a class=&quot;hash&quot; href=&quot;#bonus-trim-and-link&quot; aria-label=&quot;Link to Bonus: trim and link&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;We can actually use both trimming and linking to reduce the size of the application further. It
seems that the trimmer is better at finding unused assemblies than the linker is, but if we do both
steps, the trimmer will remove assemblies and the linker will remove the dead code from the
remaining assemblies.&lt;/p&gt;
&lt;p&gt;Just add both steps together like this:&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet new console&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet new nuget&lt;/p&gt;
&lt;p&gt;Open nuget.config and add &amp;lt;add key=&quot;dotnet-core&quot;
value=&quot;https://dotnet.myget.org/F/dotnet-core/api/v3/index.json&quot; /&amp;gt;&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet add package ILLink.Tasks -v 0.1.4-preview-981901&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet add package Microsoft.Packaging.Tools.Trimming -v 1.1.0-preview1-25818-01&lt;/p&gt;
&lt;p&gt;&amp;gt; dotnet publish -r win-x86 -c release /p:TrimUnusedDependencies=true&lt;/p&gt;
&lt;p&gt;Size: 14.9 MB&lt;/p&gt;
&lt;p&gt;Final size is pretty much as low as it gets right now without native compilation.&lt;/p&gt;
&lt;h2 id=&quot;conclusion&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;06&lt;/span&gt;Conclusion&lt;a class=&quot;hash&quot; href=&quot;#conclusion&quot; aria-label=&quot;Link to Conclusion&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;We can easily reduce the size of a self-contained .NET Core application, we just have to add the right
packages and switches. Microsoft has changed the behaviour of the .NET Core project and publish
system several times now, which has made it pretty difficult to find the right documentation, so
hopefully, this small blog post will sort it out.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Testing 36,200 DNS servers</title>
    <id>https://genbox.github.io/2017/08/testing-36200-dns-servers.html</id>
    <link href="https://genbox.github.io/2017/08/testing-36200-dns-servers.html" />
    <published>2017-08-12T14:47:00.0020000+00:00</published>
    <updated>2017-08-12T14:47:43.4870000+00:00</updated>
    <summary>Testing 36,200 DNS servers for response times and reliability, with methodology, comparisons, and benchmark results.</summary>
    <category term="Networking" />
<category term="Security" />
    <content type="html" xml:base="https://genbox.github.io/2017/08/testing-36200-dns-servers.html">&lt;h2 id=&quot;introduction&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Introduction&lt;a class=&quot;hash&quot; href=&quot;#introduction&quot; aria-label=&quot;Link to Introduction&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;DNS is an important part of the Internet and the speed and security are paramount for a good
browsing experience. I thought it would be a good idea to scan the internet for DNS servers and test
every single one of them. However, the latency of a particular DNS server depends highly on the
distance and connection technology between you and the DNS server, and since I&apos;m geographically
located in Denmark, the results speed-wise only pertain to people located in/around Denmark.&lt;/p&gt;
&lt;h2 id=&quot;testing-methodology&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;Testing methodology&lt;a class=&quot;hash&quot; href=&quot;#testing-methodology&quot; aria-label=&quot;Link to Testing methodology&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;I scanned the IPv4 Internet using &lt;a href=&quot;https://nmap.org/&quot;&gt;NMap&lt;/a&gt; on port 53/UDP and stopped the scan
after a few hours. The results were 36200 DNS servers, some of which are owned by ISPs, companies
and a whole lot of private people. Since the IP scan was randomized, it should represent a good
sample.&lt;/p&gt;
&lt;p&gt;Almost all DNS servers have some sort of caching mechanism that makes sure requested DNS names are
kept for as long as the Time-To-Live (TTL) as defined by the domain owner. To ensure we don&apos;t just
test a bunch of cached entries, we need to test uncached domains as well.&lt;/p&gt;
&lt;p&gt;Reliability is also important, so we need to register when a query is dropped. The drop might be
from a router between us and the DNS server, or it might be the DNS server itself, so we can&apos;t
exactly measure the reliability of a particular server, but a high rate of dropped packets will
disqualify nevertheless.&lt;/p&gt;
&lt;p&gt;Security is also an important factor, and there are 2 main things to look for:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions&quot;&gt;DNSSEC&lt;/a&gt; validation support&lt;/li&gt;
&lt;li&gt;&lt;a href=&quot;https://en.wikipedia.org/wiki/DNS_spoofing&quot;&gt;DNS Spoofing&lt;/a&gt; hardening&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;popular-servers&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;Popular servers&lt;a class=&quot;hash&quot; href=&quot;#popular-servers&quot; aria-label=&quot;Link to Popular servers&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;There are a bunch of &quot;known good&quot; DNS servers out there that have a good reputation for being stable
and fast. I added them to the list, just to make sure we test them as well.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Google: 8.8.8.8&lt;/li&gt;
&lt;li&gt;OpenDNS: 208.67.222.222&lt;/li&gt;
&lt;li&gt;Level3: 4.2.2.1&lt;/li&gt;
&lt;li&gt;UltraDNS: 156.154.70.1&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Note that I did not add secondary clusters for each of the nameservers.&lt;/p&gt;
&lt;h2 id=&quot;performing-the-test&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;Performing the test&lt;a class=&quot;hash&quot; href=&quot;#performing-the-test&quot; aria-label=&quot;Link to Performing the test&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;GRC has a &lt;a href=&quot;https://www.grc.com/dns/benchmark.htm&quot;&gt;DNS benchmark tool&lt;/a&gt; that accomplishes the cached,
non-cached and reliability tests, but it only supports 200 nameservers at the same time. I split the
list into chunks and used the command line to automate checking 200 servers at a time.&lt;/p&gt;
&lt;p&gt;After it finished running, I sorted the lists by average response time and ended up with the 200
fastest DNS servers. I then ran the test again on those servers and ended up with the 10 fastest DNS
servers.&lt;/p&gt;
&lt;p&gt;Each of those 10 servers then went through a DNSSEC validation test:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;dig sigok.verteiltesysteme.net @&amp;lt;DNS IP HERE&amp;gt; (should return NOERROR)&lt;/li&gt;
&lt;li&gt;dig sigfail.verteiltesysteme.net @&amp;lt;DNS IP HERE&amp;gt; (should return SERVFAIL)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The 10 servers were then subjected to the DNS spoofability test over at &lt;a href=&quot;https://www.grc.com/dns/dns.htm&quot;&gt;GRC’s DNS
page&lt;/a&gt; to determine the source port and transaction ID entropy.&lt;/p&gt;
&lt;h2 id=&quot;results&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;05&lt;/span&gt;Results&lt;a class=&quot;hash&quot; href=&quot;#results&quot; aria-label=&quot;Link to Results&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The table below is ordered from fastest to slowest based on the average response time.&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 1&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;IP&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Owner&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;DNSSEC validation&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Anti-spoof&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;195.215.95.11&lt;/td&gt;
&lt;td&gt;TDC&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Bad&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;89.233.43.71&lt;/td&gt;
&lt;td&gt;censurfridns.dk&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Bad&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;94.126.178.9&lt;/td&gt;
&lt;td&gt;censurfridns.dk&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Bad&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;94.143.10.52&lt;/td&gt;
&lt;td&gt;dandomain.dk&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Moderate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;147.29.10.6&lt;/td&gt;
&lt;td&gt;sdn.dk&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Moderate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;86.58.175.12&lt;/td&gt;
&lt;td&gt;j.net&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Excellent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;194.239.10.180&lt;/td&gt;
&lt;td&gt;webspeed.dk&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Very bad&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;94.143.10.43&lt;/td&gt;
&lt;td&gt;dandomain.dk&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Moderate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;193.219.27.222&lt;/td&gt;
&lt;td&gt;ipweb.dk&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;N/A (rate limit)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;195.190.31.129&lt;/td&gt;
&lt;td&gt;Welcom Hosting Aps&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Moderate&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;p&gt;If we take a look at just the popular DNS servers - as above, ordered by latency relative to my
geographical location.&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 2&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;IP&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Owner&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;DNSSEC validation&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Anti-spoof&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;8.8.8.8&lt;/td&gt;
&lt;td&gt;Google&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Moderate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4.2.2.1&lt;/td&gt;
&lt;td&gt;Level3&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Excellent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;208.67.222.222&lt;/td&gt;
&lt;td&gt;OpenDNS&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Excellent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;156.154.70.1&lt;/td&gt;
&lt;td&gt;UltraDNS&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Excellent&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;h2 id=&quot;conclusion&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;06&lt;/span&gt;Conclusion&lt;a class=&quot;hash&quot; href=&quot;#conclusion&quot; aria-label=&quot;Link to Conclusion&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;It seems TDC&apos;s DNS server is the fastest but unfortunately it has bad anti-spoofing capabilities. It
seems there is no clear winner regarding speed and security. UltraDNS has the best security but was
also the slowest (70% slower than TDC) of the servers in relation to my geographical location. I&apos;m
sure that people located closer to the servers will have lower latency, and then UltraDNS becomes
the clear winner.&lt;/p&gt;
&lt;p&gt;In the end, there is little to no difference between the top 10 DNS servers and even the 4 popular
servers. Run a test yourself to see just how big the difference is between your current DNS server
and the popular ones, maybe there is better latency and a little more security to get.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Secure hashing of passwords</title>
    <id>https://genbox.github.io/2016/12/secure-hashing-of-passwords.html</id>
    <link href="https://genbox.github.io/2016/12/secure-hashing-of-passwords.html" />
    <published>2016-12-09T16:35:00.0000000+00:00</published>
    <updated>2016-12-18T01:34:32.4130000+00:00</updated>
    <summary>Explaining password-cracking costs, cryptographic hashes, salts, and deliberately slow password hashing.</summary>
    <category term="Security" />
    <content type="html" xml:base="https://genbox.github.io/2016/12/secure-hashing-of-passwords.html">&lt;p&gt;Numerous breaches happen every day due to security vulnerabilities. On this blog, I have previously
analyzed some of the biggest breaches that have been made public. All of them serve as a testament
to the inadequate security many companies employ.&lt;/p&gt;
&lt;h2 id=&quot;the-time-it-takes-to-crack-a-hash&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;The time it takes to crack a hash&lt;a class=&quot;hash&quot; href=&quot;#the-time-it-takes-to-crack-a-hash&quot; aria-label=&quot;Link to The time it takes to crack a hash&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;I often get asked: &quot;How long does it take to crack an MD5 hash?&quot; - implying that the cryptographic
hash algorithm is the most important factor, which rarely is the case. It actually depends on a
couple of factors, which I&apos;ve ordered in descending importance below:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The length of the password&lt;/li&gt;
&lt;li&gt;The charsets used in the password&lt;/li&gt;
&lt;li&gt;The amount of hardware you have&lt;/li&gt;
&lt;li&gt;The methodology you use for cracking&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;secure-passwords&quot;&gt;Secure passwords&lt;a class=&quot;hash&quot; href=&quot;#secure-passwords&quot; aria-label=&quot;Link to Secure passwords&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Password lengths alone are not enough, you also have to use a good charset to control the search
space hackers have to go through. Let&apos;s make an example to show how much the search space changes:&lt;/p&gt;
&lt;p&gt;Charset = a-z, A-Z, 0-9&lt;br&gt;
Length 6: (26+26+10)^6 = 56.800.235.584&lt;br&gt;
Length 7: (26+26+10)^7 = 3.521.614.606.208&lt;br&gt;
Length 8: (26+26+10)^8 = 218.340.105.584.896&lt;/p&gt;
&lt;p&gt;As you can see, the number gets a lot bigger for each character we add. Now let’s look at charsets:&lt;/p&gt;
&lt;p&gt;Length = 8&lt;br&gt;
Charset: a-z = (26)^8 = 208.827.064.576&lt;br&gt;
Charset: a-z, A-Z = (26+26)^8 = 53.459.728.531.456&lt;br&gt;
Charset: a-z, A-Z, 0-9 = (26+26+10)^8 = 218.340.105.584.896&lt;/p&gt;
&lt;p&gt;Charsets are at least as important as the length is. In reality, we need to increase both in order
to get good passwords. For reference, MD5 hashes on a Nvidia Titan X graphics card can be cracked
with around 15.000.000.000 hashes per second. A password of a-z, A-Z, 0-9 of length 8 would take (26+26+10)^8 / 15.000.000.000 = 14556 seconds (about 4 hours). That is not long enough to deter
hackers - we need to do something about the hashing algorithm as well.&lt;/p&gt;
&lt;h2 id=&quot;the-cryptographic-hash-algorithms&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;The cryptographic hash algorithms&lt;a class=&quot;hash&quot; href=&quot;#the-cryptographic-hash-algorithms&quot; aria-label=&quot;Link to The cryptographic hash algorithms&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Through time, developers have used many different cryptographic hash algorithms to store passwords.
Actually, most systems still use rather weak hash algorithms, which is why this blog post even
exists. If we take a look at the hashing algorithms from strong to weak:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;scrypt, bcrypt, PBKDF2&lt;/li&gt;
&lt;li&gt;SHA512, SHA256&lt;/li&gt;
&lt;li&gt;SHA1, MD5&lt;/li&gt;
&lt;li&gt;NTLM, MD4&lt;/li&gt;
&lt;li&gt;LM, DES (unix crypt)&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;I could write 100 blog posts about each algorithm, but suffice to say that LM and DES used to be
VERY widespread in the days of Windows XP and Linux 2.x. They have since moved on to stronger
algorithms, but Windows is stuck on NTLM, and Linux still uses MD5, SHA256 and SHA512 for the most
part, but is transitioning nicely to scrypt and PBKDF2.&lt;/p&gt;
&lt;p&gt;You might be wondering what the speeds are of each algorithm, so I&apos;ll include each of them here. A
larger number of hashes is worse. The speeds are from a Nvidia Geforce 1080 graphics card with
Hashcat:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;scrypt - 434.000 H/s&lt;/li&gt;
&lt;li&gt;bcrypt - 13.226 H/s&lt;/li&gt;
&lt;li&gt;PBKDF2 (SHA1) - 3218.000 H/s&lt;/li&gt;
&lt;li&gt;SHA512 - 1.021.000.000 H/s&lt;/li&gt;
&lt;li&gt;SHA256 - 2.829.000.000 H/s&lt;/li&gt;
&lt;li&gt;SHA1 - 8.491.000.000 H/s&lt;/li&gt;
&lt;li&gt;MD5 - 24.809.000.000 H/s&lt;/li&gt;
&lt;li&gt;NTLM - 41.354.000.000 H/s&lt;/li&gt;
&lt;li&gt;MD4 - 43.145.000.000 H/s&lt;/li&gt;
&lt;li&gt;LM - 18.429.000.000 H/s&lt;/li&gt;
&lt;li&gt;DES - 909.000.000 H/s&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;You might say: Hold on a minute! DES is actually slower than SHA512, does that not mean it is more
secure?&lt;/p&gt;
&lt;p&gt;Nope! That&apos;s because the way the Linux guys implemented DES limited it to a maximum of 8 characters,
and they are also using the 2 first characters as salt, which means we only need to crack 6
character long passwords. LM hashes are also limited to 14 characters, but the way Microsoft made
the algorithm is flawed, and as such, we only need to crack 2x7 character passwords.&lt;/p&gt;
&lt;h2 id=&quot;the-salted-hash&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;The salted hash&lt;a class=&quot;hash&quot; href=&quot;#the-salted-hash&quot; aria-label=&quot;Link to The salted hash&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Password salts are needed in order to protect against &lt;a href=&quot;/2012/01/the-power-of-rainbow-tables.html&quot;&gt;rainbow table
attacks&lt;/a&gt;. Hackers have access to large precomputed tables
of character sets in many different cryptographic hash algorithms today, and when they target just a
single hash, it can be quite effective.&lt;/p&gt;
&lt;p&gt;The salt is just a binary blob that gets appended to the password in the hashing process. A password
of &apos;123456&apos; becomes &apos;123456[SaltHere]&apos;, thereby making it impossible to precompute a table that
contains &apos;123456[SaltHere]&apos;. Salts should be unique for each user in the database, thereby making
the protection even more effective.&lt;/p&gt;
&lt;p&gt;The right way to salt the password is to use a cryptographically secure pseudo-random number
generator (&lt;a href=&quot;https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator&quot;&gt;CSPRNG&lt;/a&gt;) to
create a salt which is unique for each user. It needs to be at least 64 bits (8 bytes) to be
effective. The salt is stored together with the hash inside the user database in plain text.&lt;/p&gt;
&lt;h2 id=&quot;the-slow-hash&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;The slow hash&lt;a class=&quot;hash&quot; href=&quot;#the-slow-hash&quot; aria-label=&quot;Link to The slow hash&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Cryptographic hash functions like MD5 and SHA1 are designed with high performance in mind. This
means that they can quickly produce a hash of an arbitrary input, but it also means that they are
good for brute force and wordlist attacks. To slow down the hashing procedure, we use something
called a key derivation function ( &lt;a href=&quot;https://en.wikipedia.org/wiki/Key_derivation_function&quot;&gt;KDF&lt;/a&gt;).
&lt;a href=&quot;https://en.wikipedia.org/wiki/Scrypt&quot;&gt;scrypt&lt;/a&gt;, &lt;a href=&quot;https://en.wikipedia.org/wiki/Bcrypt&quot;&gt;bcrypt&lt;/a&gt; and
&lt;a href=&quot;https://en.wikipedia.org/wiki/PBKDF2&quot;&gt;PBKDF2&lt;/a&gt; are such functions. They take in a password, a salt
and a number of iterations. Most implementations actually just require a password and a &quot;workload
factor&quot;, which simplifies the use of the algorithms.&lt;/p&gt;
&lt;p&gt;They work by cryptographic primitives like SHA1, SHA256 or Blowfish, which are then run the number of
iterations together with a salt. They also apply the salt in an
&lt;a href=&quot;https://en.wikipedia.org/wiki/Hash-based_message_authentication_code&quot;&gt;HMAC&lt;/a&gt; fashion to be
completely secure against many different cryptographic hash attacks.&lt;/p&gt;
&lt;p&gt;As you can see from the hash speed list above, scrypt, bcrypt and PBKDF2 are much, much slower than
their relatives. This is very configurable and perfect for password hashing. Developers that
implement these algorithms should aim for the number of iterations it takes to make a password
hashing operation last 50 ms or more on their server.&lt;/p&gt;
&lt;p&gt;If we take the example of an 8 character password with the charset a-z, A-Z, 0-9 from above, which
took 4 hours to crack with MD5, and try to crack it using PBKDF2, it would take around 2 years to
crack!&lt;/p&gt;
&lt;h2 id=&quot;conclusion&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;05&lt;/span&gt;Conclusion&lt;a class=&quot;hash&quot; href=&quot;#conclusion&quot; aria-label=&quot;Link to Conclusion&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Use scrypt, bcrypt or PBKDF2 with a high enough work factor that it takes at least 50 ms to compute
a single hash on your server. It takes care of hashing the correct way, applying a salt and usually
uses a CSPRNG to generate the salt. Stay away from everything else.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>The problem with network address translation</title>
    <id>https://genbox.github.io/2016/12/the-problem-with-network-address.html</id>
    <link href="https://genbox.github.io/2016/12/the-problem-with-network-address.html" />
    <published>2016-12-03T11:41:00.0030000+00:00</published>
    <updated>2016-12-18T01:36:36.4920000+00:00</updated>
    <summary>How Network Address Translation complicates direct Internet communication, with examples of connectivity problems and alternatives.</summary>
    <category term="Networking" />
    <content type="html" xml:base="https://genbox.github.io/2016/12/the-problem-with-network-address.html">&lt;p&gt;As a technology ideologist, &lt;a href=&quot;https://en.wikipedia.org/wiki/Network_address_translation&quot;&gt;Network Address
Translation&lt;/a&gt; (NAT) is one of my biggest
concerns when it comes to the future of the Internet. The Internet was built as a communications
tool to facilitate the sharing of information in digital form. It has vastly improved the
communication between humans around the planet and it has been one of the most important inventions
we have ever made.&lt;/p&gt;
&lt;p&gt;My concerns are with the fact that NAT is a direct inhibitor of the nature of the Internet, which
goes against everything we want the Internet to be.&lt;/p&gt;
&lt;h2 id=&quot;network-address-translation&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Network address translation&lt;a class=&quot;hash&quot; href=&quot;#network-address-translation&quot; aria-label=&quot;Link to Network address translation&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The Internet uses routers to route data from network to network, but to do so, we need addresses of
each network. Today we use &lt;a href=&quot;https://en.wikipedia.org/wiki/IPv4&quot;&gt;Internet Protocol version 4&lt;/a&gt; (IPv4),
which you usually see in dotted format such as 199.181.132.250. In its raw form, IPv4 is a 32 bit (4
bytes) addressing scheme which is able to address 2&lt;sup&gt;32 (4.294.967.296) networks, which was enough
back when the Internet was created in the 1950s, but it is nowhere near enough for the Internet
today. To solve this problem, &lt;a href=&quot;https://en.wikipedia.org/wiki/Ipv6&quot;&gt;Internet Protocol version 6&lt;/a&gt;
(IPv6) was created, which has 128 bits that give us 2&lt;/sup&gt;128 (340.282.366.920.938.463.463.374.607.431.768.211.456) addresses, which should be enough to last us
for a long time.&lt;/p&gt;
&lt;p&gt;Routers need a firmware upgrade in order to route 128-bit addresses, and back in the day when IPv6
was devised (1998), routers simply did not have the capacity to route the larger address space. A
quick workaround was to create NAT, which is a simple mechanism to translate one address to another.
With this capability, Internet Service Providers (ISP) began to distribute routers pre-configured
with NAT enabled. They had DHCP enabled on the LAN interface with &lt;a href=&quot;https://en.wikipedia.org/wiki/Private_network&quot;&gt;non-routed IP
addresses&lt;/a&gt; (192.168.0.0/24), which then got
translated to a routed IP address on the WAN interface. This allowed ISPs to extend the lifetime of
IPv4 and keep using low capacity consumer routers.&lt;/p&gt;
&lt;p&gt;That does not sound too bad, right? It is a clever way to circumvent the limitations of IPv4 and
keep using existing hardware which was produced for cheap in large quantities.&lt;/p&gt;
&lt;h2 id=&quot;the-problem&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;The problem&lt;a class=&quot;hash&quot; href=&quot;#the-problem&quot; aria-label=&quot;Link to The problem&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A side effect of NAT is that it splits the Internet into many smaller private networks, which can&apos;t
communicate directly with each other unless you do &lt;a href=&quot;https://en.wikipedia.org/wiki/Port_forwarding&quot;&gt;port
forwarding&lt;/a&gt;. This side effect has been documented as
a &quot;security feature&quot; ever since it was conceived, as it essentially functions like a simple firewall
between networks.&lt;/p&gt;
&lt;p&gt;Remember what the purpose of the Internet was? That&apos;s right, to transmit digital information between
machines! NAT completely contradicts the whole purpose of the Internet by segmenting the Internet,
just because we saw NAT as a quick fix to our IPv4 addressing problem. We have become accustomed to
the fact that NAT exists. It has served its purpose, and it is time we move on to IPv6 and make the
Internet work as intended again.&lt;/p&gt;
&lt;h2 id=&quot;the-solution&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;The solution&lt;a class=&quot;hash&quot; href=&quot;#the-solution&quot; aria-label=&quot;Link to The solution&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;By now you should already know that IPv6 is the solution to the NAT problem, but there is still an
unanswered question: What about the security NAT provides?&lt;/p&gt;
&lt;p&gt;Let me counter-question you with this: What is it we are trying to achieve?&lt;/p&gt;
&lt;p&gt;I&apos;d agree that the Internet should not just be one big monolithic security boundary, where any
hosted service can be reached by everyone else. That is why we have firewalls, which NAT is not. You
still have a router with two networks, and in order to publish a service from one network to the
other, a firewall will have to allow it. Firewalls are a much more efficient solution to the
security problem than NAT is.&lt;br&gt;
Let&apos;s say we enabled IPv6, what exactly would that mean?&lt;/p&gt;
&lt;p&gt;That is the funny part; you are &lt;strong&gt;already&lt;/strong&gt; running IPv6! That&apos;s right, you are running a
dual-stacked network layer capable of both IPv4 and IPv6, you don&apos;t have to do anything. Most
routers are also already dual-stacked, we just have to disable IPv4 and tadaaaa, you are IPv6 only.&lt;/p&gt;
&lt;p&gt;For router manufacturers, developers and network engineers it is a &lt;strong&gt;huge&lt;/strong&gt; advantage to completely
disable IPv4, which is why many ISPs today are running IPv6-only networks internally. As a
manufacturer or developer, you can remove NAT,
&lt;a href=&quot;https://en.wikipedia.org/wiki/Traversal_Using_Relays_around_NAT&quot;&gt;TURN&lt;/a&gt;,
&lt;a href=&quot;https://en.wikipedia.org/wiki/STUN&quot;&gt;STUN&lt;/a&gt;,
&lt;a href=&quot;https://en.wikipedia.org/wiki/Internet_Gateway_Device_Protocol&quot;&gt;IGDP&lt;/a&gt; and
&lt;a href=&quot;https://en.wikipedia.org/wiki/NAT_Port_Mapping_Protocol&quot;&gt;NAT-PMP&lt;/a&gt; from routers and communication
software. Of course, we still need a network control protocol that publishes a service in the
router&apos;s firewall, but it is so much more simple now that you don&apos;t have to take NAT into account.&lt;/p&gt;
&lt;h2 id=&quot;a-real-example&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;A real example&lt;a class=&quot;hash&quot; href=&quot;#a-real-example&quot; aria-label=&quot;Link to A real example&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The BitTorrent protocol is one of the most commonly occurring protocols on the Internet. It is a
simple protocol to transmit binary data between clients in an efficient manner using Peer to Peer (P2P). Clients find each other by using a Tracker or what&apos;s known as a &lt;a href=&quot;https://en.wikipedia.org/wiki/Distributed_hash_table&quot;&gt;Distributed Hash
Table&lt;/a&gt; (DHT). DHT is a simple protocol, but it
is severely limited by NAT, simply by the fact that it works in a very socialistic manner. Each
client in the DHT network has to store a chunk of data for it to work, but since most of them now
sit behind NAT-enabled routers, they can&apos;t be reached by the rest of the network, thereby making
the data they contain unreachable by others. This fundamentally destroys the whole concept of a
Distributed Hash Table! All BitTorrent applications have implemented one of the many protocols to
circumvent NAT, but none of them are perfect. If we switched to IPv6 only networks, DHTs can finally
work again, and we would see a massive gain in performance as clients can find each other more
efficiently.&lt;/p&gt;
&lt;p&gt;Other P2P protocols like Skype would also work better and would not have to route through a third
party. Back in 2011 when Microsoft bought Skype, they began transitioning the Skype P2P protocol
from a highly distributed node network into a more centralized platform based on &lt;a href=&quot;https://en.wikipedia.org/wiki/Microsoft_Notification_Protocol&quot;&gt;Microsoft
Notification Protocol&lt;/a&gt; (MSNP). The
centralized protocol works &quot;better&quot; because Microsoft&apos;s servers work as a broker between you and
other clients, thereby circumventing NAT. If the clients both have IPv6 and publish the Skype
service in their router firewall, they could reach each other directly, and instantly get a massive
performance and stability gain.&lt;/p&gt;
&lt;p&gt;Maybe this is just wishful thinking, but it does seem that the &lt;a href=&quot;https://www.google.com/intl/en/ipv6/statistics.html&quot;&gt;IPv6
adoption&lt;/a&gt; is gaining speed, which is good news
for the internet as a whole.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>.NET compression libraries benchmark</title>
    <id>https://genbox.github.io/2014/12/net-compression-libraries-benchmark.html</id>
    <link href="https://genbox.github.io/2014/12/net-compression-libraries-benchmark.html" />
    <published>2014-11-30T23:06:00.0020000+00:00</published>
    <updated>2026-10-31T16:14:26.0000000+00:00</updated>
    <summary>Comparing .NET compression libraries by compression ratio, compression speed, and decompression speed using in-memory benchmarks.</summary>
    <category term="C#" />
<category term="Performance" />
    <content type="html" xml:base="https://genbox.github.io/2014/12/net-compression-libraries-benchmark.html">&lt;p&gt;I often find myself needing a compression library in a software development project, and after a few
tests with disappointing results I usually fall back to spawning 7Zip as an external process. This
time however, the application specification prohibits any data written to disk, which my existing
7Zip workflow required. I needed to do everything in memory, and so I had to test some of the
C# libraries available for compression as well as the different compression algorithms.&lt;/p&gt;
&lt;p&gt;Here is a full list of the tested libraries:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://learn.microsoft.com/en-us/dotnet/api/system.io.compression.deflatestream?view=net-10.0&amp;amp;redirectedfrom=MSDN&quot;&gt;.NET
DeflateStream&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://icsharpcode.github.io/SharpZipLib/&quot;&gt;ICSharpCode SharpZip&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.componentace.com/zlib_.NET.htm&quot;&gt;Zlib.NET&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://github.com/DinoChiesa/DotNetZip&quot;&gt;DotNetZip&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://code.google.com/p/managed-lzma/&quot;&gt;Managed LZMA&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href=&quot;https://www.7-zip.org/sdk.html&quot;&gt;LZMA SDK&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href=&quot;http://www.quicklz.com/&quot;&gt;QuickLZ&lt;/a&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;compression-101&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Compression 101&lt;a class=&quot;hash&quot; href=&quot;#compression-101&quot; aria-label=&quot;Link to Compression 101&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Before I get to the results of the benchmark, I think it is important to understand the different
archive formats and compression algorithms in order to compare the results with other benchmarks and/
or tests. Let&apos;s get to it!&lt;/p&gt;
&lt;p&gt;A compression algorithm can be compared to a small machine that takes in some data, does some math on
it and transforms it to a representation intended to be smaller; some inputs grow instead. It sounds simple, but it involves a lot
of math to do it efficiently. The best compression algorithms are the ones which have a high
compression ratio while only using small amounts of RAM and CPU. One such algorithm is the very well
known DEFLATE which internally uses the famous LZ77 algorithm created by Abraham Lempel and Jacob
Ziv. The DEFLATE algorithm has become the de facto standard when it comes to compression, and it is
used in nearly all instances when a compression algorithm is needed. There are many compression
algorithms out there, some are better than DEFLATE and also very widespread. Examples include LZW,
BWT, LZSS and LZMA.&lt;/p&gt;
&lt;p&gt;Each of the compression algorithms only takes in a single file, either as a blob of bytes, or a
stream of bytes. The question is, how do compression applications such as 7Zip, WinZip and WinRAR
compress multiple files?&lt;/p&gt;
&lt;p&gt;Enter the world of archive and compression formats. Zip and RAR hold files, while gzip and bzip2
compress byte streams. They all simply
describe the content of themselves either in terms of files or streams of data, along with some
metadata such as original size, file name and path. The archive formats are more complex than you
would think, simply because there are a lot of features. Some of them support multiple data streams (for multi-threading), encryption, password protection, file name obfuscation, self-extracting
archives and more.&lt;/p&gt;
&lt;figure class=&quot;diagram diagram-panel zip-diagram&quot; role=&quot;img&quot; aria-label=&quot;File 1 and File 2 are compressed separately. Their compressed data is stored as Entry 1 and Entry 2 in one ZIP archive.&quot;&gt;
    &lt;div class=&quot;zip-grid&quot; aria-hidden=&quot;true&quot;&gt;
        &lt;span class=&quot;zip-heading&quot;&gt;Input files&lt;/span&gt;
        &lt;span class=&quot;zip-heading&quot;&gt;Compression&lt;/span&gt;
        &lt;span class=&quot;zip-heading&quot;&gt;ZIP archive&lt;/span&gt;
        &lt;div class=&quot;zip-step&quot;&gt;File 1&lt;/div&gt;
        &lt;div class=&quot;zip-step&quot;&gt;Compress&lt;/div&gt;
        &lt;div class=&quot;zip-archive&quot;&gt;
            &lt;div class=&quot;zip-entry&quot;&gt;Entry 1&lt;/div&gt;
            &lt;div class=&quot;zip-entry&quot;&gt;Entry 2&lt;/div&gt;
        &lt;/div&gt;
        &lt;div class=&quot;zip-step&quot;&gt;File 2&lt;/div&gt;
        &lt;div class=&quot;zip-step&quot;&gt;Compress&lt;/div&gt;
    &lt;/div&gt;
    &lt;figcaption aria-hidden=&quot;true&quot;&gt;Each file is compressed separately and stored as an entry in the archive.&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;Figure 1: The relationship between files, compression algorithms and archive formats.&lt;/p&gt;
&lt;p&gt;Many of the archiving formats are now synonymous with the compression algorithm they use. We say
&quot;Zip the file&quot; or &quot;Make a 7z of the folder&quot;. Archive formats can support several compression
algorithms, with the allowed combinations defined by each format, such as Zip with LZMA or 7z with
DEFLATE. That being said, there are some default compression algorithms in some
archive formats. I&apos;ve listed some examples here in archive/algorithm format:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Zip / DEFLATE&lt;/li&gt;
&lt;li&gt;7zip / LZMA&lt;/li&gt;
&lt;li&gt;Bzip2 / BWT&lt;/li&gt;
&lt;li&gt;Gzip / DEFLATE&lt;/li&gt;
&lt;li&gt;GIF / LZW&lt;/li&gt;
&lt;li&gt;PNG / DEFLATE&lt;/li&gt;
&lt;li&gt;Tar / none&lt;/li&gt;
&lt;li&gt;Rar / LZSS&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The avid reader noticed that I put a couple of image formats in the list (GIF, PNG). Both of them
use compression algorithms to minimize the size of the image data within their image formats. The same applies to Microsoft Excel 2013 files (xlsx / DEFLATE) and Java Archives (Jar / DEFLATE).&lt;/p&gt;
&lt;p&gt;In the Linux world, the solution to compressing multiple files was solved using an archiving format
called Tar. It is a pretty simple format designed for tape archiving, which preserves the filesystem
information such as timestamps, owner information and file size on the files held inside the
archive. To compress a directory, Tar simply stores all the files within the directory inside a
single file. A separate compressor such as gzip or bzip2 then compresses that file.&lt;/p&gt;
&lt;h2 id=&quot;benchmark&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;Benchmark&lt;a class=&quot;hash&quot; href=&quot;#benchmark&quot; aria-label=&quot;Link to Benchmark&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Now that we know the difference between compression algorithms and archive formats, we can continue
to the benchmark. The libraries tested here are overlapping in the sense that they are simply
different implementations of different compression algorithms, and even though the name of the
library might contain Zip, it has nothing to do with the actual Zip archiving format, but everything
to do with the compression algorithm used by default in Zip files.&lt;/p&gt;
&lt;p&gt;The benchmark uses a 128 MB test file that contains images, music, text and some random data. The
file represents arbitrary data usually found on a filesystem.&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 1&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;Library&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Type&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Packed (MB)&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Pack (ms)&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Unpack (ms)&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Ratio&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Speed (MB/s)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;.NET 4.5.1&lt;/td&gt;
&lt;td&gt;Zip&lt;/td&gt;
&lt;td&gt;34.10&lt;/td&gt;
&lt;td&gt;2,921&lt;/td&gt;
&lt;td&gt;895&lt;/td&gt;
&lt;td&gt;3.58&lt;/td&gt;
&lt;td&gt;43.82&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ICSharpCode&lt;/td&gt;
&lt;td&gt;Zip&lt;/td&gt;
&lt;td&gt;34.07&lt;/td&gt;
&lt;td&gt;6,373&lt;/td&gt;
&lt;td&gt;1,524&lt;/td&gt;
&lt;td&gt;3.58&lt;/td&gt;
&lt;td&gt;20.08&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ICSharpCode&lt;/td&gt;
&lt;td&gt;Gzip&lt;/td&gt;
&lt;td&gt;34.07&lt;/td&gt;
&lt;td&gt;6,347&lt;/td&gt;
&lt;td&gt;1,511&lt;/td&gt;
&lt;td&gt;3.58&lt;/td&gt;
&lt;td&gt;20.17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ICSharpCode&lt;/td&gt;
&lt;td&gt;BZip2&lt;/td&gt;
&lt;td&gt;35.69&lt;/td&gt;
&lt;td&gt;45,002&lt;/td&gt;
&lt;td&gt;6,811&lt;/td&gt;
&lt;td&gt;3.42&lt;/td&gt;
&lt;td&gt;2.84&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7z SDK&lt;/td&gt;
&lt;td&gt;LZMA&lt;/td&gt;
&lt;td&gt;28.51&lt;/td&gt;
&lt;td&gt;40,235&lt;/td&gt;
&lt;td&gt;3,254&lt;/td&gt;
&lt;td&gt;4.28&lt;/td&gt;
&lt;td&gt;3.18&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Zlib.NET&lt;/td&gt;
&lt;td&gt;Zlib&lt;/td&gt;
&lt;td&gt;33.97&lt;/td&gt;
&lt;td&gt;4,837&lt;/td&gt;
&lt;td&gt;985&lt;/td&gt;
&lt;td&gt;3.59&lt;/td&gt;
&lt;td&gt;26.46&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DotNETZip&lt;/td&gt;
&lt;td&gt;Zip&lt;/td&gt;
&lt;td&gt;33.97&lt;/td&gt;
&lt;td&gt;5,120&lt;/td&gt;
&lt;td&gt;1,482&lt;/td&gt;
&lt;td&gt;3.59&lt;/td&gt;
&lt;td&gt;25.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DotNETZip&lt;/td&gt;
&lt;td&gt;Gzip&lt;/td&gt;
&lt;td&gt;35.67&lt;/td&gt;
&lt;td&gt;31,739&lt;/td&gt;
&lt;td&gt;4,556&lt;/td&gt;
&lt;td&gt;3.42&lt;/td&gt;
&lt;td&gt;4.03&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DotNETZip&lt;/td&gt;
&lt;td&gt;Zlib&lt;/td&gt;
&lt;td&gt;33.97&lt;/td&gt;
&lt;td&gt;4,509&lt;/td&gt;
&lt;td&gt;887&lt;/td&gt;
&lt;td&gt;3.59&lt;/td&gt;
&lt;td&gt;28.39&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LZMA-managed&lt;/td&gt;
&lt;td&gt;LZMA&lt;/td&gt;
&lt;td&gt;28.07&lt;/td&gt;
&lt;td&gt;35,555&lt;/td&gt;
&lt;td&gt;7,402&lt;/td&gt;
&lt;td&gt;4.35&lt;/td&gt;
&lt;td&gt;3.60&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;QuickLZ&lt;/td&gt;
&lt;td&gt;QuickLZ&lt;/td&gt;
&lt;td&gt;42.90&lt;/td&gt;
&lt;td&gt;1,015&lt;/td&gt;
&lt;td&gt;1,010&lt;/td&gt;
&lt;td&gt;2.85&lt;/td&gt;
&lt;td&gt;126.11&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;h2 id=&quot;conclusion&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;Conclusion&lt;a class=&quot;hash&quot; href=&quot;#conclusion&quot; aria-label=&quot;Link to Conclusion&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The fastest algorithm was QuickLZ, but it also has the worst compression ratio. The best compression
ratio was gained by LZMA, but it is also the third-slowest compression speed. It would seem the
built-in DeflateStream in .NET is the overall winner when it comes to speed vs. compression ratio.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>Analysis of the Gamigo hashes</title>
    <id>https://genbox.github.io/2012/08/analysis-of-gamigo-hashes.html</id>
    <link href="https://genbox.github.io/2012/08/analysis-of-gamigo-hashes.html" />
    <published>2012-08-27T16:09:00.0000000+00:00</published>
    <updated>2016-12-03T09:51:20.7940000+00:00</updated>
    <summary>Analyzing the Gamigo password hashes through cracking results, password strength, character patterns, email addresses, and domains.</summary>
    <category term="Security" />
    <content type="html" xml:base="https://genbox.github.io/2012/08/analysis-of-gamigo-hashes.html">&lt;p&gt;This leak analysis is dedicated to Steve Gibson of &lt;a href=&quot;https://www.grc.com/intro.htm&quot;&gt;Gibson Research
Corporation&lt;/a&gt;.&lt;br&gt;
Thanks for a great show Steve!&lt;/p&gt;
&lt;p&gt;Back in February, a large gaming site called Gamigo was
&lt;a href=&quot;https://web.archive.org/web/20130103055435/http://www.zdnet.com/8-24-million-gamigo-passwords-leaked-after-hack-7000001403/&quot;&gt;hacked&lt;/a&gt;, and in
July, the full list of emails and hashes found their way onto the Internet.
&lt;a href=&quot;http://pwnedlist.com/&quot;&gt;Pwnedlist.com&lt;/a&gt; are the kind providers of the hashes for this analysis, and
as an added bonus, they also sent me the emails for this analysis. A huge thanks goes out to
Pwnedlist.com for their kindness.&lt;/p&gt;
&lt;p&gt;The leak contains 9.475.226 valid MD5 hashes and 8.261.454 emails. It has 7.028.067 unique hashes
and 8.244.423 unique emails.&lt;/p&gt;
&lt;p&gt;In the time-frame of 7 days, 19 hours and 3 seconds, I was able to crack 7.731.708 hashes (81,6%).&lt;/p&gt;
&lt;h2 id=&quot;cracking-system&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;Cracking system&lt;a class=&quot;hash&quot; href=&quot;#cracking-system&quot; aria-label=&quot;Link to Cracking system&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The cracking was done on an ordinary GeForce 560 TI, 1024MB RAM graphics card using Hashcat-Plus
v0.081 64bit. The settings of Hashcat were set to low, which resulted in only 90% cracking
efficiency. Had it been optimized, the full crack would have taken 6 days and 20 hours.&lt;/p&gt;
&lt;p&gt;Different cracking techniques such as keyboard-walking, character repetition and leet speak were
used to obtain the high number of cracked passwords.&lt;/p&gt;
&lt;h2 id=&quot;character-distribution&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;Character distribution&lt;a class=&quot;hash&quot; href=&quot;#character-distribution&quot; aria-label=&quot;Link to Character distribution&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A healthy character distribution has equal amounts of each character type. Gamigo must have had a
less than optimal password policy, as it seems that you could create passwords with whatever
characters you want.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-characters&quot; class=&quot;chart chart-bars&quot; aria-labelledby=&quot;chart-gamigo-characters-note&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-characters-note&quot;&gt;Share of character types across all passwords&lt;/span&gt;&lt;/figcaption&gt;&lt;ol class=&quot;chart-rows&quot;&gt;&lt;li title=&quot;Lowercase letters: 46%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Lowercase letters&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 46%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;46%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Uppercase letters: 7%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Uppercase letters&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 7%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;7%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Symbols: 10%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Symbols&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 10%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;10%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Digits: 37%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Digits&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 37%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;37%&lt;/span&gt;&lt;/li&gt;&lt;/ol&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 1&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Lowercase letters&lt;/td&gt;&lt;td&gt;46%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Uppercase letters&lt;/td&gt;&lt;td&gt;7%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Symbols&lt;/td&gt;&lt;td&gt;10%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Digits&lt;/td&gt;&lt;td&gt;37%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;h2 id=&quot;unique-character-distribution&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;Unique character distribution&lt;a class=&quot;hash&quot; href=&quot;#unique-character-distribution&quot; aria-label=&quot;Link to Unique character distribution&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Again, we are reminded of the missing password policy at Gamigo. 28% of the passwords are digits
only, with a good password policy, this would not have been possible.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-combinations&quot; class=&quot;chart chart-bars&quot; aria-labelledby=&quot;chart-gamigo-combinations-note&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-combinations-note&quot;&gt;Share of passwords using only one character type&lt;/span&gt;&lt;/figcaption&gt;&lt;ol class=&quot;chart-rows&quot;&gt;&lt;li title=&quot;Lowercase only: 71%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Lowercase only&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 71%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;71%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Uppercase only: 1%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Uppercase only&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 1%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;1%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Symbols only: 0%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Symbols only&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 0%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;0%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Digits only: 28%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Digits only&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 28%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;28%&lt;/span&gt;&lt;/li&gt;&lt;/ol&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 2&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Lowercase only&lt;/td&gt;&lt;td&gt;71%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Uppercase only&lt;/td&gt;&lt;td&gt;1%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Symbols only&lt;/td&gt;&lt;td&gt;0%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Digits only&lt;/td&gt;&lt;td&gt;28%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;h2 id=&quot;password-composition&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;Password composition&lt;a class=&quot;hash&quot; href=&quot;#password-composition&quot; aria-label=&quot;Link to Password composition&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;When we take a look at the password composition, we count how many passwords contain symbols. It
turns out that only 13.566 passwords that were recovered contained a symbol, and 7.718.142 that did
not. This resulted in the rather boring chart:&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-composition&quot; class=&quot;chart chart-split&quot; aria-labelledby=&quot;chart-gamigo-composition-note&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-composition-note&quot;&gt;Share of cracked passwords&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;chart-segments&quot;&gt;&lt;span class=&quot;chart-part chart-part-0&quot; style=&quot;--v: 100%&quot; title=&quot;Alphanumeric: 100%&quot;&gt;&lt;/span&gt;&lt;span class=&quot;chart-part chart-part-1&quot; style=&quot;--v: 0%&quot; title=&quot;Not alphanumeric: 0%&quot;&gt;&lt;/span&gt;&lt;/div&gt;&lt;ul class=&quot;chart-keys&quot;&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-0&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;Alphanumeric &lt;span class=&quot;chart-value&quot;&gt;100%&lt;/span&gt;&lt;/li&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-1&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;Not alphanumeric &lt;span class=&quot;chart-value&quot;&gt;0%&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 3&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Alphanumeric&lt;/td&gt;&lt;td&gt;100%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Not alphanumeric&lt;/td&gt;&lt;td&gt;0%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;h2 id=&quot;password-length&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;05&lt;/span&gt;Password length&lt;a class=&quot;hash&quot; href=&quot;#password-length&quot; aria-label=&quot;Link to Password length&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Gamigo has 2 passwords with length 0 and 20 passwords with length 1. These might be passwords
created during the development of the site - and they might have been the way the hackers originally
hacked the site (first password check in most brute forcers is nothing).&lt;/p&gt;
&lt;p&gt;We observe something quite unusual here, and that is the high number of 10 character passwords. The
password lengths also show the missing password policy, as you can have whatever length you like.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-length&quot; class=&quot;chart chart-columns&quot; aria-labelledby=&quot;chart-gamigo-length-note&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-length-note&quot;&gt;Share of cracked passwords&lt;/span&gt;&lt;/figcaption&gt;&lt;ol class=&quot;chart-cols&quot;&gt;&lt;li title=&quot;0: 0%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;0%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 0%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;0&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;1: 0%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;0%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 0%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;1&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;2: 0%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;0%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 0%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;2&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;3: 0%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;0%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 0%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;3&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;4: 1%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;1%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 2.86%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;4&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;5: 2%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;2%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 5.71%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;5&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;6: 14%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;14%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 40%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;6&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;7: 9%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;9%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 25.71%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;7&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;8: 26%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;26%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 74.29%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;8&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;9: 11%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;11%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 31.43%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;9&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;10: 29%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;29%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 82.86%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;10&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;11: 4%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;4%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 11.43%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;11&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;12: 2%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;2%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 5.71%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;12&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;13: 1%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;1%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 2.86%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;13&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;14: 0%&quot;&gt;&lt;span class=&quot;chart-colbox&quot;&gt;&lt;span class=&quot;chart-value&quot;&gt;0%&lt;/span&gt;&lt;span class=&quot;chart-col&quot; style=&quot;--v: 0%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-label&quot;&gt;14&lt;/span&gt;&lt;/li&gt;&lt;/ol&gt;&lt;p class=&quot;chart-axis-label&quot;&gt;Characters&lt;/p&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 4&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Characters&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;0&lt;/td&gt;&lt;td&gt;0%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;1&lt;/td&gt;&lt;td&gt;0%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;2&lt;/td&gt;&lt;td&gt;0%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;3&lt;/td&gt;&lt;td&gt;0%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;4&lt;/td&gt;&lt;td&gt;1%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;5&lt;/td&gt;&lt;td&gt;2%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;6&lt;/td&gt;&lt;td&gt;14%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;7&lt;/td&gt;&lt;td&gt;9%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;8&lt;/td&gt;&lt;td&gt;26%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;9&lt;/td&gt;&lt;td&gt;11%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;10&lt;/td&gt;&lt;td&gt;29%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;11&lt;/td&gt;&lt;td&gt;4%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;12&lt;/td&gt;&lt;td&gt;2%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;13&lt;/td&gt;&lt;td&gt;1%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;14&lt;/td&gt;&lt;td&gt;0%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;p&gt;The positive side of the case, is that the high number of 10 character passwords pushed the average
password length up to 8,52 characters; higher than usual.&lt;/p&gt;
&lt;h3 id=&quot;longest-passwords&quot;&gt;10 longest passwords&lt;a class=&quot;hash&quot; href=&quot;#longest-passwords&quot; aria-label=&quot;Link to 10 longest passwords&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;The 10 longest passwords give an idea of how low strength long passwords can have. Repetition and
common patterns are used to create long passwords; however, they are easily cracked using modern
password cracking techniques.&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 5&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;Password&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Length&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;frontschweinefrontschweine&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwertyuiopasdfghjklzxcvbnm&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;stevieboy74stevieboy74&lt;/td&gt;
&lt;td&gt;22&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;needforspeedmostwanted&lt;/td&gt;
&lt;td&gt;22&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;lllllllllllllllllllll&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qaywsxedcrfvtgbzhnujm&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;sivispacemparabellum&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;abcdefghijklmnopqrst&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12345678900987600000&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1qazxcvbnm1qazxcvbnm&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;h3 id=&quot;shortest-passwords&quot;&gt;10 shortest passwords&lt;a class=&quot;hash&quot; href=&quot;#shortest-passwords&quot; aria-label=&quot;Link to 10 shortest passwords&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;The 10 shortest passwords are hopefully not user passwords. Both a, b and c are among the shortest
passwords, which leads me to believe that they are internal testing passwords or passwords created
during development.&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 6&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;Password&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Length&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;x&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;g&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;½&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;b&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;c&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;S&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;h2 id=&quot;password-strength&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;06&lt;/span&gt;Password strength&lt;a class=&quot;hash&quot; href=&quot;#password-strength&quot; aria-label=&quot;Link to Password strength&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A strong password has 9 or more characters, 1 digit, 1 lowercase, 1 uppercase and 1 symbol. However,
in Gamigo&apos;s case, only 1250 passwords are considered strong.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-strength&quot; class=&quot;chart chart-split&quot; aria-labelledby=&quot;chart-gamigo-strength-note&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-strength-note&quot;&gt;Share of cracked passwords&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;chart-segments&quot;&gt;&lt;span class=&quot;chart-part chart-part-0&quot; style=&quot;--v: 0%&quot; title=&quot;Considered strong: 0%&quot;&gt;&lt;/span&gt;&lt;span class=&quot;chart-part chart-part-1&quot; style=&quot;--v: 100%&quot; title=&quot;Not strong: 100%&quot;&gt;&lt;/span&gt;&lt;/div&gt;&lt;ul class=&quot;chart-keys&quot;&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-0&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;Considered strong &lt;span class=&quot;chart-value&quot;&gt;0%&lt;/span&gt;&lt;/li&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-1&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;Not strong &lt;span class=&quot;chart-value&quot;&gt;100%&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 7&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Considered strong&lt;/td&gt;&lt;td&gt;0%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Not strong&lt;/td&gt;&lt;td&gt;100%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;p&gt;This low number of strong passwords is a combination of the fact that the passwords were cracked,
but also because of the missing password policy. A password of 10+ lowercase characters is not
considered secure, and a single symbol in a 10+ character password would exponentially increase the
security of the password.&lt;/p&gt;
&lt;h2 id=&quot;wordlists&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;07&lt;/span&gt;Wordlists&lt;a class=&quot;hash&quot; href=&quot;#wordlists&quot; aria-label=&quot;Link to Wordlists&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;17% of the passwords were cracked right away using common wordlists. That is pretty normal on sites
with limited or no password policy.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-wordlists&quot; class=&quot;chart chart-split&quot; aria-labelledby=&quot;chart-gamigo-wordlists-title&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-title&quot; id=&quot;chart-gamigo-wordlists-title&quot;&gt;Common wordlists&lt;/span&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-wordlists-note&quot;&gt;Share of cracked passwords&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;chart-segments&quot;&gt;&lt;span class=&quot;chart-part chart-part-0&quot; style=&quot;--v: 17%&quot; title=&quot;In wordlists: 17%&quot;&gt;&lt;/span&gt;&lt;span class=&quot;chart-part chart-part-1&quot; style=&quot;--v: 83%&quot; title=&quot;Not in wordlists: 83%&quot;&gt;&lt;/span&gt;&lt;/div&gt;&lt;ul class=&quot;chart-keys&quot;&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-0&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;In wordlists &lt;span class=&quot;chart-value&quot;&gt;17%&lt;/span&gt;&lt;/li&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-1&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;Not in wordlists &lt;span class=&quot;chart-value&quot;&gt;83%&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 8&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;In wordlists&lt;/td&gt;&lt;td&gt;17%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Not in wordlists&lt;/td&gt;&lt;td&gt;83%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;p&gt;The IQ Wordlist (more info
&lt;a href=&quot;/2012/01/building-perfect-wordlist.html&quot;&gt;here&lt;/a&gt;) cracked 20% of the
passwords.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-iq&quot; class=&quot;chart chart-split&quot; aria-labelledby=&quot;chart-gamigo-iq-title&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-title&quot; id=&quot;chart-gamigo-iq-title&quot;&gt;IQ wordlist&lt;/span&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-iq-note&quot;&gt;Share of cracked passwords&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;chart-segments&quot;&gt;&lt;span class=&quot;chart-part chart-part-0&quot; style=&quot;--v: 20%&quot; title=&quot;In IQ wordlist: 20%&quot;&gt;&lt;/span&gt;&lt;span class=&quot;chart-part chart-part-1&quot; style=&quot;--v: 80%&quot; title=&quot;Not in IQ wordlist: 80%&quot;&gt;&lt;/span&gt;&lt;/div&gt;&lt;ul class=&quot;chart-keys&quot;&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-0&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;In IQ wordlist &lt;span class=&quot;chart-value&quot;&gt;20%&lt;/span&gt;&lt;/li&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-1&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;Not in IQ wordlist &lt;span class=&quot;chart-value&quot;&gt;80%&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 9&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;In IQ wordlist&lt;/td&gt;&lt;td&gt;20%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Not in IQ wordlist&lt;/td&gt;&lt;td&gt;80%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;h2 id=&quot;most-common-passwords&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;08&lt;/span&gt;10 most common passwords&lt;a class=&quot;hash&quot; href=&quot;#most-common-passwords&quot; aria-label=&quot;Link to 10 most common passwords&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The passwords in this list are the most common passwords used by Gamigo users. Most of them are
numbers only, but a few stand out. &apos;azerty&apos; is used a lot by French users since they have
&lt;a href=&quot;https://en.wikipedia.org/wiki/AZERTY&quot;&gt;AZERTY&lt;/a&gt; keyboard layouts (see the email analysis below for
more info on French users). I&apos;m unsure why &apos;perach&apos; is on the list, it might be the &lt;a href=&quot;https://de.wikipedia.org/wiki/Perach&quot;&gt;German
municipality&lt;/a&gt; or a password used multiple times by a bot.&lt;/p&gt;
&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 10&quot;&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th scope=&quot;col&quot;&gt;Password&lt;/th&gt;
&lt;th scope=&quot;col&quot;&gt;Frequency&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;123456&lt;/td&gt;
&lt;td&gt;35.621&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;123456789&lt;/td&gt;
&lt;td&gt;24.377&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;czz000&lt;/td&gt;
&lt;td&gt;6.166&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12345&lt;/td&gt;
&lt;td&gt;5.300&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;111111&lt;/td&gt;
&lt;td&gt;4.765&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;azerty&lt;/td&gt;
&lt;td&gt;4.614&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;12345678&lt;/td&gt;
&lt;td&gt;4.547&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;perach&lt;/td&gt;
&lt;td&gt;3.913&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;123123&lt;/td&gt;
&lt;td&gt;3.749&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1234567890&lt;/td&gt;
&lt;td&gt;3.679&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;h2 id=&quot;email-analysis&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;09&lt;/span&gt;Email analysis&lt;a class=&quot;hash&quot; href=&quot;#email-analysis&quot; aria-label=&quot;Link to Email analysis&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;The leak also contained a large amount of emails. So as an added bonus, I&apos;ve made an analysis of the
emails.&lt;/p&gt;
&lt;p&gt;8.261.454 emails were in the leak and 8.244.423 (99,7%) of them are unique. 79065 of the emails were
invalid according to &lt;a href=&quot;https://www.rfc-editor.org/rfc/rfc2822.txt&quot;&gt;RFC 2822&lt;/a&gt;, that leaves 8.165.358 (99%
of unique) valid emails.&lt;/p&gt;
&lt;h2 id=&quot;domain-statistics&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;10&lt;/span&gt;Domain statistics&lt;a class=&quot;hash&quot; href=&quot;#domain-statistics&quot; aria-label=&quot;Link to Domain statistics&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;In the domain statistics, we can see that &quot;hotmail.com&quot; and &quot;hotmail.fr&quot; are the two most used
email providers. This hints at a large percentage of French users.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-domains&quot; class=&quot;chart chart-bars&quot; aria-labelledby=&quot;chart-gamigo-domains-note&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-domains-note&quot;&gt;Share of email addresses by provider&lt;/span&gt;&lt;/figcaption&gt;&lt;ol class=&quot;chart-rows&quot;&gt;&lt;li title=&quot;hotmail.com: 27%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;hotmail.com&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 90%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;27%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;hotmail.fr: 15%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;hotmail.fr&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 50%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;15%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;web.de: 14%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;web.de&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 46.67%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;14%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;gmail.com: 11%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;gmail.com&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 36.67%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;11%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;gmx.de: 8%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;gmx.de&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 26.67%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;8%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;hotmail.de: 7%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;hotmail.de&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 23.33%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;7%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;yahoo.com: 6%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;yahoo.com&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 20%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;6%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;yahoo.de: 5%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;yahoo.de&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 16.67%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;5%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;live.fr: 4%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;live.fr&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 13.33%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;4%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;o2.pl: 3%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;o2.pl&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 10%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;3%&lt;/span&gt;&lt;/li&gt;&lt;/ol&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 11&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;hotmail.com&lt;/td&gt;&lt;td&gt;27%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;hotmail.fr&lt;/td&gt;&lt;td&gt;15%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;web.de&lt;/td&gt;&lt;td&gt;14%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;gmail.com&lt;/td&gt;&lt;td&gt;11%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;gmx.de&lt;/td&gt;&lt;td&gt;8%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;hotmail.de&lt;/td&gt;&lt;td&gt;7%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;yahoo.com&lt;/td&gt;&lt;td&gt;6%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;yahoo.de&lt;/td&gt;&lt;td&gt;5%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;live.fr&lt;/td&gt;&lt;td&gt;4%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;o2.pl&lt;/td&gt;&lt;td&gt;3%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;h2 id=&quot;top-level-domain-statistics&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;11&lt;/span&gt;Top level domain statistics&lt;a class=&quot;hash&quot; href=&quot;#top-level-domain-statistics&quot; aria-label=&quot;Link to Top level domain statistics&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;In the TLD statistics, we can see that 30% of the users of Gamigo were using German email providers.
Gamigo is a German service, so we would have expected that. However, the second-largest country in
the statistics is France.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-tlds&quot; class=&quot;chart chart-bars&quot; aria-labelledby=&quot;chart-gamigo-tlds-note&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-tlds-note&quot;&gt;Share of email addresses by top-level domain&lt;/span&gt;&lt;/figcaption&gt;&lt;ol class=&quot;chart-rows&quot;&gt;&lt;li title=&quot;.com: 37%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;.com&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 92.5%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;37%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Germany: 30%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Germany&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 75%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;30%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;France: 17%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;France&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 42.5%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;17%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Poland: 6%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Poland&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 15%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;6%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Italy: 3%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Italy&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 7.5%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;3%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;.net: 3%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;.net&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 7.5%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;3%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Spain: 1%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Spain&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 2.5%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;1%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;United Kingdom: 1%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;United Kingdom&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 2.5%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;1%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Austria: 1%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Austria&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 2.5%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;1%&lt;/span&gt;&lt;/li&gt;&lt;li title=&quot;Russian Federation: 1%&quot;&gt;&lt;span class=&quot;chart-label&quot;&gt;Russian Federation&lt;/span&gt;&lt;span class=&quot;chart-track&quot;&gt;&lt;span class=&quot;chart-bar&quot; style=&quot;--v: 2.5%&quot;&gt;&lt;/span&gt;&lt;/span&gt;&lt;span class=&quot;chart-value&quot;&gt;1%&lt;/span&gt;&lt;/li&gt;&lt;/ol&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 12&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;.com&lt;/td&gt;&lt;td&gt;37%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Germany&lt;/td&gt;&lt;td&gt;30%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;France&lt;/td&gt;&lt;td&gt;17%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Poland&lt;/td&gt;&lt;td&gt;6%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Italy&lt;/td&gt;&lt;td&gt;3%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;.net&lt;/td&gt;&lt;td&gt;3%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Spain&lt;/td&gt;&lt;td&gt;1%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;United Kingdom&lt;/td&gt;&lt;td&gt;1%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Austria&lt;/td&gt;&lt;td&gt;1%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Russian Federation&lt;/td&gt;&lt;td&gt;1%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;h2 id=&quot;email-username-as-passwords&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;12&lt;/span&gt;Email username as passwords&lt;a class=&quot;hash&quot; href=&quot;#email-username-as-passwords&quot; aria-label=&quot;Link to Email username as passwords&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;A total of 1.399.689 hashes (13%) could be cracked using the username part of the email, as the
password. To maximize the efficiency, I removed dot, underscore and hyphen characters from the
emails. This is a surprisingly large number of hashes cracked using the emails only. Note that I do not
have email and hash pairs, but rather a bunch of emails and a bunch of hashes. Each email username
was tested against all the hashes, so a user with the email john@gmail.com would have resulted in
&quot;john&quot; being tried against all the hashes, resulting in multiple cracked hashes.&lt;/p&gt;
&lt;figure id=&quot;chart-gamigo-email-username&quot; class=&quot;chart chart-split&quot; aria-labelledby=&quot;chart-gamigo-email-username-note&quot;&gt;&lt;figcaption class=&quot;chart-head&quot;&gt;&lt;span class=&quot;chart-note&quot; id=&quot;chart-gamigo-email-username-note&quot;&gt;Share of cracked hashes&lt;/span&gt;&lt;/figcaption&gt;&lt;div class=&quot;chart-segments&quot;&gt;&lt;span class=&quot;chart-part chart-part-0&quot; style=&quot;--v: 13%&quot; title=&quot;Email username as password: 13%&quot;&gt;&lt;/span&gt;&lt;span class=&quot;chart-part chart-part-1&quot; style=&quot;--v: 87%&quot; title=&quot;Other: 87%&quot;&gt;&lt;/span&gt;&lt;/div&gt;&lt;ul class=&quot;chart-keys&quot;&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-0&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;Email username as password &lt;span class=&quot;chart-value&quot;&gt;13%&lt;/span&gt;&lt;/li&gt;&lt;li&gt;&lt;span class=&quot;chart-key chart-part-1&quot; aria-hidden=&quot;true&quot;&gt;&lt;/span&gt;Other &lt;span class=&quot;chart-value&quot;&gt;87%&lt;/span&gt;&lt;/li&gt;&lt;/ul&gt;&lt;details class=&quot;chart-data&quot;&gt;&lt;summary&gt;Show data&lt;/summary&gt;&lt;div class=&quot;table-scroll&quot; tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Table 13&quot;&gt;&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th scope=&quot;col&quot;&gt;Category&lt;/th&gt;&lt;th scope=&quot;col&quot;&gt;Share&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Email username as password&lt;/td&gt;&lt;td&gt;13%&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Other&lt;/td&gt;&lt;td&gt;87%&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;&lt;/div&gt;&lt;/details&gt;&lt;/figure&gt;
&lt;h2 id=&quot;conclusion&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;13&lt;/span&gt;Conclusion&lt;a class=&quot;hash&quot; href=&quot;#conclusion&quot; aria-label=&quot;Link to Conclusion&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;It seems that Gamigo, like so many other large online services, failed to implement a good password
policy. Not only were they using raw MD5 as the hashing algorithm, but they also had little to no
password strength requirements for their users, which resulted in over 80% of their user hashes
cracked in about a week.&lt;/p&gt;
&lt;p&gt;The lesson is that you should implement a stronger hashing algorithm and require some sort of
minimum password strength of your users. Gamigo did force a password reset on all users, but many
users probably just chose the same password or a password that closely resembles the old password.
One good thing came of this, and that is Gamigo took responsibility and told their users of the
hack, this greatly minimizes the damage that can potentially happen to the users.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>The power of Wolfram Alpha – now in a .NET API</title>
    <id>https://genbox.github.io/2012/07/power-of-wolfram-alpha-now-in-net-api.html</id>
    <link href="https://genbox.github.io/2012/07/power-of-wolfram-alpha-now-in-net-api.html" />
    <published>2012-07-09T21:45:00.0020000+00:00</published>
    <updated>2012-07-09T23:50:56.9080000+00:00</updated>
    <summary>Introducing WolframAlpha.net, a .NET API for accessing the computational knowledge and query results of Wolfram Alpha.</summary>
    <category term="C#" />
<category term="Networking" />
    <content type="html" xml:base="https://genbox.github.io/2012/07/power-of-wolfram-alpha-now-in-net-api.html">&lt;p&gt;In an ongoing project of mine, I needed access to a good source of information that is free,
powerful and provides some sort of API. My first thought was on
&lt;a href=&quot;https://www.wolframalpha.com/&quot;&gt;Wolfram|Alpha&lt;/a&gt;, a powerful search engine that has thousands of
databases filled with useful information and a really great interpreter that tries to guess what you
mean by the &lt;a href=&quot;https://www.wolframalpha.com/input/?i=22th+president+of+the+USA&quot;&gt;22nd president of the
USA&lt;/a&gt;. If you have not yet tried
Wolfram|Alpha, I highly encourage you to take a look at it.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;/assets/images/posts/f0f423bdb0262177007e.webp?v=a4f2859fcffd02a4&quot; width=&quot;672&quot; height=&quot;350&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; alt=&quot;Wolfram Alpha results for Google stock, showing a price-history chart and summary statistics.&quot;&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;The results from Wolfram|Alpha when looking up &quot;Google stock&quot;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Wolfram|Alpha has a REST based API that outputs beautiful XML data ready to parse. They really put
some work into making it easy to use, so I thought it would be a no-brainer to find an existing C#
library that would parse it and give me objects that I could work with. However, it turns out that
there are only a few of them, and only a fraction of those support the 2.0 API that Wolfram|Alpha
&lt;a href=&quot;https://blog.wolframalpha.com/2011/01/20/knowledge-based-computing-and-version-20-of-the-wolframalpha-api/&quot;&gt;introduced&lt;/a&gt;
back in January 2011. I decided to code an API myself that could be easily extended with new
features and would provide me with the full API.&lt;/p&gt;
&lt;p&gt;I present to you, the all new &lt;a href=&quot;https://github.com/Genbox/WolframAlpha&quot;&gt;WolframAlpha.net API&lt;/a&gt;.
It is based on the open source &lt;a href=&quot;https://github.com/restsharp/RestSharp&quot;&gt;RestSharp&lt;/a&gt; framework for
parsing the XML data, so a big thanks goes out to the guys behind it for an awesome job.&lt;/p&gt;
</content>
</entry>
<entry>
    <title>A complete guide to encrypting and signing emails</title>
    <id>https://genbox.github.io/2012/06/a-complete-guide-to-encrypting-and.html</id>
    <link href="https://genbox.github.io/2012/06/a-complete-guide-to-encrypting-and.html" />
    <published>2012-06-25T02:52:00.0000000+00:00</published>
    <updated>2016-12-03T09:51:20.8020000+00:00</updated>
    <summary>Explaining email signing and encryption with OpenPGP and S/MIME, followed by setup instructions and troubleshooting notes.</summary>
    <category term="Security" />
<category term="Networking" />
    <content type="html" xml:base="https://genbox.github.io/2012/06/a-complete-guide-to-encrypting-and.html">&lt;p&gt;A friend of mine recently asked how PGP email encryption works, and I remember how hard it was for
me to understand how it all works, so I decided to write this guide to email security.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;If you just need to know how to sign and encrypt emails, go to headline 3.1 to use OpenPGP or
headline 3.4 to use S/MIME&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id=&quot;email-security-an-overview&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;01&lt;/span&gt;1. Email security - an overview&lt;a class=&quot;hash&quot; href=&quot;#email-security-an-overview&quot; aria-label=&quot;Link to 1. Email security - an overview&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Whenever you write an email, it is sent to the receiver in clear text. Email services like Gmail
support TLS (Transport Layer Security) that encrypts the message in transition, but the email is
still stored in clear text on the end user’s machine. In information security, we have something
called the &lt;a href=&quot;https://en.wikipedia.org/wiki/Information_security#Key_concepts&quot;&gt;CIA triad&lt;/a&gt; that consists
of 3 areas: Confidentiality, Integrity and Availability. In this post, we will only focus on the
first 2 areas, and we will start with a subsection of integrity called authenticity.&lt;/p&gt;
&lt;h3 id=&quot;email-authenticity&quot;&gt;1.1 Email authenticity&lt;a class=&quot;hash&quot; href=&quot;#email-authenticity&quot; aria-label=&quot;Link to 1.1 Email authenticity&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;When you receive an email from someone, how can you be sure of the following:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The email was not altered in transition.&lt;/li&gt;
&lt;li&gt;The email was sent from the right person.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;With ordinary emails, you can&apos;t be sure of either 1 or 2. A hacker intercepting the message using
&lt;a href=&quot;https://en.wikipedia.org/wiki/Man-in-the-middle_attack&quot;&gt;MitM attacks&lt;/a&gt; or a rogue network
administrator might have altered the message in transition. Spammers are also known for using
&lt;a href=&quot;https://en.wikipedia.org/wiki/Forgery&quot;&gt;spoofed&lt;/a&gt; email addresses to get their message out to people.
They might write advertising@target.com as the sender of the email to make the message more
effective.&lt;/p&gt;
&lt;p&gt;This is where &lt;a href=&quot;https://en.wikipedia.org/wiki/Digital_signature&quot;&gt;digital signatures&lt;/a&gt; come in. We can
solve both problems by signing our emails using &lt;a href=&quot;https://en.wikipedia.org/wiki/Public-key_cryptography&quot;&gt;asymmetric cryptographic
keys&lt;/a&gt;, also known as public/private keys.&lt;/p&gt;
&lt;h3 id=&quot;publicprivate-keys&quot;&gt;1.2 Public/private keys&lt;a class=&quot;hash&quot; href=&quot;#publicprivate-keys&quot; aria-label=&quot;Link to 1.2 Public/private keys&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Using some clever math and cryptographic functions, we can create 2 keys that have different
functionality. The keys are used differently depending on what you are trying to achieve. When we
are signing a message, we keep one key to ourselves (the private key) and send the other key (the
public key) to anyone that needs to verify our email.&lt;/p&gt;
&lt;figure class=&quot;diagram diagram-panel signature-diagram&quot; role=&quot;img&quot; aria-label=&quot;Alice signs the message I will pay $500 using her private key. She sends the message and signature to Bob. Bob verifies them using Alice&apos;s public key, producing a valid or invalid result. Signing does not encrypt the message.&quot;&gt;
    &lt;div aria-hidden=&quot;true&quot;&gt;
        &lt;div class=&quot;signature-stage&quot;&gt;
            &lt;div class=&quot;signature-heading&quot;&gt;Alice&lt;span&gt;Sender&lt;/span&gt;&lt;/div&gt;
            &lt;div class=&quot;signature-inputs&quot;&gt;
                &lt;div class=&quot;signature-input&quot;&gt;&lt;small&gt;Message&lt;/small&gt;I will pay $500&lt;/div&gt;
                &lt;div class=&quot;signature-input&quot;&gt;&lt;small&gt;Kept secret&lt;/small&gt;Alice’s private key&lt;/div&gt;
            &lt;/div&gt;
            &lt;div class=&quot;signature-arrow&quot;&gt;↓&lt;/div&gt;
            &lt;div class=&quot;signature-operation&quot;&gt;Sign&lt;small&gt;Produce a digital signature&lt;/small&gt;&lt;/div&gt;
        &lt;/div&gt;
        &lt;div class=&quot;signature-transfer&quot;&gt;Send message + signature ↓&lt;/div&gt;
        &lt;div class=&quot;signature-stage&quot;&gt;
            &lt;div class=&quot;signature-heading&quot;&gt;Bob&lt;span&gt;Receiver&lt;/span&gt;&lt;/div&gt;
            &lt;div class=&quot;signature-inputs&quot;&gt;
                &lt;div class=&quot;signature-input&quot;&gt;&lt;small&gt;Received&lt;/small&gt;Message + signature&lt;/div&gt;
                &lt;div class=&quot;signature-input&quot;&gt;&lt;small&gt;Trusted as Alice’s&lt;/small&gt;Alice’s public key&lt;/div&gt;
            &lt;/div&gt;
            &lt;div class=&quot;signature-arrow&quot;&gt;↓&lt;/div&gt;
            &lt;div class=&quot;signature-operation&quot;&gt;Verify&lt;small&gt;Result: valid or invalid&lt;/small&gt;&lt;/div&gt;
        &lt;/div&gt;
    &lt;/div&gt;
    &lt;figcaption aria-hidden=&quot;true&quot;&gt;Signing protects authenticity and integrity. It does not hide the message’s contents.&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;p&gt;A signature algorithm uses your private key to sign the message, typically through a
&lt;a href=&quot;https://en.wikipedia.org/wiki/Cryptographic_hash_function&quot;&gt;cryptographic hash&lt;/a&gt; of its contents.
The message and signature are sent together. The receiver verifies the signature against the
received message using your public key.&lt;/p&gt;
&lt;p&gt;If the message or signature has been altered, verification fails. Successful verification shows
that the signature matches the message and public key. To attribute it to you, the receiver also
needs to trust that the public key belongs to you. As &lt;a href=&quot;https://csrc.nist.gov/glossary/term/digital_signature&quot;&gt;NIST explains&lt;/a&gt;,
digital signatures provide authenticity and integrity; signing does not encrypt the message.&lt;/p&gt;
&lt;h3 id=&quot;email-confidentiality&quot;&gt;1.3 Email confidentiality&lt;a class=&quot;hash&quot; href=&quot;#email-confidentiality&quot; aria-label=&quot;Link to 1.3 Email confidentiality&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Encryption serves a different purpose: it hides the contents of an email so that only the intended
receiver can read it. This time we need to get the public key of the receiver of the email before we send the
email. We simply encrypt the email using the receiver’s public key, and only the receiver with the
private key can decrypt the message.&lt;/p&gt;
&lt;p&gt;Remember that encrypting the email using the public key of the receiver does not prove the
authenticity of the email, but merely makes it unreadable to everyone else. If we need both
authenticity and confidentiality, we need to both sign and encrypt the email.&lt;/p&gt;
&lt;h2 id=&quot;the-security-standards&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;02&lt;/span&gt;2. The security standards&lt;a class=&quot;hash&quot; href=&quot;#the-security-standards&quot; aria-label=&quot;Link to 2. The security standards&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;When it comes to email security, we have 2 major standards competing on the market.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;OpenPGP&lt;/li&gt;
&lt;li&gt;S/MIME&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;openpgp&quot;&gt;2.1 OpenPGP&lt;a class=&quot;hash&quot; href=&quot;#openpgp&quot; aria-label=&quot;Link to 2.1 OpenPGP&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;OpenPGP stands for Open Pretty Good Privacy and is a format based on the original PGP format. It
uses its own standard for describing encrypted email content. OpenPGP supports signatures using algorithms such as DSA or RSA, public-key encryption using RSA or
&lt;a href=&quot;https://en.wikipedia.org/wiki/ElGamal&quot;&gt;ElGamal&lt;/a&gt;, and message encryption using symmetric algorithms
such as &lt;a href=&quot;https://en.wikipedia.org/wiki/3des&quot;&gt;TripleDES&lt;/a&gt; or AES. The format is sent
over the Internet using multipart/encrypted &lt;a href=&quot;https://en.wikipedia.org/wiki/MIME&quot;&gt;MIME&lt;/a&gt; format. It is
widely supported by the open source community and has been built into a bunch of mail applications,
most notably the Mozilla Thunderbird client with the Enigma plugin.&lt;/p&gt;
&lt;h3 id=&quot;smime&quot;&gt;2.2 S/MIME&lt;a class=&quot;hash&quot; href=&quot;#smime&quot; aria-label=&quot;Link to 2.2 S/MIME&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;S/MIME stands for Secure/Multipurpose Internet Mail Extensions, and while it does the same thing as
OpenPGP, the two formats are in no way interoperable and messages created in one format can&apos;t be
read in the other. S/MIME supports signatures using algorithms such as DSA or
&lt;a href=&quot;https://en.wikipedia.org/wiki/RSA_%28algorithm%29&quot;&gt;RSA&lt;/a&gt;, key agreement using
&lt;a href=&quot;https://en.wikipedia.org/wiki/Diffie-Hellman&quot;&gt;Diffie-Hellman&lt;/a&gt;, and message encryption using symmetric
algorithms such as TripleDES or AES.
The MIME format is application/pkcs7-mime and in most email clients that do not understand the
format, you will get an attachment called &apos;smime.p7m&apos; that contains the signed and/or encrypted
email.&lt;/p&gt;
&lt;h3 id=&quot;the-format-problem&quot;&gt;2.3 The format problem&lt;a class=&quot;hash&quot; href=&quot;#the-format-problem&quot; aria-label=&quot;Link to 2.3 The format problem&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;OpenPGP and S/MIME use different formats for signing and encrypting messages, and thus, they need
different applications. The lack of traction in the general population of Internet users, means that
there are only a few good tools out there that give us email signing and encryption for free. Most
tools only support one of the two protocols and most popular online email providers do not support
either of the two.&lt;/p&gt;
&lt;p&gt;The biggest differences between the two formats are the concepts on which they are built. OpenPGP is
built on the &lt;a href=&quot;https://en.wikipedia.org/wiki/Web_of_trust&quot;&gt;WOT&lt;/a&gt; concept, a distributed approach to
sharing keys, while S/MIME is built on &lt;a href=&quot;https://en.wikipedia.org/wiki/Public_key_infrastructure&quot;&gt;PKI&lt;/a&gt;
a centralized key sharing concept. The biggest difference between the two concepts is that PKI
relies on a Certification Authority (CA) that signs the keys. Everyone trusts the CA, and so all the
keys the CA signs are trusted. However, in order to get your key signed by a CA, you need to pay a
third party CA. In OpenPGP there is no such thing as a CA and each user can sign another user’s key
to endorse the binding between the key and the user. It is all a matter of trust, but it becomes
important when you need to make sure that the public key of a user is the right public key, and not
one a hacker made to impersonate a user.&lt;/p&gt;
&lt;h2 id=&quot;the-practical-approach&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;03&lt;/span&gt;3. The practical approach&lt;a class=&quot;hash&quot; href=&quot;#the-practical-approach&quot; aria-label=&quot;Link to 3. The practical approach&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;p&gt;Now that we know what we need to get us going, let&apos;s take a look at how to implement digital
signatures using both OpenPGP and S/MIME.&lt;/p&gt;
&lt;h3 id=&quot;openpgp-setup&quot;&gt;3.1 OpenPGP setup&lt;a class=&quot;hash&quot; href=&quot;#openpgp-setup&quot; aria-label=&quot;Link to 3.1 OpenPGP setup&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;GnuPG stands for GNU Privacy Guard and it supports both OpenPGP and S/MIME in the latest versions (since 2.0). Gpg4Win is a Windows port of the application that can be &lt;a href=&quot;https://gpg4win.org/&quot;&gt;found
here&lt;/a&gt;.&lt;br&gt;
To create an OpenPGP certificate, do the following:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href=&quot;https://gpg4win.org/download.html&quot;&gt;Download&lt;/a&gt; the Gpg4Win full installation and install it&lt;/li&gt;
&lt;li&gt;Open Kleopatra, go to &lt;em&gt;File&lt;/em&gt;-&amp;gt;&lt;em&gt;New Certificate&lt;/em&gt; and click &quot;Create a personal OpenPGP key pair&quot;&lt;/li&gt;
&lt;li&gt;Enter name and email, click &lt;em&gt;Next&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Make sure the information is correct. Click &lt;em&gt;Create Key&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Enter a &lt;a href=&quot;https://www.grc.com/haystack.htm&quot;&gt;secure&lt;/a&gt; pass phrase. Security is indicated by the
green bar.&lt;/li&gt;
&lt;li&gt;Click &lt;em&gt;Finish&lt;/em&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Now you have a certificate with a public/private key pair.&lt;/p&gt;
&lt;h3 id=&quot;using-openpgp-with-mozilla-thunderbird&quot;&gt;3.2 Using OpenPGP with Mozilla Thunderbird&lt;a class=&quot;hash&quot; href=&quot;#using-openpgp-with-mozilla-thunderbird&quot; aria-label=&quot;Link to 3.2 Using OpenPGP with Mozilla Thunderbird&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Mozilla Thunderbird supports S/MIME out of the box, but does not support OpenPGP. In order to get
that support, you will have to use Enigmail. Here is how you set up OpenPGP in Thunderbird:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href=&quot;https://www.enigmail.net/&quot;&gt;Download&lt;/a&gt; and install Enigmail. &lt;strong&gt;Note:&lt;/strong&gt;
You can do this from inside Thunderbird. Click &lt;em&gt;Tools&lt;/em&gt;-&amp;gt; &lt;em&gt;Add-ons&lt;/em&gt; and install Enigmail.&lt;/li&gt;
&lt;li&gt;Open Thunderbird and right-click on the account you would like to use OpenPGP with and click &lt;em&gt;Settings&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Find the &quot;OpenPGP Security&quot; settings and check the box in &quot;Enable OpenPGP support (Enigmail) for
this identity&quot;.&lt;/li&gt;
&lt;li&gt;Click &lt;em&gt;Select Key&lt;/em&gt; and choose the key you created earlier in Kleopatra.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;/assets/images/posts/6e0c5c8749aa5177e85a.png?v=9b92975f1bbe32ce&quot; width=&quot;377&quot; height=&quot;106&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; alt=&quot;Thunderbird message toolbar with the OpenPGP controls highlighted.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Whenever you write an email, you will get some options in the toolbar with OpenPGP. There you can
select whether to sign or encrypt the email. It is that simple to get OpenPGP going in Mozilla
Thunderbird.&lt;/p&gt;
&lt;h3 id=&quot;smime-setup&quot;&gt;3.4 S/MIME setup&lt;a class=&quot;hash&quot; href=&quot;#smime-setup&quot; aria-label=&quot;Link to 3.4 S/MIME setup&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;In order to get email signing and encryption with S/MIME, we need a digital signature. Mozilla has a
&lt;a href=&quot;https://kb.mozillazine.org/Getting_an_SMIME_certificate&quot;&gt;great article&lt;/a&gt; that lists the sites where
you can get a free certificate. Scroll down to &quot;Self-signed certificates&quot; to see how you can create
one yourself. An alternative is to use &lt;a href=&quot;https://www.mobilefish.com/services/ssl_certificates/ssl_certificates.php&quot;&gt;Mobile
Fish&lt;/a&gt;, which is what I did
with this guide.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Go to &lt;a href=&quot;https://www.mobilefish.com/services/ssl_certificates/ssl_certificates.php&quot;&gt;Mobile Fish&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Enter Country, State name, Locality name and Common name. Press the question mark next to the
item to show more info about the field.&lt;/li&gt;
&lt;li&gt;Enter your email in the email field. Note: This email has to be the email you wish to give a
digital signature.&lt;/li&gt;
&lt;li&gt;Set the pass-phrase to something &lt;a href=&quot;https://www.grc.com/haystack.htm&quot;&gt;secure&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Set the bit length to 1024 or more&lt;/li&gt;
&lt;li&gt;Set the number of days to 365&lt;/li&gt;
&lt;li&gt;Click &lt;em&gt;Generate&lt;/em&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Now you should be presented with a bunch of fields. You need to download the following:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The Self-Signed SSL Certificate - Name it selfsigned.cert&lt;/li&gt;
&lt;li&gt;The Private Key - Name it private.key&lt;/li&gt;
&lt;li&gt;The Public Key - Name it public.key&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Now we have a certificate, but in order for Windows to use it, we need to convert it to the PFX
format. Here is how you do it:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Download the &lt;a href=&quot;https://slproweb.com/products/Win32OpenSSL.html&quot;&gt;OpenSSL library for Windows&lt;/a&gt;
(Direct link: &lt;a href=&quot;https://web.archive.org/web/20121019130525/http://slproweb.com/download/Win32OpenSSL_Light-1_0_1c.exe&quot;&gt;Win32 OpenSSL v1.0.1c
Light&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;Install the library with standard settings.&lt;/li&gt;
&lt;li&gt;Copy the 3 files (private.key, public.key and selfsigned.cert) to the
&apos;C:\OpenSSL-Win32\bin&apos; folder&lt;/li&gt;
&lt;li&gt;Go to the Start menu, enter &apos;cmd&apos; in the search and press &lt;em&gt;Enter&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Write &apos;cd C:\OpenSSL-Win32\bin&apos; and press &lt;em&gt;Enter&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Write &apos;openssl pkcs12 -inkey private.key -in selfsigned.cert -export -out mycert.pfx&apos; and press &lt;em&gt;Enter&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;You should now be prompted with the private key password and an export password. Enter the same
password you chose on Mobile Fish.&lt;/li&gt;
&lt;li&gt;You should now have a mycert.pfx inside the
&apos;C:\OpenSSL-Win32\bin&apos; folder.&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id=&quot;using-smime-in-windows-live-mail&quot;&gt;3.5 Using S/MIME in Windows Live Mail&lt;a class=&quot;hash&quot; href=&quot;#using-smime-in-windows-live-mail&quot; aria-label=&quot;Link to 3.5 Using S/MIME in Windows Live Mail&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Windows Live Mail uses the Certificate Store inside Microsoft Windows. The easiest way to use a
certificate in Windows Live Mail is to simply double-click the mycert.pfx file inside
&apos;C:\OpenSSL-Win32\bin&apos; and follow the wizard. When you get
prompted with a password, enter the private key password.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; If you are using a self-signed certificate, you need to run the wizard one more time. This
time, when you get to the wizard page where you choose the Certificate Store, select &quot;Trusted Root
Certification Authorities&quot;. This is necessary as you have no third party CA that signed the
certificate, otherwise the certificate will not work!&lt;/p&gt;
&lt;p&gt;Now that the certificate is installed, you only need to associate it with your Windows Live Mail
account.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Open Windows Live Mail&lt;/li&gt;
&lt;li&gt;Right-click the account with the email that is in the certificate and click &lt;em&gt;Properties&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Go to the &lt;em&gt;Security&lt;/em&gt; tab and click the &lt;em&gt;Select&lt;/em&gt; button under &quot;Signing certificate&quot;. The
certificate you just installed should be selected. Do the same with the &quot;Encrypting preferences&quot;
if Windows Live Mail did not do it for you already.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;/assets/images/posts/87bdb4c0d58de3b06740.png?v=68f3bed4371af451&quot; width=&quot;273&quot; height=&quot;162&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; alt=&quot;Windows Live Mail toolbar showing the Encrypt and Digitally sign buttons.&quot;&gt;&lt;/p&gt;
&lt;p&gt;Whenever you write an email in Windows Live Mail, you can go to the &lt;em&gt;Options&lt;/em&gt; tab and select &lt;em&gt;Encrypt&lt;/em&gt; and &lt;em&gt;Digitally Sign&lt;/em&gt;. When you send the email, it will be encrypted and signed.&lt;/p&gt;
&lt;h3 id=&quot;using-smime-in-thunderbird&quot;&gt;3.6 Using S/MIME in Thunderbird&lt;a class=&quot;hash&quot; href=&quot;#using-smime-in-thunderbird&quot; aria-label=&quot;Link to 3.6 Using S/MIME in Thunderbird&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Thunderbird uses the Mozilla Certificate Store, so we need to import the certificate into that
store.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; If the certificate is self-signed, you need to follow &lt;a href=&quot;https://kb.mozillazine.org/Installing_an_SMIME_certificate#Installing_a_Self-Signed_SMIME_Certificate_for_Your_Own_Identity&quot;&gt;this short
guide&lt;/a&gt;
first.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Open Thunderbird, go to &lt;em&gt;Tools&lt;/em&gt; -&amp;gt; &lt;em&gt;Options&lt;/em&gt; -&amp;gt; &lt;em&gt;Advanced&lt;/em&gt; -&amp;gt; &lt;em&gt;Certificates&lt;/em&gt; -&amp;gt; &lt;em&gt;View
Certificates&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Under the &quot;Your Certificates&quot; tab, click on &lt;em&gt;Import&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Select the &quot;mycert.pfx&quot; file inside the
&apos;C:\OpenSSL-Win32\bin&apos; folder.&lt;/li&gt;
&lt;li&gt;Enter the private key password.&lt;/li&gt;
&lt;li&gt;Click &lt;em&gt;OK&lt;/em&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Now you only need to associate the certificate with the account.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Open Thunderbird, right-click on the account you want to use.&lt;/li&gt;
&lt;li&gt;Click &lt;em&gt;Settings&lt;/em&gt; and go to the &quot;Security&quot; properties.&lt;/li&gt;
&lt;li&gt;Click &lt;em&gt;Select&lt;/em&gt; next to &quot;Digital Signing&quot; and select your certificate.&lt;/li&gt;
&lt;li&gt;When it asks you to use the same certificate for encryption, click &lt;em&gt;Yes&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;Click &lt;em&gt;OK&lt;/em&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Now you have S/MIME encryption in Thunderbird.&lt;/p&gt;
&lt;h3 id=&quot;using-smime-in-gmail&quot;&gt;3.7 Using S/MIME in Gmail&lt;a class=&quot;hash&quot; href=&quot;#using-smime-in-gmail&quot; aria-label=&quot;Link to 3.7 Using S/MIME in Gmail&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Gmail does not have native S/MIME support even though there have been countless requests by
concerned users. One might argue that the end-to-end encryption concept will break if Gmail is
responsible for decrypting the messages, but it would heighten the security nonetheless.&lt;/p&gt;
&lt;p&gt;If you are running Firefox, you can try the &lt;a href=&quot;https://web.archive.org/web/20120717022655/http://www.penango.com:80/pricing.shtml&quot;&gt;Penango plugin&lt;/a&gt;,
it is free for Gmail and Firefox users. Firefox uses the Mozilla Certificate Store, the same as
Thunderbird, so if you are using Thunderbird and Firefox together, Penango will make it seamlessly
work together.&lt;/p&gt;
&lt;h2 id=&quot;notes&quot;&gt;&lt;span class=&quot;num&quot; aria-hidden=&quot;true&quot;&gt;04&lt;/span&gt;4. Notes&lt;a class=&quot;hash&quot; href=&quot;#notes&quot; aria-label=&quot;Link to 4. Notes&quot;&gt;#&lt;/a&gt;&lt;/h2&gt;
&lt;h3 id=&quot;windows-live-mail-does-not-list-any-certificates&quot;&gt;4.1 Windows Live Mail does not list any certificates&lt;a class=&quot;hash&quot; href=&quot;#windows-live-mail-does-not-list-any-certificates&quot; aria-label=&quot;Link to 4.1 Windows Live Mail does not list any certificates&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;Make sure that the email inside the certificate is identical to the account you are trying to bind
with. Also make sure that the certificate is trusted by a CA. If the certificate is self-signed, add
the certificate to the Trusted Certificate Authority certificate store on your computer.&lt;/p&gt;
&lt;h3 id=&quot;penango-tells-me-that-the-email-is-unreliable&quot;&gt;4.2 Penango tells me that the email is unreliable&lt;a class=&quot;hash&quot; href=&quot;#penango-tells-me-that-the-email-is-unreliable&quot; aria-label=&quot;Link to 4.2 Penango tells me that the email is unreliable&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;This is because Penango uses S/MIME that is based on the PKI. If your certificate is not trusted by
a third party, it will give you a warning telling you that the certificate is self-signed or not
issued by a trusted CA.&lt;/p&gt;
&lt;h3 id=&quot;my-application-is-not-described&quot;&gt;4.3 My application is not described&lt;a class=&quot;hash&quot; href=&quot;#my-application-is-not-described&quot; aria-label=&quot;Link to 4.3 My application is not described&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;A lot of applications have S/MIME support, however, if your favorite mail application does not
support S/MIME and you get the smime.p7m attachment, you can just use the
&lt;a href=&quot;https://web.archive.org/web/20130617114938/http://www.cryptigo.eu/p7mViewer/&quot;&gt;p7mViewer&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id=&quot;openpgp-for-apple&quot;&gt;4.4 OpenPGP for Apple&lt;a class=&quot;hash&quot; href=&quot;#openpgp-for-apple&quot; aria-label=&quot;Link to 4.4 OpenPGP for Apple&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;If you are using a Mac, you can install a &lt;a href=&quot;https://gpgtools.org/gpgmail/index.html&quot;&gt;plugin to Apple
Mail&lt;/a&gt; that enables OpenPGP support.&lt;/p&gt;
&lt;h3 id=&quot;openpgp-for-gmail-in-chrome&quot;&gt;4.5 OpenPGP for Gmail in Chrome&lt;a class=&quot;hash&quot; href=&quot;#openpgp-for-gmail-in-chrome&quot; aria-label=&quot;Link to 4.5 OpenPGP for Gmail in Chrome&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;A research team have implemented OpenPGP as JS in the form of a &lt;a href=&quot;https://web.archive.org/web/20130127210854/http://gpg4browsers.recurity.com:80/&quot;&gt;Chrome
extension&lt;/a&gt;. It is called GPG4Browsers and advertises support for
OpenPGP in Gmail.&lt;/p&gt;
&lt;h3 id=&quot;openpgp-in-outlook&quot;&gt;4.6 OpenPGP in Outlook&lt;a class=&quot;hash&quot; href=&quot;#openpgp-in-outlook&quot; aria-label=&quot;Link to 4.6 OpenPGP in Outlook&quot;&gt;#&lt;/a&gt;&lt;/h3&gt;
&lt;p&gt;If you are using Outlook 2010, you might want to try out the &lt;a href=&quot;https://github.com/Meddington/OutlookPrivacyPlugin&quot;&gt;Outlook Privacy
Plugin&lt;/a&gt;.&lt;/p&gt;
</content>
</entry>
</feed>