<blockquote>
Time moves in one direction, memory in another. We are that strange species that constructs artifacts intended to counter the natural flow of forgetting.
William Gibson
</blockquote>
This website is a small collection of things I want to remember. Since I'm a software engineer, most of it is related to software. It was inspired by Simon Willison's post <a href="https://simonwillison.net/2021/May/2/one-year-of-tils/">One year of TILs</a>.
If you liked reading my notes, you might like my <a href="https://blog.nindalf.com/">blog</a>.
<img src="/book-cover.jpg" alt="O&#x27;RLY? book cover">


Time moves in one direction, memory in another. We are that strange species that constructs artifacts intended to counter the natural flow of forgetting.
William Gibson

This website is a small collection of things I want to remember. Since I'm a software engineer, most of it is related to software. It was inspired by Simon Willison's post One year of TILs.
If you liked reading my notes, you might like my blog.


Krishna's notes

<ul>
<li><a href="#checklist">Checklist</a></li>
<li><a href="#primitives">Primitives</a></li>
<li><a href="#clarifying-questions">Clarifying questions</a>
<ul>
<li><a href="#rate-limiter">Rate Limiter</a></li>
<li><a href="#key-value-store">Key Value Store</a></li>
<li><a href="#id-generator">ID generator</a></li>
<li><a href="#url-shortener">URL shortener</a></li>
<li><a href="#web-crawler">Web Crawler</a></li>
<li><a href="#notification-system">Notification system</a></li>
<li><a href="#news-feed">News Feed</a></li>
<li><a href="#chat-system">Chat system</a></li>
<li><a href="#search-autocomplete">Search autocomplete</a></li>
<li><a href="#youtube">Youtube</a></li>
<li><a href="#google-drive">Google drive</a></li>
</ul>
</li>
<li><a href="#final-designs">Final designs</a>
<ul>
<li><a href="#rate-limiter-1">Rate Limiter</a></li>
<li><a href="#key-value-store-1">Key Value Store</a></li>
<li><a href="#url-shortener-1">URL shortener</a></li>
<li><a href="#web-crawler-1">Web Crawler</a></li>
<li><a href="#notification-service">Notification Service</a></li>
<li><a href="#news-feed-1">News Feed</a></li>
<li><a href="#chat-service">Chat service</a></li>
<li><a href="#search-autocomplete-1">Search Autocomplete</a></li>
</ul>
</li>
</ul>
<h2 id="checklist"><a href="#checklist" aria-hidden="true" tabindex="-1"></a>Checklist</h2>
Go through this checklist in every interview
<ul>
<li>Clarify requirements
<ul>
<li>Write QPS, average and peak (peak is 2x average)</li>
<li>Read QPS (approx 10x write QPS)</li>
<li>Total storage required</li>
<li>Latency requirement</li>
<li>Consistency vs Availability</li>
</ul>
</li>
<li>High level diagram that matches requirements. Get buy-in</li>
<li>Deep Dive
<ul>
<li>API</li>
<li>Database schema</li>
</ul>
</li>
<li>Wrap up
<ul>
<li>Error cases</li>
<li>Potential bottlenecks</li>
</ul>
</li>
<li>Extra credit
<ul>
<li>Cost (development vs ongoing)</li>
<li>Logging, metrics, alerts</li>
<li>Automation - CI, CD</li>
<li>i18n</li>
<li>CDN</li>
<li>RPC</li>
</ul>
</li>
</ul>
<hr>
<h2 id="primitives"><a href="#primitives" aria-hidden="true" tabindex="-1"></a>Primitives</h2>
<ul>
<li>Load balancer</li>
<li>Gateway (for ratelimiting, authentication)</li>
<li>Server (optionally horizonatally scaled)</li>
<li>Database
<ul>
<li>SQL vs NoSQL, sharding strategy.</li>
<li>If SQL, indices</li>
</ul>
</li>
<li>Cache (optionally clustered/sharded)</li>
<li>Message queue, with workers</li>
<li>CDN</li>
<li>Lambdas</li>
</ul>
Be explicit about replication
<hr>
<h2 id="clarifying-questions"><a href="#clarifying-questions" aria-hidden="true" tabindex="-1"></a>Clarifying questions</h2>
Good for any problem
<ul>
<li>Is this a mobile app? Or a web app? Or both?</li>
<li>What are the most important features for the product?</li>
<li>DAU, MAU? Derive peak/avg QPS from this.</li>
<li>Is there any limit per request/stored object? Derive bandwidth/storage requirements from this.</li>
<li>Will the system work in a distributed environment?</li>
<li>Where are our users? Geographical constraints, as well as i18n.</li>
<li>Can we leverage some of the existing cloud infrastructures provided by Amazon, Google, or Microsoft?</li>
</ul>
<h3 id="rate-limiter"><a href="#rate-limiter" aria-hidden="true" tabindex="-1"></a>Rate Limiter</h3>
<ul>
<li>Does the rate limiter throttle API requests based on IP, the user ID, or other properties?</li>
<li>Is the rate limiter a separate service or should it be implemented in application code?</li>
<li>Do we need to inform users who are throttled?</li>
</ul>
<h3 id="key-value-store"><a href="#key-value-store" aria-hidden="true" tabindex="-1"></a>Key Value Store</h3>
Not questions, but these are the constraints
<ul>
<li>Scalable</li>
<li>Available</li>
<li>Tunable consistency</li>
<li>Auto scaling</li>
<li>Size of key value pair - 10kB</li>
<li>Low latency</li>
</ul>
<h3 id="id-generator"><a href="#id-generator" aria-hidden="true" tabindex="-1"></a>ID generator</h3>
<ul>
<li>What are the characteristics of unique IDs?</li>
<li>For each new record, does ID increment by 1?</li>
<li>Do IDs only contain numerical values?</li>
<li>What is the ID length requirement?</li>
</ul>
<h3 id="url-shortener"><a href="#url-shortener" aria-hidden="true" tabindex="-1"></a>URL shortener</h3>
<ul>
<li>Can you give an example of how a URL shortener work?</li>
<li>How long is the shortened URL?</li>
<li>What characters are allowed in the shortened URL?</li>
<li>Can shortened URLs be deleted or updated?</li>
</ul>
<h3 id="web-crawler"><a href="#web-crawler" aria-hidden="true" tabindex="-1"></a>Web Crawler</h3>
<ul>
<li>What is the main purpose of the crawler? Is it used for search engine indexing, data mining, or something else?</li>
<li>How many web pages does the web crawler collect per month?</li>
<li>What content types are included? HTML only or other content types such as PDFs and images as well?</li>
<li>Shall we consider newly added or edited web pages?</li>
<li>Do we need to store HTML pages crawled from the web?</li>
<li>How do we handle web pages with duplicate content?</li>
</ul>
<h3 id="notification-system"><a href="#notification-system" aria-hidden="true" tabindex="-1"></a>Notification system</h3>
<ul>
<li>What types of notifications does the system support?</li>
<li>Is it a real-time system? (ie, are delays allowed?)</li>
<li>What triggers notifications?</li>
<li>Will users be able to opt-out?</li>
</ul>
<h3 id="news-feed"><a href="#news-feed" aria-hidden="true" tabindex="-1"></a>News Feed</h3>
<ul>
<li>Is the news feed sorted in reverse chronological order or a particular order? The particular order means each post is given a different weight. For instance, posts from your close friends are more important than posts from a group.</li>
<li>How many friends can a user have?</li>
<li>Can feed contain images, videos, or just text?</li>
</ul>
<h3 id="chat-system"><a href="#chat-system" aria-hidden="true" tabindex="-1"></a>Chat system</h3>
<ul>
<li>What kind of chat app shall we design? 1 on 1 or group based?</li>
<li>For group chat, what is the group member limit?</li>
<li>Is end-to-end encryption required?</li>
<li>How long shall we store the chat history? How about not storing chat history?</li>
</ul>
<h3 id="search-autocomplete"><a href="#search-autocomplete" aria-hidden="true" tabindex="-1"></a>Search autocomplete</h3>
<ul>
<li>Is the matching only supported at the beginning of a search query or in the middle as well?</li>
<li>How many autocomplete suggestions should the system return?</li>
<li>What criteria should decide the ordering of the suggestions?</li>
<li>Does the system support spell check?</li>
<li>Are search queries in English?</li>
<li>Do we allow capitalization and special characters?</li>
</ul>
<h3 id="youtube"><a href="#youtube" aria-hidden="true" tabindex="-1"></a>Youtube</h3>
<ul>
<li>What are the supported video resolutions?</li>
</ul>
<h3 id="google-drive"><a href="#google-drive" aria-hidden="true" tabindex="-1"></a>Google drive</h3>
<ul>
<li>What are the supported file formats?</li>
</ul>
<hr>
<h2 id="final-designs"><a href="#final-designs" aria-hidden="true" tabindex="-1"></a>Final designs</h2>
<h3 id="rate-limiter-1"><a href="#rate-limiter-1" aria-hidden="true" tabindex="-1"></a>Rate Limiter</h3>
<details>
 <summary>Final design</summary>
 <img src="/rate-limiter.png">
</details>
<hr>
<h3 id="key-value-store-1"><a href="#key-value-store-1" aria-hidden="true" tabindex="-1"></a>Key Value Store</h3>
<details>
 <summary>Final design</summary>
 <img src="/kv-write.png">
 <img src="/kv-read.png">
 <img src="/kv-read-uncached.png">
</details>
<hr>
<h3 id="url-shortener-1"><a href="#url-shortener-1" aria-hidden="true" tabindex="-1"></a>URL shortener</h3>
<details>
 <summary>Final design</summary>
 <img src="/url-shortener.png">
</details>
<hr>
<h3 id="web-crawler-1"><a href="#web-crawler-1" aria-hidden="true" tabindex="-1"></a>Web Crawler</h3>
<details>
 <summary>Final design</summary>
 <img src="/web-crawler.png">
</details>
<hr>
<h3 id="notification-service"><a href="#notification-service" aria-hidden="true" tabindex="-1"></a>Notification Service</h3>
<details>
 <summary>Final design</summary>
 <img src="/notification.png">
</details>
<hr>
<h3 id="news-feed-1"><a href="#news-feed-1" aria-hidden="true" tabindex="-1"></a>News Feed</h3>
<details>
 <summary>Final design</summary>
 <img src="/news-feed.png">
</details>
<hr>
<h3 id="chat-service"><a href="#chat-service" aria-hidden="true" tabindex="-1"></a>Chat service</h3>
<ul>
<li>Only one system is stateful - the chat service</li>
<li>Notification is an important part</li>
<li>Storage of chat messages in Cassandra</li>
<li>Each user has a queue of messages waiting to sync to them</li>
</ul>
<details>
 <summary>Final design</summary>
 <img src="/chat-service.png">
</details>
<hr>
<h3 id="search-autocomplete-1"><a href="#search-autocomplete-1" aria-hidden="true" tabindex="-1"></a>Search Autocomplete</h3>
<details>
 <summary>Final design</summary>
 <img src="/name.png">
</details>
<hr>


Checklist
Primitives
Clarifying questions

Rate Limiter
Key Value Store
ID generator
URL shortener
Web Crawler
Notification system
News Feed
Chat system
Search autocomplete
Youtube
Google drive


Final designs

Rate Limiter
Key Value Store
URL shortener
Web Crawler
Notification Service
News Feed
Chat service
Search Autocomplete



Checklist
Go through this checklist in every interview

Clarify requirements

Write QPS, average and peak (peak is 2x average)
Read QPS (approx 10x write QPS)
Total storage required
Latency requirement
Consistency vs Availability


High level diagram that matches requirements. Get buy-in
Deep Dive

API
Database schema


Wrap up

Error cases
Potential bottlenecks


Extra credit

Cost (development vs ongoing)
Logging, metrics, alerts
Automation - CI, CD
i18n
CDN
RPC




Primitives

Load balancer
Gateway (for ratelimiting, authentication)
Server (optionally horizonatally scaled)
Database

SQL vs NoSQL, sharding strategy.
If SQL, indices


Cache (optionally clustered/sharded)
Message queue, with workers
CDN
Lambdas

Be explicit about replication

Clarifying questions
Good for any problem

Is this a mobile app? Or a web app? Or both?
What are the most important features for the product?
DAU, MAU? Derive peak/avg QPS from this.
Is there any limit per request/stored object? Derive bandwidth/storage requirements from this.
Will the system work in a distributed environment?
Where are our users? Geographical constraints, as well as i18n.
Can we leverage some of the existing cloud infrastructures provided by Amazon, Google, or Microsoft?

Rate Limiter

Does the rate limiter throttle API requests based on IP, the user ID, or other properties?
Is the rate limiter a separate service or should it be implemented in application code?
Do we need to inform users who are throttled?

Key Value Store
Not questions, but these are the constraints

Scalable
Available
Tunable consistency
Auto scaling
Size of key value pair - 10kB
Low latency

ID generator

What are the characteristics of unique IDs?
For each new record, does ID increment by 1?
Do IDs only contain numerical values?
What is the ID length requirement?

URL shortener

Can you give an example of how a URL shortener work?
How long is the shortened URL?
What characters are allowed in the shortened URL?
Can shortened URLs be deleted or updated?

Web Crawler

What is the main purpose of the crawler? Is it used for search engine indexing, data mining, or something else?
How many web pages does the web crawler collect per month?
What content types are included? HTML only or other content types such as PDFs and images as well?
Shall we consider newly added or edited web pages?
Do we need to store HTML pages crawled from the web?
How do we handle web pages with duplicate content?

Notification system

What types of notifications does the system support?
Is it a real-time system? (ie, are delays allowed?)
What triggers notifications?
Will users be able to opt-out?

News Feed

Is the news feed sorted in reverse chronological order or a particular order? The particular order means each post is given a different weight. For instance, posts from your close friends are more important than posts from a group.
How many friends can a user have?
Can feed contain images, videos, or just text?

Chat system

What kind of chat app shall we design? 1 on 1 or group based?
For group chat, what is the group member limit?
Is end-to-end encryption required?
How long shall we store the chat history? How about not storing chat history?

Search autocomplete

Is the matching only supported at the beginning of a search query or in the middle as well?
How many autocomplete suggestions should the system return?
What criteria should decide the ordering of the suggestions?
Does the system support spell check?
Are search queries in English?
Do we allow capitalization and special characters?

Youtube

What are the supported video resolutions?

Google drive

What are the supported file formats?


Final designs
Rate Limiter

    Final design
    


Key Value Store

    Final design
    
    
    


URL shortener

    Final design
    


Web Crawler

    Final design
    


Notification Service

    Final design
    


News Feed

    Final design
    


Chat service

Only one system is stateful - the chat service
Notification is an important part
Storage of chat messages in Cassandra
Each user has a queue of messages waiting to sync to them


    Final design
    


Search Autocomplete

    Final design
    



System Design Interview by Alex Xu

<a href="https://hpbn.co/">Link to the full book</a>
Talk by Ilya Grigorik about improving the performance of the web - <a href="https://www.youtube.com/watch?v=vtIfVPtN6io">performance.now().reject(reasons)</a>
<ul>
<li><a href="#chapter-1---four-components-of-latency">Chapter 1 - Four components of latency</a></li>
<li><a href="#chapter-2---tcp-rfc-793-and-ip-rfc-791">Chapter 2 - TCP (RFC 793) and IP (RFC 791)</a></li>
<li><a href="#chapter-3---udp-rfc-768">Chapter 3 - UDP (RFC 768)</a>
<ul>
<li><a href="#udp-and-nat">UDP and NAT</a></li>
</ul>
</li>
<li><a href="#chapter-4---tls">Chapter 4 - TLS</a></li>
<li><a href="#chapter-8---optimizing-for-mobile-networks">Chapter 8 - Optimizing for mobile networks</a></li>
<li><a href="#chapter-9---primer">Chapter 9 - Primer</a></li>
</ul>
<hr>
<h2 id="chapter-1---four-components-of-latency"><a href="#chapter-1---four-components-of-latency" aria-hidden="true" tabindex="-1"></a>Chapter 1 - Four components of latency</h2>
<ul>
<li>Transmission delay. Time taken to write the entire packet into the link. Function of data rate of this link</li>
<li>Propagation delay. Time taken for the packet to travel from sender to receiver</li>
<li>Processing delay. Time taken to verify the packet, check for bit level errors, process the header and determine the packet's destination</li>
<li>Queuing delay. Time packet spends in the receiver's buffer waiting to be processed.</li>
</ul>
If packets are arriving faster than a router can process them, it should drop them so TCP's congestion control kicks in. Routers have very large buffers to avoid drops.
Speed of light in fibre optic cable is approximately 2*10^8 m/s or 66% of the speed of light in vacuum. Reducing the refractive index of fibre optic cable is an ongoing effort.
The latency from New York to London is 21ms, but the last mile latency between the ISP and home could be anywhere between 10ms (good) and 65ms (bad).
The available bandwidth to the user is a function of the lowest capacity link between the client and the destination server. Theoretically we can keep improving this - either by more fibre links between nodes or by improving the method we use to multiplex data.
The latency between two nodes cannot be improved indefinitely. Its possible to improve the fibre such that the speed of light within it increases. However, there is a hard limit - c. Alternately, its possible to lay cable that takes a shorter route.
<hr>
<h2 id="chapter-2---tcp-rfc-793-and-ip-rfc-791"><a href="#chapter-2---tcp-rfc-793-and-ip-rfc-791" aria-hidden="true" tabindex="-1"></a>Chapter 2 - TCP (RFC 793) and IP (RFC 791)</h2>
<ul>
<li>Features
<ol>
<li>Retransmission of lost data (waiting for ACKs)</li>
<li>In-order delivery (through unique sequence IDs)</li>
<li>Flow control, congestion control, congestion avoidance.</li>
<li>Data integrity (through checksums)</li>
</ol>
</li>
</ul>
Optimized for accurate delivery, not timely delivery.
<ul>
<li>
3 way Handshake
<ol>
<li>Client sends a SYN containing x=rand()</li>
<li>Server sends a SYN ACK containing x+1, y=rand()</li>
<li>Client sends ACK containing x+1, y+1 along with application data.</li>
</ol>
</li>
<li>
TCP Fast Open (TFO) - since the 3 way handshake takes at least 1 roundtrip, this can be optimized by sending the application data with the first SYN packet. Limitations - only certain kinds of requests can be sent (I think idempotent), only works for sessions that are resumed because a cookie is required, size of the first payload is limited and lastly, both the client OS, server OS should support it before the client application can opt in.
</li>
<li>
Congestion collapse
<ul>
<li>happens when networks of differing bandwidth are connected</li>
<li>when the total round trip time exceeds the retransmission time, the source introduces more copies of the same datagram into the network</li>
<li>this fills up buffers and packets need to be dropped, increasing retransmission</li>
<li>eventually all buffers are full and the network operates at a degraded level.</li>
</ul>
</li>
</ul>
To avoid this - flow control, congestion control, congestion avoidance.
<ul>
<li>
Flow control - With every ACK, the receiver advertises <code>rwind</code>, the size of the receive window, ie the buffer which will hold the incoming data. If the receiver is unable to keep up, it advertises a smaller <code>rwind</code>. Initially this was a 16 bit value because who would ever have a buffer larger than 65,535 bytes? RFC 1323 supports "scaling" this number so higher buffer sizes can be specified by this 16 bit value.
</li>
<li>
Congestion control. Slow start. The sender starts out by sending <code>cwind</code> (congestion window) sized data, which is 10 network segments as of 2013. The size of <code>cwind</code> is exponentially increased; with every successful ACK received, two more packets can be sent. The maximum amount of data in flight is the <code>min(cwind, rwind)</code>. It often takes a few round trips before the maximum throughput is achieved. For small downloads, that's a problem. If a TCP connection has been idle, the <code>cwind</code> is reset to its original value. This behaviour can be disabled on the server.
</li>
<li>
Congestion avoidance. The number of packets sent keeps doubling until a packet is dropped by the network. At that point <code>cwind</code> is reduced and increased based on some avoidance algorithm. (This shows up as a sawtooth). Assumption - the packet loss occurred because a router along the way was overwhelmed. Reducing <code>cwind</code> decreases the chance of packet loss
</li>
<li>
The Bandwidth-Delay product - the maximum amount of un-ACKed data on the wire. Since this is a function of the latency, this is a limitation
<ul>
<li><code>min(cwind, rwind)</code> = 16kB. Delay - 100ms. Max available bandwidth = 16kB/100ms = 1.31Mbps</li>
<li>Client's bandwidth = 10Mbps. Delay - 100ms. Ideal window size = 10 x 1000 x 1000 / 8 * 0.1 = 125000 bytes = 122.13 kB</li>
</ul>
</li>
<li>
To summarize congestion on TCP networks. If a link between client and server isn't being saturated it could be because
<ul>
<li>packet loss triggering congestion avoidance</li>
<li>small default windows in the config of client or server</li>
<li>explicit traffic shaping</li>
</ul>
</li>
<li>
Tuning TCP
<ol>
<li>Disable slow-start while resuming idle connections</li>
<li>Keep a high initial <code>cwind</code></li>
<li>TCP Fast Open (TFO)</li>
<li>Window scaling for larger buffers</li>
</ol>
</li>
<li>
Other optimizations
<ol>
<li>Less data on the wire - either send less data, or compress</li>
<li>Use CDNs so latency is lower</li>
<li>Upgrade to latest kernels for the best algos</li>
<li>Reuse TCP connections wherever possible</li>
</ol>
</li>
<li>
Why not TCP for some applications - to avoid head-of-line blocking - if a single packet is dropped, all subsequent packets can't be processed until that is re-transmitted and received. Some applications (audio, video, games) would prefer to receive packets as they are received and deal with ordering themselves (some applications might not even need ordering)
</li>
</ul>
<hr>
<h2 id="chapter-3---udp-rfc-768"><a href="#chapter-3---udp-rfc-768" aria-hidden="true" tabindex="-1"></a>Chapter 3 - UDP (RFC 768)</h2>
<ol>
<li>No retransmission</li>
<li>No ordering</li>
<li>No congestion control</li>
<li>No connection establishment or teardown</li>
<li>No nothing! I guess you will never be anything more than a mere 🐵</li>
</ol>
DNS and WebRTC use UDP
<h3 id="udp-and-nat"><a href="#udp-and-nat" aria-hidden="true" tabindex="-1"></a>UDP and NAT</h3>
<ul>
<li>
UDP is a stateless protocol, but NAT requires state to decide how to route traffic. When there is outbound UDP traffic, it adds a tranlation record with some TTL on how to route the response. Since this record might expire after some time, its the de facto best practice to send keep-alive packets regularly. This is true even for TCP because some poorly implemented NAT boxes drop TCP translation records too.
</li>
<li>
Worse, some p2p applications like games, VoIP and file sharing apps need to act as servers too. These applications don't know their public IP. Even if these applications did find out their public IP and communicated that to the peer, an in-bound packet to that IP would still be dropped because the NAT box wouldn't know how to route it. Workarounds
<ol>
<li>STUN (RFC 5389) - Session Traversal Utilities for NAT. The wannabe server asks a STUN server on the public internet what its IP is. When the application finds out the IP port tuple it communicates that with its peers. Works 92% of the time.</li>
<li>TURN (RFC 5766) - Traversal Using Relays around NAT. If STUN doesn't work, all traffic in both directions is routed through a relay. The traffic is no longer p2p. Works remaining 8% of the time.</li>
<li>ICE (RFC 5245) - Interactive Connectivity Establishment. Establishes the most efficient tunnel between two participants whether thats through STUN or TURN.</li>
</ol>
</li>
<li>
Recommendations for UDP (RFC 5405)
<ol>
<li>The application should handle most of what is handled by TCP. It should therefore implement congestion control, congestion avoidance, flow control, error checking, and handle loss, duplication and re-ordering</li>
<li>It should use similar bandwidth to TCP</li>
<li>Size of the datagram shouldn't exceed the Maximum Transmission Unit (MTU)</li>
<li>Should enable checksums</li>
</ol>
</li>
</ul>
<hr>
<h2 id="chapter-4---tls"><a href="#chapter-4---tls" aria-hidden="true" tabindex="-1"></a>Chapter 4 - TLS</h2>
<ul>
<li>TLS 1.2 (RFC 5246)</li>
<li>TLS 1.3</li>
</ul>
Refer existing note on tls (TODO link)
<hr>
Skipping Chapters 5, 6, 7
<hr>
<h2 id="chapter-8---optimizing-for-mobile-networks"><a href="#chapter-8---optimizing-for-mobile-networks" aria-hidden="true" tabindex="-1"></a>Chapter 8 - Optimizing for mobile networks</h2>
<ul>
<li>3 main considerations - presentation considering the form factor, battery life, performance of the radio</li>
<li>For presentation read some other book.</li>
<li>Radio and battery
<ul>
<li>Radio use at full power can drain the battery in hours</li>
<li>Each successive generation of radio requires more battery</li>
<li>Radio power consumption has a non-linear relationship with data transferred. Even a little data requires a full power radio, consuming power.</li>
</ul>
</li>
<li>Simple rules
<ol>
<li>Avoid polling</li>
<li>Push notifications should be used. However, high frequency pushes consume similar amount of battery as polling, so pushes should be aggregated on the server</li>
<li>Inbound and outbound requests should be coalesced</li>
<li>Not-critical requests should be deferred until the radio is active</li>
</ol>
</li>
<li>Energy conversions
<ul>
<li>3Ah * 5V = 15 Wh (phones nowadays are 3000 mah and voltage supplied is 5V)</li>
<li>converting to joules, 15 Wh x 3600 J/Wh = 54000 J</li>
<li>radio state transition to high costs 10J. 1 minute polling interval for an hour = 600J. That's 1% of total battery capacity per application that's polling per hour</li>
</ul>
</li>
<li>Energy consumption
<ul>
<li>With WiFi, each device sets its own transmit power, which is usually in the 30–200 mW range. By comparison, the transmit power of the 3G/4G radio is managed by the network and can consume as low as 15 mW when in an idle state. However, to account for larger range and interference, the same radio can require 1,000–3,500 mW when transmitting in a high-power state!</li>
<li>In practice, when transferring large amounts of data, WiFi is often far more efficient if the signal strength is good. But if the device is mostly idle, then the 3G/4G radio is more effective.</li>
</ul>
</li>
</ul>
Latencies on 3G (200ms/RTT) and 4G (100ms/RTT)
| Leg | 3G | 4G |
| ---------------------- | ------------ | ---------- |
| Control plane | 200–2,500 ms | 50–100 ms |
| DNS lookup | 200 ms | 100 ms |
| TCP handshake | 200 ms | 100 ms |
| TLS handshake | 200–400 ms | 100–200 ms |
| HTTP request | 200 ms | 100 ms |
| Total latency overhead | 200–3500 ms | 100–600 ms |
The initial delay for 3.5G+ networks is 150-500ms
<ul>
<li>Recommendations
<ul>
<li>While streaming, if the entire file will definitely be used, download in one shot and shut off the radio. If it might not be, then stream it, preferably with adaptive bitrate streaming</li>
<li>Do not cache or attempt to guess the state of the network.</li>
<li>Dispatch the request, listen for failures, and diagnose what happened.</li>
<li>Transient errors will happen; plan for them, and use a retry strategy.</li>
<li>Listen to connection state to anticipate the best request strategy.</li>
<li>Use a backoff algorithm for request retries; do not spin forever.</li>
<li>If offline, log and dispatch the request later if possible.</li>
<li>Leverage HTML5 AppCache and localStorage for offline mode.</li>
</ul>
</li>
</ul>
<hr>
<h2 id="chapter-9---primer"><a href="#chapter-9---primer" aria-hidden="true" tabindex="-1"></a>Chapter 9 - Primer</h2>
<ul>
<li>HTTP 0.9
<ul>
<li>only one document would be requested</li>
<li>1 TCP connection per request.</li>
</ul>
</li>
<li>HTTP 1.0
<ul>
<li>multiple documents would be requested on a single page. Features introduced as a result:</li>
<li>introduced the concept of headers</li>
</ul>
</li>
<li>HTTP 1.1
<ul>
<li>byte range requests</li>
<li>keepalive! (reusing existing TCP connection, reducing latency)</li>
<li>caching</li>
<li>request pipelining (failed due to lack of support)</li>
</ul>
</li>
<li>HTTP 2.0
<ul>
<li>header compression</li>
<li>request multiplexing</li>
</ul>
</li>
</ul>
<hr>
HTTP: The Definitive Guide has further details about the protocol
Optimizing tips from High Performance Web Sites
<ol>
<li>Make fewer requests</li>
<li>Make fewer DNS requests</li>
<li>Use gzip</li>
<li>Configure etags and expires header</li>
<li>Use CDNs</li>
<li>Avoid HTTP redirects</li>
</ol>
Request pipelining doesn't work. Workarounds
<ol>
<li>Spriting - increased memory consumption because the entire image isn't usually needed. Even a small change means a cache invalidation</li>
<li>Concatenation of all resources - slows down time to first byte. Becomes impossible to cache these resources.</li>
</ol>

Link to the full book
Talk by Ilya Grigorik about improving the performance of the web - performance.now().reject(reasons)

Chapter 1 - Four components of latency
Chapter 2 - TCP (RFC 793) and IP (RFC 791)
Chapter 3 - UDP (RFC 768)

UDP and NAT


Chapter 4 - TLS
Chapter 8 - Optimizing for mobile networks
Chapter 9 - Primer


Chapter 1 - Four components of latency

Transmission delay. Time taken to write the entire packet into the link. Function of data rate of this link
Propagation delay. Time taken for the packet to travel from sender to receiver
Processing delay. Time taken to verify the packet, check for bit level errors, process the header and determine the packet's destination
Queuing delay. Time packet spends in the receiver's buffer waiting to be processed.

If packets are arriving faster than a router can process them, it should drop them so TCP's congestion control kicks in. Routers have very large buffers to avoid drops.
Speed of light in fibre optic cable is approximately 2*10^8 m/s or 66% of the speed of light in vacuum. Reducing the refractive index of fibre optic cable is an ongoing effort.
The latency from New York to London is 21ms, but the last mile latency between the ISP and home could be anywhere between 10ms (good) and 65ms (bad).
The available bandwidth to the user is a function of the lowest capacity link between the client and the destination server. Theoretically we can keep improving this - either by more fibre links between nodes or by improving the method we use to multiplex data.
The latency between two nodes cannot be improved indefinitely. Its possible to improve the fibre such that the speed of light within it increases. However, there is a hard limit - c. Alternately, its possible to lay cable that takes a shorter route.

Chapter 2 - TCP (RFC 793) and IP (RFC 791)

Features

Retransmission of lost data (waiting for ACKs)
In-order delivery (through unique sequence IDs)
Flow control, congestion control, congestion avoidance.
Data integrity (through checksums)



Optimized for accurate delivery, not timely delivery.


3 way Handshake

Client sends a SYN containing x=rand()
Server sends a SYN ACK containing x+1, y=rand()
Client sends ACK containing x+1, y+1 along with application data.



TCP Fast Open (TFO) - since the 3 way handshake takes at least 1 roundtrip, this can be optimized by sending the application data with the first SYN packet. Limitations - only certain kinds of requests can be sent (I think idempotent), only works for sessions that are resumed because a cookie is required, size of the first payload is limited and lastly, both the client OS, server OS should support it before the client application can opt in.


Congestion collapse

happens when networks of differing bandwidth are connected
when the total round trip time exceeds the retransmission time, the source introduces more copies of the same datagram into the network
this fills up buffers and packets need to be dropped, increasing retransmission
eventually all buffers are full and the network operates at a degraded level.



To avoid this - flow control, congestion control, congestion avoidance.


Flow control - With every ACK, the receiver advertises rwind, the size of the receive window, ie the buffer which will hold the incoming data. If the receiver is unable to keep up, it advertises a smaller rwind. Initially this was a 16 bit value because who would ever have a buffer larger than 65,535 bytes? RFC 1323 supports "scaling" this number so higher buffer sizes can be specified by this 16 bit value.


Congestion control. Slow start. The sender starts out by sending cwind (congestion window) sized data, which is 10 network segments as of 2013. The size of cwind is exponentially increased; with every successful ACK received, two more packets can be sent. The maximum amount of data in flight is the min(cwind, rwind). It often takes a few round trips before the maximum throughput is achieved. For small downloads, that's a problem. If a TCP connection has been idle, the cwind is reset to its original value. This behaviour can be disabled on the server.


Congestion avoidance. The number of packets sent keeps doubling until a packet is dropped by the network. At that point cwind is reduced and increased based on some avoidance algorithm. (This shows up as a sawtooth). Assumption - the packet loss occurred because a router along the way was overwhelmed. Reducing cwind decreases the chance of packet loss


The Bandwidth-Delay product - the maximum amount of un-ACKed data on the wire. Since this is a function of the latency, this is a limitation

min(cwind, rwind) = 16kB. Delay - 100ms. Max available bandwidth = 16kB/100ms = 1.31Mbps
Client's bandwidth = 10Mbps. Delay - 100ms. Ideal window size = 10 x 1000 x 1000 / 8 * 0.1 = 125000 bytes = 122.13 kB



To summarize congestion on TCP networks. If a link between client and server isn't being saturated it could be because

packet loss triggering congestion avoidance
small default windows in the config of client or server
explicit traffic shaping



Tuning TCP

Disable slow-start while resuming idle connections
Keep a high initial cwind
TCP Fast Open (TFO)
Window scaling for larger buffers



Other optimizations

Less data on the wire - either send less data, or compress
Use CDNs so latency is lower
Upgrade to latest kernels for the best algos
Reuse TCP connections wherever possible



Why not TCP for some applications - to avoid head-of-line blocking - if a single packet is dropped, all subsequent packets can't be processed until that is re-transmitted and received. Some applications (audio, video, games) would prefer to receive packets as they are received and deal with ordering themselves (some applications might not even need ordering)



Chapter 3 - UDP (RFC 768)

No retransmission
No ordering
No congestion control
No connection establishment or teardown
No nothing! I guess you will never be anything more than a mere 🐵

DNS and WebRTC use UDP
UDP and NAT


UDP is a stateless protocol, but NAT requires state to decide how to route traffic. When there is outbound UDP traffic, it adds a tranlation record with some TTL on how to route the response. Since this record might expire after some time, its the de facto best practice to send keep-alive packets regularly. This is true even for TCP because some poorly implemented NAT boxes drop TCP translation records too.


Worse, some p2p applications like games, VoIP and file sharing apps need to act as servers too. These applications don't know their public IP. Even if these applications did find out their public IP and communicated that to the peer, an in-bound packet to that IP would still be dropped because the NAT box wouldn't know how to route it. Workarounds

STUN (RFC 5389) - Session Traversal Utilities for NAT. The wannabe server asks a STUN server on the public internet what its IP is. When the application finds out the IP port tuple it communicates that with its peers. Works 92% of the time.
TURN (RFC 5766) - Traversal Using Relays around NAT. If STUN doesn't work, all traffic in both directions is routed through a relay. The traffic is no longer p2p. Works remaining 8% of the time.
ICE (RFC 5245) - Interactive Connectivity Establishment. Establishes the most efficient tunnel between two participants whether thats through STUN or TURN.



Recommendations for UDP (RFC 5405)

The application should handle most of what is handled by TCP. It should therefore implement congestion control, congestion avoidance, flow control, error checking, and handle loss, duplication and re-ordering
It should use similar bandwidth to TCP
Size of the datagram shouldn't exceed the Maximum Transmission Unit (MTU)
Should enable checksums




Chapter 4 - TLS

TLS 1.2 (RFC 5246)
TLS 1.3

Refer existing note on tls (TODO link)

Skipping Chapters 5, 6, 7

Chapter 8 - Optimizing for mobile networks

3 main considerations - presentation considering the form factor, battery life, performance of the radio
For presentation read some other book.
Radio and battery

Radio use at full power can drain the battery in hours
Each successive generation of radio requires more battery
Radio power consumption has a non-linear relationship with data transferred. Even a little data requires a full power radio, consuming power.


Simple rules

Avoid polling
Push notifications should be used. However, high frequency pushes consume similar amount of battery as polling, so pushes should be aggregated on the server
Inbound and outbound requests should be coalesced
Not-critical requests should be deferred until the radio is active


Energy conversions

3Ah * 5V = 15 Wh (phones nowadays are 3000 mah and voltage supplied is 5V)
converting to joules, 15 Wh x 3600 J/Wh = 54000 J
radio state transition to high costs 10J. 1 minute polling interval for an hour = 600J. That's 1% of total battery capacity per application that's polling per hour


Energy consumption

With WiFi, each device sets its own transmit power, which is usually in the 30–200 mW range. By comparison, the transmit power of the 3G/4G radio is managed by the network and can consume as low as 15 mW when in an idle state. However, to account for larger range and interference, the same radio can require 1,000–3,500 mW when transmitting in a high-power state!
In practice, when transferring large amounts of data, WiFi is often far more efficient if the signal strength is good. But if the device is mostly idle, then the 3G/4G radio is more effective.



Latencies on 3G (200ms/RTT) and 4G (100ms/RTT)
| Leg                    | 3G           | 4G         |
| ---------------------- | ------------ | ---------- |
| Control plane          | 200–2,500 ms | 50–100 ms  |
| DNS lookup             | 200 ms       | 100 ms     |
| TCP handshake          | 200 ms       | 100 ms     |
| TLS handshake          | 200–400 ms   | 100–200 ms |
| HTTP request           | 200 ms       | 100 ms     |
| Total latency overhead | 200–3500 ms  | 100–600 ms |
The initial delay for 3.5G+ networks is 150-500ms

Recommendations

While streaming, if the entire file will definitely be used, download in one shot and shut off the radio. If it might not be, then stream it, preferably with adaptive bitrate streaming
Do not cache or attempt to guess the state of the network.
Dispatch the request, listen for failures, and diagnose what happened.
Transient errors will happen; plan for them, and use a retry strategy.
Listen to connection state to anticipate the best request strategy.
Use a backoff algorithm for request retries; do not spin forever.
If offline, log and dispatch the request later if possible.
Leverage HTML5 AppCache and localStorage for offline mode.




Chapter 9 - Primer

HTTP 0.9

only one document would be requested
1 TCP connection per request.


HTTP 1.0

multiple documents would be requested on a single page. Features introduced as a result:
introduced the concept of headers


HTTP 1.1

byte range requests
keepalive! (reusing existing TCP connection, reducing latency)
caching
request pipelining (failed due to lack of support)


HTTP 2.0

header compression
request multiplexing




HTTP: The Definitive Guide has further details about the protocol
Optimizing tips from High Performance Web Sites

Make fewer requests
Make fewer DNS requests
Use gzip
Configure etags and expires header
Use CDNs
Avoid HTTP redirects

Request pipelining doesn't work. Workarounds

Spriting - increased memory consumption because the entire image isn't usually needed. Even a small change means a cache invalidation
Concatenation of all resources - slows down time to first byte. Becomes impossible to cache these resources.


High Performance Browser Networking by Ilya Grigorik

SQL Performance Explained by Markus Winand

Java Concurrency in Practice by Joshua Bloch

<div class="remark-highlight"><pre class="language-sh"><code class="language-sh"># this also installs xcode tools

/bin/bash -c &#x26;quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)&#x26;quot;
echo &#x26;#39;eval $(/opt/homebrew/bin/brew shellenv)&#x26;#39; &#x26;gt;&#x26;gt; /Users/nindalf/.zprofile

# Rust
curl --proto &#x26;#39;=https&#x26;#39; --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Rust replacements

brew install starship
eval &#x26;quot;$(starship init zsh)&#x26;quot; # Add to .zprofile
# Configure according to https://starship.rs/config/

brew install fd
brew install hyperfine

brew install git-delta

brew install gh
gh auth login
</code></pre></div>
<h2 id="git-config"><a href="#git-config" aria-hidden="true" tabindex="-1"></a>Git config</h2>
See <a href="https://til.nindalf.com/til/git/better-diff/">git/better-diff</a> and <a href="https://til.nindalf.com/til/git/aliases/">git/aliases</a>.
<h2 id="non-cmd-line-apps"><a href="#non-cmd-line-apps" aria-hidden="true" tabindex="-1"></a>Non cmd line apps</h2>
Dropbox
Obsidian
Whatsapp Desktop
ITerm 2
Steam
Spotify
Day One
LastPass
2Do
Discord
Wireguard

\# this also installs xcode tools

/bin/bash -c \&quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\&quot;
echo \&#39;eval $(/opt/homebrew/bin/brew shellenv)\&#39; \&gt;\&gt; /Users/nindalf/.zprofile

# Rust

curl --proto \&#39;=https\&#39; --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Rust replacements

brew install starship
eval \&quot;$(starship init zsh)\&quot; # Add to .zprofile

# Configure according to https://starship.rs/config/

brew install fd
brew install hyperfine

brew install git-delta

brew install gh
gh auth login


Git config
See git/better-diff and git/aliases.
Non cmd line apps
Dropbox
Obsidian
Whatsapp Desktop
ITerm 2
Steam
Spotify
Day One
LastPass
2Do
Discord
Wireguard


New computer setup

<div class="remark-highlight"><pre class="language-css"><code class="language-css">/* Assume light mode by default */
@import 'light.css' screen;

/* Supersede with dark mode when applicable */
@import 'dark.css' screen and (prefers-color-scheme: dark);
</code></pre></div>

/* Assume light mode by default */
@import 'light.css' screen;

/\* Supersede with dark mode when applicable \*/
@import 'dark.css' screen and (prefers-color-scheme: dark);


Dark Mode

<h2 id="-l"><a href="#-l" aria-hidden="true" tabindex="-1"></a><code>-L</code></h2>
Example <code>curl 'https://www.goodreads.com/search?q=9780340766163' -L</code>

\-L
Example curl 'https://www.goodreads.com/search?q=9780340766163' -L


Follow redirects

<a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.html">Source</a>
<h2 id="terms"><a href="#terms" aria-hidden="true" tabindex="-1"></a>Terms</h2>
<ul>
<li>Table = Table</li>
<li>Item = Row</li>
<li>Attribute = Column (but this is schemaless)</li>
</ul>
<h2 id="primary-key"><a href="#primary-key" aria-hidden="true" tabindex="-1"></a>Primary Key</h2>
Primary key can be one of
<ol>
<li>Partition Key. Single attribute. Hash(Partition Key) determines partition.</li>
<li>Partition Key + Sort key. All items with the same partition key value are stored together, in sorted order by sort key value.</li>
</ol>
Primary key attributes have to be scalar - string, number or binary.
<h2 id="secondary-index"><a href="#secondary-index" aria-hidden="true" tabindex="-1"></a>Secondary Index</h2>
<ul>
<li>Global secondary index. Basically a copy of the table with a different Primary key.</li>
<li>Local secondary index. Same partition key, different sort key.</li>
</ul>
<h2 id="streams"><a href="#streams" aria-hidden="true" tabindex="-1"></a>Streams</h2>
Near real time events when items are added/updated/deleted. Can combine with Lambda.
<h2 id="api"><a href="#api" aria-hidden="true" tabindex="-1"></a>API</h2>
Mostly obvious - <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.API.html">link</a>
<h2 id="read-consistency"><a href="#read-consistency" aria-hidden="true" tabindex="-1"></a>Read consistency</h2>
When your application writes data to a DynamoDB table and receives an HTTP 200 response (OK), the write has occurred and is durable. The data is eventually consistent across all storage locations, usually within one second or less.
<ul>
<li>Eventually Consistent Reads - When you read data from a DynamoDB table, the response might not reflect the results of a recently completed write operation. The response might include some stale data. If you repeat your read request after a short time, the response should return the latest data.</li>
<li>Strongly Consistent Reads - When you request a strongly consistent read, DynamoDB returns a response with the most up-to-date data, reflecting the updates from all prior write operations that were successful. Issues:
<ul>
<li>Network outage/delay - 500</li>
<li>Higher latency</li>
<li>Not supported on global secondary indexes</li>
<li>Uses more throughput capacity</li>
</ul>
</li>
</ul>

Source
Terms

Table = Table
Item = Row
Attribute = Column (but this is schemaless)

Primary Key
Primary key can be one of

Partition Key. Single attribute. Hash(Partition Key) determines partition.
Partition Key + Sort key. All items with the same partition key value are stored together, in sorted order by sort key value.

Primary key attributes have to be scalar - string, number or binary.
Secondary Index

Global secondary index. Basically a copy of the table with a different Primary key.
Local secondary index. Same partition key, different sort key.

Streams
Near real time events when items are added/updated/deleted. Can combine with Lambda.
API
Mostly obvious - link
Read consistency
When your application writes data to a DynamoDB table and receives an HTTP 200 response (OK), the write has occurred and is durable. The data is eventually consistent across all storage locations, usually within one second or less.

Eventually Consistent Reads - When you read data from a DynamoDB table, the response might not reflect the results of a recently completed write operation. The response might include some stale data. If you repeat your read request after a short time, the response should return the latest data.
Strongly Consistent Reads - When you request a strongly consistent read, DynamoDB returns a response with the most up-to-date data, reflecting the updates from all prior write operations that were successful. Issues:

Network outage/delay - 500
Higher latency
Not supported on global secondary indexes
Uses more throughput capacity




DynamoDB basics

<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">
[alias]
	st = status -uno
	ci = commit -m
	cm = commit -m
	cob = checkout -b
	co = checkout
	uncommit = reset --soft HEAD^
	feature = !git fetch origin master:master &#x26;amp;&#x26;amp; git checkout -b $1 master &#x26;amp;&#x26;amp; : # https://stackoverflow.com/a/25915221/16851956</code></pre></div>
All of this except <code>git feature</code> is self explanatory.


\[alias]
	st = status -uno
	ci = commit -m
	cm = commit -m
	cob = checkout -b
	co = checkout
	uncommit = reset --soft HEAD^
	feature = !git fetch origin master:master &#x26;amp;&#x26;amp; git checkout -b $1 master &#x26;amp;&#x26;amp; : # https://stackoverflow.com/a/25915221/16851956
All of this except git feature is self explanatory.


Git Aliases

<h1 id="add-this-to-gitconfig-after-brew-install-git-delta"><a href="#add-this-to-gitconfig-after-brew-install-git-delta" aria-hidden="true" tabindex="-1"></a>Add this to .gitconfig after <code>brew install git-delta</code></h1>
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">[core]
 pager = delta

[interactive]
 diffFilter = delta --color-only

[delta]
 features = side-by-side line-numbers decorations
 whitespace-error-style = 22 reverse

[delta &#x26;quot;decorations&#x26;quot;]
 commit-decoration-style = bold yellow box ul
 file-style = bold yellow ul
 file-decoration-style = none</code></pre></div>

Add this to .gitconfig after brew install git-delta
\[core]
    pager = delta

\[interactive]
diffFilter = delta --color-only

\[delta]
features = side-by-side line-numbers decorations
whitespace-error-style = 22 reverse

\[delta \&quot;decorations\&quot;]
commit-decoration-style = bold yellow box ul
file-style = bold yellow ul
file-decoration-style = none


Better Diff

Switch to previous branch
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">git switch -</code></pre></div>
Remove all local git branches but keep master
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">git branch | grep -v &#x26;quot;master&#x26;quot; | xargs git branch -D</code></pre></div>
Branches I recently modified
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">git branch --sort=-committerdate ## DESC
git branch --sort=committerdate ## ASC</code></pre></div>

Switch to previous branch
git switch -
Remove all local git branches but keep master
git branch | grep -v &#x26;quot;master&#x26;quot; | xargs git branch -D
Branches I recently modified
git branch --sort=-committerdate ## DESC
git branch --sort=committerdate ## ASC


Git Branch

<h2 id="commands"><a href="#commands" aria-hidden="true" tabindex="-1"></a>Commands</h2>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash"># Cloning a repo that contains submodules

git clone --recurse-submodules git://github.com/foo/bar.git

# Forgot to clone the submodule? It's ok

git submodule update --init --recursive

# Cloning in a github action

 steps:
 - uses: actions/checkout@v2
 with:
 submodules: recursive
</code></pre></div>
<h2 id="advantages"><a href="#advantages" aria-hidden="true" tabindex="-1"></a>Advantages</h2>
<ul>
<li>Changes in the submodule are kept separate from the main repo. It's impossible to commit submodule changes to the main repo or vice versa.</li>
<li><code>git status</code> in the main repo tells you if the submodule has uncommitted work, is behind origin.</li>
</ul>
<h2 id="how-i-got-started"><a href="#how-i-got-started" aria-hidden="true" tabindex="-1"></a>How I got started</h2>
I had PTSD from trying <code>git subtree</code> back in 2014. I assumed submodules were the same. I first used them on March 29th to add my fork of <code>hugo-theme-cactus-plus</code> to the blog repo. Then I used to create the <a href="https://github.com/nindalf/blog-template">blog-template repo</a> and pull it into the <a href="https://github.com/nindalf/blog">blog repo</a>. It's worked excellently.
In <a href="https://github.com/nindalf/crawler/">crawler</a> I added a submodule for the sake of tests. Didn't pollute a Go repo with a massive amount of HTML.

Commands
\# Cloning a repo that contains submodules

git clone --recurse-submodules git://github.com/foo/bar.git

\# Forgot to clone the submodule? It's ok

git submodule update --init --recursive

\# Cloning in a github action

    steps:
    - uses: actions/checkout@v2
      with:
        submodules: recursive



Advantages

Changes in the submodule are kept separate from the main repo. It's impossible to commit submodule changes to the main repo or vice versa.
git status in the main repo tells you if the submodule has uncommitted work, is behind origin.

How I got started
I had PTSD from trying git subtree back in 2014. I assumed submodules were the same. I first used them on March 29th to add my fork of hugo-theme-cactus-plus to the blog repo. Then I used to create the blog-template repo and pull it into the blog repo. It's worked excellently.
In crawler I added a submodule for the sake of tests. Didn't pollute a Go repo with a massive amount of HTML.


Git Submodules

Just use HTML.
<div class="remark-highlight"><pre class="language-markdown"><code class="language-markdown">a [link](#unique) to the same page

&#x3C;div id="unique">&#x3C;/div>
</code></pre></div>
The official markdown spec doesn't support this feature. That's why HTML is required.
Source for this idea - <a href="https://github.com/BurntSushi/ripgrep/">ripgrep's Readme</a>.
<hr>
A markdown processor like <code>remark-autolink-headings</code> adds these IDs to all heading tags.

Just use HTML.
a [link](#unique) to the same page

\<div id="unique">\</div>


The official markdown spec doesn't support this feature. That's why HTML is required.
Source for this idea - ripgrep's Readme.

A markdown processor like remark-autolink-headings adds these IDs to all heading tags.

HTML Intra page links

I use these python APIs in interviews
<ul>
<li><a href="#dataclasses">Dataclasses</a></li>
<li><a href="#defaultdict">DefaultDict</a></li>
<li><a href="#deque">Deque</a></li>
<li><a href="#priority-queue">Priority Queue</a></li>
<li><a href="#random">Random</a></li>
</ul>
<h2 id="dataclasses"><a href="#dataclasses" aria-hidden="true" tabindex="-1"></a>Dataclasses</h2>
Create a class with autogenerated
<ul>
<li><code>__init__()</code></li>
<li><code>__repr__()</code>. So it looks good when you print it</li>
<li><code>__eq__()</code>. Compares like it's a tuple.</li>
<li><code>__lt__()</code>, <code>__le__()</code>, <code>__gt__()</code>, and <code>__ge__()</code> if you pass in <code>order=True</code></li>
</ul>
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-python"><code class="language-python">from dataclasses import dataclass

@dataclass
class Node:
 foo: str
 bar: int

n = Node('baz', 1)

print(n) # Node(foo='baz', bar=1) 
</code></pre></div>
</details>
<hr>
<h2 id="defaultdict"><a href="#defaultdict" aria-hidden="true" tabindex="-1"></a>DefaultDict</h2>
When accessing an element for the first time, it returns a default value. Useful when counting occurrences of a key.
<code>contains()</code> still works correctly
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-python"><code class="language-python">from collections import defaultdict

x = defaultdict(int) # default value 0
y = defaultdict(list) # default value []

print('foo' in x)
x['foo'] += 1
print(x)
</code></pre></div>
</details>
<hr>
<h2 id="deque"><a href="#deque" aria-hidden="true" tabindex="-1"></a>Deque</h2>
A double ended queue. Can add/remove elements to either end. It's not a linked list because it doesn't have constant time addition/removal in the middle.
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-python"><code class="language-python">from collections import deque

de = deque()
de.appendleft(1)
de.append(2)
de.appendleft(0)
print(de) # deque([0, 1, 2])

de.remove(1) # Careful, this is O(N) complexity, not O(1) like a linked list
print(de) # deque([0, 2])

n = de.pop() 
print(n, de) # 2 deque([0])

n = de.popleft() 
print(n, de) # 0 deque([])

</code></pre></div>
</details>
<hr>
<h2 id="priority-queue"><a href="#priority-queue" aria-hidden="true" tabindex="-1"></a>Priority Queue</h2>
A min priority queue. Add integers, tuples of comparable objects or objects that implement <code>__lt__</code> and <code>__eq__</code>. Can hack it to work as a max priority queue.
The difference between this and <code>heapq</code> is that this is thread safe.
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-python"><code class="language-python">from dataclasses import dataclass

@dataclass(order=True)
class Job(object):
 priority: int
 description: str
 
from queue import PriorityQueue
q = PriorityQueue()

q.put( Job(3, 'Mid-level job') )
q.put( Job(10, 'Low-level job') )
q.put( Job(1, 'Important job') )

# A simple tuple would also work
# q.put((1, 'important job'))

# To reverse the priorities, multiple the priority by -1

print(q.qsize()) # 3

while not q.empty():
 next_job = q.get()
 print ('Processing job:', next_job.description)
</code></pre></div>
</details>
<hr>
<h2 id="random"><a href="#random" aria-hidden="true" tabindex="-1"></a>Random</h2>
Generate random ints or floats.
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-python"><code class="language-python">import random

print(random.random()) # Float from 0.0 &#x3C;= x &#x3C; 1.0
random.randrange(10) # Integer from 0 &#x3C;= x &#x3C; 10 
</code></pre></div>
</details>
<hr>

I use these python APIs in interviews

Dataclasses
DefaultDict
Deque
Priority Queue
Random

Dataclasses
Create a class with autogenerated

\__init__()
\__repr__(). So it looks good when you print it
\__eq__(). Compares like it's a tuple.
\__lt__(), __le__(), __gt__(), and __ge__() if you pass in order=True


 Snippet
from dataclasses import dataclass

@dataclass
class Node:
foo: str
bar: int

n = Node('baz', 1)

print(n) # Node(foo='baz', bar=1) 




DefaultDict
When accessing an element for the first time, it returns a default value. Useful when counting occurrences of a key.
contains() still works correctly

 Snippet
from collections import defaultdict

x = defaultdict(int) # default value 0
y = defaultdict(list) # default value \[]

print('foo' in x)
x\['foo'] += 1
print(x)




Deque
A double ended queue. Can add/remove elements to either end. It's not a linked list because it doesn't have constant time addition/removal in the middle.

 Snippet
from collections import deque

de = deque()
de.appendleft(1)
de.append(2)
de.appendleft(0)
print(de) # deque(\[0, 1, 2])

de.remove(1) # Careful, this is O(N) complexity, not O(1) like a linked list
print(de) # deque(\[0, 2])

n = de.pop()
print(n, de) # 2 deque(\[0])

n = de.popleft()
print(n, de) # 0 deque(\[])





Priority Queue
A min priority queue. Add integers, tuples of comparable objects or objects that implement __lt__ and __eq__. Can hack it to work as a max priority queue.
The difference between this and heapq is that this is thread safe.

 Snippet
from dataclasses import dataclass

@dataclass(order=True)
class Job(object):
priority: int
description: str

from queue import PriorityQueue
q = PriorityQueue()

q.put( Job(3, 'Mid-level job') )
q.put( Job(10, 'Low-level job') )
q.put( Job(1, 'Important job') )

\# A simple tuple would also work
\# q.put((1, 'important job'))

\# To reverse the priorities, multiple the priority by -1

print(q.qsize()) # 3

while not q.empty():
next_job = q.get()
print ('Processing job:', next_job.description)




Random
Generate random ints or floats.

 Snippet
import random

print(random.random()) # Float from 0.0 <= x < 1.0
random.randrange(10) # Integer from 0 <= x < 10

Interview Code Snippets

Use <a href="http://jsonresume.org">jsonresume</a> to simplify creation of a resume. Add resume details to a json file and store it in a <a href="https://github.com/nindalf/resume/">git repo</a>.
<ol>
<li>Keep updating a doc with achievements. <a href="https://github.com/nindalf/resume/edit/master/brag.md">web interface</a>.</li>
<li>Once in 6 or 12 months, update <code>resume.json</code> using that doc.</li>
<li>Follow instructions in the Readme to build the PDF.</li>
</ol>
<h2 id="context"><a href="#context" aria-hidden="true" tabindex="-1"></a>Context</h2>
The friction to update CV in delayed my previous job search by months. This process lowers the friction. Hopefully it's easier in the future.

Use jsonresume to simplify creation of a resume. Add resume details to a json file and store it in a git repo.

Keep updating a doc with achievements. web interface.
Once in 6 or 12 months, update resume.json using that doc.
Follow instructions in the Readme to build the PDF.

Context
The friction to update CV in delayed my previous job search by months. This process lowers the friction. Hopefully it's easier in the future.


Resume system

Shoutout to Sriraghavan for suggesting the snippet and the debugging process
While debugging why a class was being constructed repeatedly at build time, I added this snippet
<div class="remark-highlight"><pre class="language-javascript"><code class="language-javascript">class ContentStore {
 static _instance: ContentStore;
 constructor() {
 if (ContentStore._instance) {
 return ContentStore._instance;
 }
 ContentStore._instance = this;
 console.log("tree", process.pid, process.uptime());
 // expensive initialisation
 }
}
</code></pre></div>
Leading to this output. There were 4 processes, so this was going to be initialised a minimum of 4 times anyway.
It was unclear why the last process was repeating it though.
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">➜ npm run build

&#x26;gt; nextjs-blog-template@0.1.0 build
&#x26;gt; next build &#x26;amp;&#x26;amp; next export &#x26;amp;&#x26;amp; npm run rssGen

info - Using webpack 5. Reason: future.webpack5 option enabled https://nextjs.org/docs/messages/webpack5
info - Checking validity of types
info - Creating an optimized production build
info - Compiled successfully
info - Collecting page data .tree 48586 0.646691458
tree 48587 0.64301475
tree 48585 0.647991917
info - Collecting page data
[ =] info - Generating static pages (0/42)tree 48581 9.745386625
tree 48581 9.746271083
tree 48581 9.752843167
[ ] info - Generating static pages (0/42)tree 48581 9.770965042
tree 48581 9.779078292
tree 48581 9.802078208
tree 48581 9.825228583
info - Generating static pages (42/42)
info - Finalizing page optimization</code></pre></div>

Shoutout to Sriraghavan for suggesting the snippet and the debugging process
While debugging why a class was being constructed repeatedly at build time, I added this snippet
class ContentStore {
    static _instance: ContentStore;
    constructor() {
        if (ContentStore._instance) {
            return ContentStore._instance;
        }
        ContentStore._instance = this;
        console.log("tree", process.pid, process.uptime());
        // expensive initialisation
    }
}

Leading to this output. There were 4 processes, so this was going to be initialised a minimum of 4 times anyway.
It was unclear why the last process was repeating it though.
➜ npm run build

\&gt; nextjs-blog-template@0.1.0 build
\&gt; next build \&amp;\&amp; next export \&amp;\&amp; npm run rssGen

info  - Using webpack 5. Reason: future.webpack5 option enabled https://nextjs.org/docs/messages/webpack5
info  - Checking validity of types
info  - Creating an optimized production build
info  - Compiled successfully
info  - Collecting page data .tree 48586 0.646691458
tree 48587 0.64301475
tree 48585 0.647991917
info  - Collecting page data
\[   =] info  - Generating static pages (0/42)tree 48581 9.745386625
tree 48581 9.746271083
tree 48581 9.752843167
\[    ] info  - Generating static pages (0/42)tree 48581 9.770965042
tree 48581 9.779078292
tree 48581 9.802078208
tree 48581 9.825228583
info  - Generating static pages (42/42)
info  - Finalizing page optimization


Node process IDs

<a href="https://gist.github.com/willurd/5720255">Source</a>
<h2 id="python"><a href="#python" aria-hidden="true" tabindex="-1"></a>Python</h2>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ python -m http.server 8000

# Force bind to IPv4
➜ python -m http.server 8000 -b 0.0.0.0
</code></pre></div>
<hr>
<h2 id="rust"><a href="#rust" aria-hidden="true" tabindex="-1"></a>Rust</h2>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ cargo install https
➜ http
Hosting "." on port 8000...
Ctrl-C to stop.
</code></pre></div>
<hr>
<h2 id="go"><a href="#go" aria-hidden="true" tabindex="-1"></a>Go</h2>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ go get github.com/sno6/gommand
➜ $GOPATH/bin/gommand 'http.ListenAndServe(":8000", http.FileServer(http.Dir(".")))'
</code></pre></div>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash"># [Install Caddy](https://caddyserver.com/docs/install)
➜ caddy -port 8000
</code></pre></div>
<hr>
<h2 id="unix-tools"><a href="#unix-tools" aria-hidden="true" tabindex="-1"></a>Unix tools</h2>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">while true ; do nc -l 8000 &#x3C; test.html ; done
</code></pre></div>
<hr>
<h2 id="javascript"><a href="#javascript" aria-hidden="true" tabindex="-1"></a>Javascript</h2>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ npm install -g http-server
➜ http-server -p 8000

# or
➜ npm install -g node-static
➜ static -p 8000

#or
➜ npm install -g serve
➜ serve .
</code></pre></div>
<hr>

Source
Python
➜ python -m http.server 8000

\# Force bind to IPv4
➜ python -m http.server 8000 -b 0.0.0.0



Rust
➜ cargo install https
➜ http
Hosting "." on port 8000...
Ctrl-C to stop.


Go
➜ go get github.com/sno6/gommand
➜ $GOPATH/bin/gommand 'http.ListenAndServe(":8000", http.FileServer(http.Dir(".")))'

\# [Install Caddy](https://caddyserver.com/docs/install)
➜ caddy -port 8000


Unix tools
while true ; do nc -l 8000 &#x3C; test.html ; done


Javascript
➜ npm install -g http-server
➜ http-server -p 8000

\# or
➜ npm install -g node-static
➜ static -p 8000

\#or
➜ npm install -g serve
➜ serve .




Serve a directory with HTTP

Binding to <code>0</code> assigns you a random port
<div class="remark-highlight"><pre class="language-rust"><code class="language-rust">use std::net::TcpListener;
let listener = TcpListener::bind("localhost:0")?;
println!("{}", listener.local_addr()?.port()); // 51673
</code></pre></div>
<div class="remark-highlight"><pre class="language-go"><code class="language-go">conn, _ := net.Listen("tcp", "localhost:0")
fmt.Println(conn.Addr().String()) // 127.0.0.1:51930
</code></pre></div>
Both Rust and Go frameworks support passing that listener to a HTTP server.

Binding to 0 assigns you a random port
use std::net::TcpListener;
let listener = TcpListener::bind("localhost:0")?;
println!("{}", listener.local_addr()?.port()); // 51673

conn, _ := net.Listen("tcp", "localhost:0")
fmt.Println(conn.Addr().String()) // 127.0.0.1:51930

Both Rust and Go frameworks support passing that listener to a HTTP server.


Bind to a random port

<h2 id="connecting"><a href="#connecting" aria-hidden="true" tabindex="-1"></a>Connecting</h2>
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown"># connect to database pokemon as user postgres
psql pokemon postgres

pokemon=# \conninfo
You are connected to database &#x26;quot;pokemon&#x26;quot; as user &#x26;quot;postgres&#x26;quot; via socket in &#x26;quot;/var/run/postgresql&#x26;quot; at port &#x26;quot;5432&#x26;quot;.
</code></pre></div>
<hr>
<h2 id="listing-databases-and-tables"><a href="#listing-databases-and-tables" aria-hidden="true" tabindex="-1"></a>Listing databases and tables</h2>
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown"># List databases
postgres=# \l

# Connect to database
postgres=# \c pokemon

# List tables
# \dt+ is also an option
pokemon=# \dt
 List of relations
 Schema | Name | Type | Owner
--------+------------------+-------+----------
 public | _sqlx_migrations | table | postgres
 public | pokemon | table | postgres
(2 rows)</code></pre></div>
<hr>
<h2 id="schema-of-a-table"><a href="#schema-of-a-table" aria-hidden="true" tabindex="-1"></a>Schema of a table</h2>
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">pokemon=# \d pokemon
 Table &#x26;quot;public.pokemon&#x26;quot;
 Column | Type | Collation | Nullable | Default
-------------------------+---------+-----------+----------+---------
 id | integer | | not null |
 name | text | | not null |
 description | text | | |
 shakespeare_description | text | | |
Indexes:
 &#x26;quot;pokemon_pkey&#x26;quot; PRIMARY KEY, btree (id)
 &#x26;quot;pokemon_name&#x26;quot; UNIQUE, btree (name)
 &#x26;quot;pokemon_name_key&#x26;quot; UNIQUE CONSTRAINT, btree (name)</code></pre></div>
<hr>

Connecting
\# connect to database pokemon as user postgres
psql pokemon postgres

pokemon=# \conninfo
You are connected to database \&quot;pokemon\&quot; as user \&quot;postgres\&quot; via socket in \&quot;/var/run/postgresql\&quot; at port \&quot;5432\&quot;.



Listing databases and tables
\# List databases
postgres=# \l

# Connect to database

postgres=# \c pokemon

# List tables

# \dt+ is also an option

pokemon=# \dt
List of relations
Schema |       Name       | Type  |  Owner
\--------+------------------+-------+----------
public | \_sqlx_migrations | table | postgres
public | pokemon          | table | postgres
(2 rows)


Schema of a table
pokemon=# \d pokemon
                       Table &#x26;quot;public.pokemon&#x26;quot;
         Column          |  Type   | Collation | Nullable | Default
\-------------------------+---------+-----------+----------+---------
 id                      | integer |           | not null |
 name                    | text    |           | not null |
 description             | text    |           |          |
 shakespeare_description | text    |           |          |
Indexes:
    &#x26;quot;pokemon_pkey&#x26;quot; PRIMARY KEY, btree (id)
    &#x26;quot;pokemon_name&#x26;quot; UNIQUE, btree (name)
    &#x26;quot;pokemon_name_key&#x26;quot; UNIQUE CONSTRAINT, btree (name)


Using PSQL (Postgres CLI)

Inspired by <a href="https://simonwillison.net/2021/May/2/one-year-of-tils/">this post</a>
I've tried keeping them before, but I stop after some time. I think the structure I tried before didn't work.
<ul>
<li>One .md file per subject</li>
<li>One file organised chronologically</li>
</ul>
Both got too long and not searchable. The new approach is organised by folder, with each file being short.
I don't think I've ever agreed so hard with something.
<blockquote>
TIL stands for Today I Learned. The thing I like most about TILs is that they drop the barrier to publishing something online to almost nothing.
</blockquote>
<blockquote>
If I’m writing a blog entry, I feel like it needs to say something new. This pressure for originality leads to vast numbers of incomplete, draft posts and a sporadic publishing schedule that trends towards not publishing anything at all.
</blockquote>
<blockquote>
The bar for a TIL is literally “did I just learn something?”—they effectively act as a public notebook.
</blockquote>
<blockquote>
They also reflect my values as a software engineer. The thing I love most about this career is that the opportunities to learn new things never reduce—there will always be new sub-disciplines to explore, and I aspire to learn something new every single working day.
</blockquote>
<blockquote>
My hope is that by publishing a constant stream of TILs I can reinforce the idea that even if you’ve been working in this industry for twenty years there will always be new things to learn, and learning any new trick—even the most basic thing—should be celebrated.
</blockquote>

Inspired by this post
I've tried keeping them before, but I stop after some time. I think the structure I tried before didn't work.

One .md file per subject
One file organised chronologically

Both got too long and not searchable. The new approach is organised by folder, with each file being short.
I don't think I've ever agreed so hard with something.

TIL stands for Today I Learned. The thing I like most about TILs is that they drop the barrier to publishing something online to almost nothing.


If I’m writing a blog entry, I feel like it needs to say something new. This pressure for originality leads to vast numbers of incomplete, draft posts and a sporadic publishing schedule that trends towards not publishing anything at all.


The bar for a TIL is literally “did I just learn something?”—they effectively act as a public notebook.


They also reflect my values as a software engineer. The thing I love most about this career is that the opportunities to learn new things never reduce—there will always be new sub-disciplines to explore, and I aspire to learn something new every single working day.


My hope is that by publishing a constant stream of TILs I can reinforce the idea that even if you’ve been working in this industry for twenty years there will always be new things to learn, and learning any new trick—even the most basic thing—should be celebrated.


Collect TIL snippets

Starting <a href="https://blog.rust-lang.org/2022/01/13/Rust-1.58.0.html#captured-identifiers-in-format-strings">Rust 1.58</a>
<blockquote>
Format strings can now capture arguments simply by writing <code>{ident}</code> in the string.
</blockquote>
<h2 id="examples"><a href="#examples" aria-hidden="true" tabindex="-1"></a>Examples:</h2>
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">let x = &#x26;quot;world&#x26;quot;;
println!(&#x26;quot;Hello {x}!&#x26;quot;); // Hello world

let items = vec![10, 20, 30];
println!(&#x26;quot;{items:?}&#x26;quot;) // Debug printing
println!(&#x26;quot;{items:#?}&#x26;quot;) // Pretty printing</code></pre></div>
Optionally, flags can be specified like <code>{identifier:flags}</code>
<h2 id="alignment"><a href="#alignment" aria-hidden="true" tabindex="-1"></a>Alignment</h2>
<ul>
<li>print item min-width of 10 - <code>println!("{item:10})</code></li>
<li>same, but center align - <code>println!("{item:^10})</code></li>
<li>same, but right align - <code>println!("{item:>10})</code></li>
</ul>
<h2 id="formatting-numbers"><a href="#formatting-numbers" aria-hidden="true" tabindex="-1"></a>Formatting numbers</h2>
<ul>
<li>Always print the sign of the number - <code>println!("{y:+}");</code></li>
<li>Hex, binary and octal - <code>println!("{y:#x} {y:#b} {y:#o}");</code></li>
<li>Float precision. 5 digits after decimal points - <code>println!("{z:.5}")</code></li>
</ul>
Sources
<ul>
<li><a href="https://www.rustnote.com/blog/format_strings.html">Format Strings in Rust 1.58</a></li>
<li><a href="https://news.ycombinator.com/item?id=29932467">nicoburns on HN</a></li>
</ul>

Starting Rust 1.58

Format strings can now capture arguments simply by writing {ident} in the string.

Examples:
let x = &#x26;quot;world&#x26;quot;;
println!(&#x26;quot;Hello {x}!&#x26;quot;); // Hello world

let items = vec!\[10, 20, 30];
println!(\&quot;{items:?}\&quot;)  // Debug printing
println!(\&quot;{items:#?}\&quot;) // Pretty printing

Optionally, flags can be specified like {identifier:flags}
Alignment

print item min-width of 10 - println!("{item:10})
same, but center align - println!("{item:^10})
same, but right align - println!("{item:>10})

Formatting numbers

Always print the sign of the number - println!("{y:+}");
Hex, binary and octal - println!("{y:#x} {y:#b} {y:#o}");
Float precision. 5 digits after decimal points - println!("{z:.5}")

Sources

Format Strings in Rust 1.58
nicoburns on HN


String formatting

This needs the nightly compiler to be installed but not active
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ cargo install cargo-expand
➜ cargo expand
</code></pre></div>

This needs the nightly compiler to be installed but not active
➜ cargo install cargo-expand
➜ cargo expand


Expand Rust macros

<h2 id="embedded-test-module"><a href="#embedded-test-module" aria-hidden="true" tabindex="-1"></a>Embedded test module</h2>
These tests can interact with private variables. Unit tests go here.
<div class="remark-highlight"><pre class="language-rust"><code class="language-rust">// Some code I want to test

#[cfg(test)]
mod tests {
 // Import the code I want to test
 use super::*;

 // My tests
}
</code></pre></div>
<h2 id="external-tests-folder"><a href="#external-tests-folder" aria-hidden="true" tabindex="-1"></a>External <code>tests</code> folder</h2>
These tests have the same level of access as an external crate importing this one. Integration tests go here.
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ ls
src/
tests/
Cargo.toml
Cargo.lock
...
</code></pre></div>
Combine all tests into one module - <code>crate/tests/integration/*</code> instead of multiple <code>.rs</code> files in <code>crate/tests/</code>. Otherwise, each individual file is compiled into a separate binary.
<ol start="3">
<li>Doc</li>
</ol>
Same level of access as #2
<div class="remark-highlight"><pre class="language-rust"><code class="language-rust">/// Check if a number is even.
/// ```rust
/// use zero2prod::is_even;
///
/// assert!(is_even(2));
/// assert!(!is_even(1));
/// ```
pub fn is_even(x: u64) -> bool {
 x % 2 == 0 
}
</code></pre></div>
Sources
<ul>
<li><a href="http://zero2prod.com">Zero to Production in Rust</a></li>
<li><a href="https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html">Delete Cargo Integration Tests</a></li>
</ul>

Embedded test module
These tests can interact with private variables. Unit tests go here.
// Some code I want to test

\#\[cfg(test)]
mod tests {
// Import the code I want to test
use super::\*;

 // My tests

}


External tests folder
These tests have the same level of access as an external crate importing this one. Integration tests go here.
➜ ls
src/
tests/
Cargo.toml
Cargo.lock
...

Combine all tests into one module - crate/tests/integration/* instead of multiple .rs files in crate/tests/. Otherwise, each individual file is compiled into a separate binary.

Doc

Same level of access as #2
/// Check if a number is even.
/// ```rust
/// use zero2prod::is_even;
///
/// assert!(is_even(2));
/// assert!(!is_even(1));
/// ```
pub fn is_even(x: u64) -> bool {
 x % 2 == 0 
}

Sources

Zero to Production in Rust
Delete Cargo Integration Tests

Rust test folder location

<ul>
<li><a href="#cargo-add">cargo add</a></li>
<li><a href="#cargo-tarpaulin">cargo tarpaulin</a></li>
<li><a href="#cargo-fmt">cargo fmt</a></li>
<li><a href="#cargo-clippy">cargo clippy</a></li>
<li><a href="#cargo-audit">cargo audit</a></li>
</ul>
These are all good plugins to run in CI. Source: <a href="http://zero2prod.com">Zero to Production in Rust</a>
<ul>
<li><a href="https://gist.github.com/LukeMathWalker/5ae1107432ce283310c3e601fac915f3">Github Actions</a></li>
<li><a href="https://gist.github.com/LukeMathWalker/d98fa8d0fc5394b347adf734ef0e85ec">Gitlab CI</a></li>
</ul>
<hr>
<h2 id="cargo-add"><a href="#cargo-add" aria-hidden="true" tabindex="-1"></a>cargo add</h2>
Add and remove crates to <code>Cargo.toml</code>
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ cargo install cargo-edit

➜ cargo add actix-web
 Updating 'https://github.com/rust-lang/crates.io-index' index
 Adding actix-web v3.3.2 to dependencies

➜ cargo add actix-rt --dev # Dev dependency
</code></pre></div>
</details>
<hr>
<h2 id="cargo-tarpaulin"><a href="#cargo-tarpaulin" aria-hidden="true" tabindex="-1"></a>cargo tarpaulin</h2>
Measures test coverage.
As of May 2021, only works on Linux-x86.
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ cargo install cargo-tarpaulin

➜ cargo tarpaulin --ignore-tests
</code></pre></div>
</details>
<hr>
<h2 id="cargo-fmt"><a href="#cargo-fmt" aria-hidden="true" tabindex="-1"></a>cargo fmt</h2>
Format code automatically.
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ rustup component add rustfmt

# Format entire project
➜ cargo fmt

# Run in CI. Fail the build if the code isn't formatted
➜ cargo fmt -- --check
</code></pre></div>
</details>
<hr>
<h2 id="cargo-clippy"><a href="#cargo-clippy" aria-hidden="true" tabindex="-1"></a>cargo clippy</h2>
Get warnings and helpful tips à la Microsoft Clippy.
<details>
 <summary>Snippet</summary>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">➜ rustup component add clippy

# Run on entire project
➜ cargo clippy

# Run in CI. Fail the build if there are any warnings
➜ cargo clippy -- -D warnings
</code></pre></div>
</details>
<hr>
<h2 id="cargo-audit"><a href="#cargo-audit" aria-hidden="true" tabindex="-1"></a>cargo audit</h2>
Audit your dependencies for security issues.
<details>
 <summary>Snippet</summary>
```bash
➜ cargo install cargo-audit
➜ cargo audit
<div class="remark-highlight"><pre class="language-unknown"><code class="language-unknown">&#x26;lt;/details&#x26;gt;

----</code></pre></div>


cargo add
cargo tarpaulin
cargo fmt
cargo clippy
cargo audit

These are all good plugins to run in CI. Source: Zero to Production in Rust

Github Actions
Gitlab CI


cargo add
Add and remove crates to Cargo.toml

    Snippet
➜ cargo install cargo-edit

➜ cargo add actix-web
Updating 'https://github.com/rust-lang/crates.io-index' index
Adding actix-web v3.3.2 to dependencies

➜ cargo add actix-rt --dev # Dev dependency




cargo tarpaulin
Measures test coverage.
As of May 2021, only works on Linux-x86.

    Snippet
➜ cargo install cargo-tarpaulin

➜ cargo tarpaulin --ignore-tests




cargo fmt
Format code automatically.

    Snippet
➜ rustup component add rustfmt

\# Format entire project
➜ cargo fmt

\# Run in CI. Fail the build if the code isn't formatted
➜ cargo fmt -- --check




cargo clippy
Get warnings and helpful tips à la Microsoft Clippy.

    Snippet
➜ rustup component add clippy

\# Run on entire project
➜ cargo clippy

\# Run in CI. Fail the build if there are any warnings
➜ cargo clippy -- -D warnings




cargo audit
Audit your dependencies for security issues.

    Snippet
\```bash
➜ cargo install cargo-audit
➜ cargo audit
&#x26;lt;/details&#x26;gt;

\----


Useful Cargo plugins

<h2 id="current-currency-value"><a href="#current-currency-value" aria-hidden="true" tabindex="-1"></a>Current currency value</h2>
<code>=GOOGLEFINANCE("Currency:USDGBP")</code>
<hr>
<h2 id="historical-currency-value"><a href="#historical-currency-value" aria-hidden="true" tabindex="-1"></a>Historical currency value</h2>
<code>=GOOGLEFINANCE("Currency:USDGBP", "price", DATE(YYYY,MM,DD), DATE(YYYY,MM,DD))</code> for the rates on each day between the two dates.
For the rate on one particular date, in one cell
<code>=index(GOOGLEFINANCE("Currency:USDGBP", "price", DATE(YYYY,MM,DD)), 2, 2)</code>
Sources
<ul>
<li><a href="https://www.howtogeek.com/449986/how-to-convert-currency-in-google-sheets/">How to Convert Currency in Google Sheets</a></li>
<li><a href="https://support.google.com/docs/thread/13455470/google-finance-retrieve-historical-prices-in-a-single-cell?hl=en">Google finance retrieve historical prices in a single cell</a></li>
</ul>

Current currency value
\=GOOGLEFINANCE("Currency:USDGBP")

Historical currency value
\=GOOGLEFINANCE("Currency:USDGBP", "price", DATE(YYYY,MM,DD), DATE(YYYY,MM,DD)) for the rates on each day between the two dates.
For the rate on one particular date, in one cell
\=index(GOOGLEFINANCE("Currency:USDGBP", "price", DATE(YYYY,MM,DD)), 2, 2)
Sources

How to Convert Currency in Google Sheets
Google finance retrieve historical prices in a single cell


Currency conversion

<a href="http://blog.kablamo.org/2015/11/08/bash-tricks-eux/">Source</a>
<h3 id="set--e"><a href="#set--e" aria-hidden="true" tabindex="-1"></a>set -e</h3>
Makes <code>cmd1 &#x26;&#x26; cmd2 &#x26;&#x26; cmd3</code> equivalent to
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">set -e
cmd1
cmd2
cmd3
</code></pre></div>
<hr>
<h3 id="set--u"><a href="#set--u" aria-hidden="true" tabindex="-1"></a>set -u</h3>
The shell prints a message to stderr when it tries to expand a variable that is not set. Also it immediately exits. An interactive shell will not exit.
<hr>
<h3 id="set--x"><a href="#set--x" aria-hidden="true" tabindex="-1"></a>set -x</h3>
The shell prints each command in a script to stderr before running it.
<hr>
<h3 id="set--o-pipefail"><a href="#set--o-pipefail" aria-hidden="true" tabindex="-1"></a>set -o pipefail</h3>
Pipelines fail on the first command which fails instead of dying later on down the pipeline. This is especially good when cmd3 is a command that always succeeds (like echo):
<code>cmd1 | cmd2 | cmd3</code>
<hr>

Source
set -e
Makes cmd1 &#x26;&#x26; cmd2 &#x26;&#x26; cmd3 equivalent to
set -e
cmd1
cmd2
cmd3


set -u
The shell prints a message to stderr when it tries to expand a variable that is not set. Also it immediately exits. An interactive shell will not exit.

set -x
The shell prints each command in a script to stderr before running it.

set -o pipefail
Pipelines fail on the first command which fails instead of dying later on down the pipeline. This is especially good when cmd3 is a command that always succeeds (like echo):
cmd1 | cmd2 | cmd3


Shell execution options

When dates are stored in epoch timestamps, make them human readable like so:
<div class="remark-highlight"><pre class="language-sql"><code class="language-sql">sqlite> select cmd, dir, datetime(when_run, 'unixepoch', 'localtime') from commands where cmd like 'git submodule%';
</code></pre></div>
Source: <a href="https://stackoverflow.com/questions/14629347/how-to-convert-unix-epoch-time-in-sqlite">Stackoverflow</a>.

When dates are stored in epoch timestamps, make them human readable like so:
sqlite> select cmd, dir, datetime(when_run, 'unixepoch', 'localtime') from commands where cmd like 'git submodule%';

Source: Stackoverflow.


Converting datetime in Sqlite

<code>.excel</code> - Display the output of next command in spreadsheet

.excel - Display the output of next command in spreadsheet


Generate CSV from SQLite

In a new database
<div class="remark-highlight"><pre class="language-sql"><code class="language-sql">sqlite> .databases
main: /Users/nindalf/.mcfly/history.db

sqlite> .table
commands schema_versions selected_commands

sqlite> .schema commands
CREATE TABLE commands( id INTEGER PRIMARY KEY AUTOINCREMENT ...

</code></pre></div>
For more, try <code>.help</code>, but I didn't see anything that seemed immediately helpful.

In a new database
sqlite> .databases
main: /Users/nindalf/.mcfly/history.db

sqlite> .table
commands           schema_versions    selected_commands

sqlite> .schema commands
CREATE TABLE commands( id INTEGER PRIMARY KEY AUTOINCREMENT ...



For more, try .help, but I didn't see anything that seemed immediately helpful.


Getting started with SQLite

<ul>
<li>If the character can be expressed in 7 bits, it will be. There's no difference from a pure ASCII string and a UTF-8 string containing those characters.</li>
<li>If the character needs up to 11 bits, it will be spread across 2 bytes with padding.</li>
<li>The padding indicates that this is one codepoint spread across two characters, not 2 1-byte codepoints.</li>
<li>Take á. It corresponds to 225. Or '00011100001' in binary.</li>
<li>With padding this becomes the 2 bytes '110 00011' and '10 100001'.</li>
<li>Unicode aware systems will render these two bytes as as á.</li>
<li>Unaware systems will interpret each byte as a separate codepoint and print ��.</li>
</ul>
For codepoints that need 12-16 bits will be spread across 3 bytes. And codepoints 17-21 bits will be spread across 4 bytes.
<img src="/unicode.png" alt="Unicode Padding">
Source - <a href="https://sethmlarson.dev/blog/utf-8">How does UTF-8 turn “😂” into “F09F9882”?</a>


If the character can be expressed in 7 bits, it will be. There's no difference from a pure ASCII string and a UTF-8 string containing those characters.
If the character needs up to 11 bits, it will be spread across 2 bytes with padding.
The padding indicates that this is one codepoint spread across two characters, not 2 1-byte codepoints.
Take á. It corresponds to 225. Or '00011100001' in binary.
With padding this becomes the 2 bytes '110 00011' and '10 100001'.
Unicode aware systems will render these two bytes as as á.
Unaware systems will interpret each byte as a separate codepoint and print ��.

For codepoints that need 12-16 bits will be spread across 3 bytes. And codepoints 17-21 bits will be spread across 4 bytes.

Source - How does UTF-8 turn “😂” into “F09F9882”?


Unicode

<h2 id="server-setup"><a href="#server-setup" aria-hidden="true" tabindex="-1"></a>Server setup</h2>
<div class="remark-highlight"><pre class="language-bash"><code class="language-bash">apt update
apt install wireguard-tools

# On the peer that will act as the "server", enable IPv4 forwarding:
sysctl -w net.ipv4.ip_forward=1

# write the server config to /etc/wireguard/wg0.conf
vim /etc/wireguard/wg0.conf
wg-quick up wg0

# To bring down the network
# wg-quick down wg0
</code></pre></div>
<h2 id="client-set-up"><a href="#client-set-up" aria-hidden="true" tabindex="-1"></a>Client set up</h2>
MacOS - install the Wireguard app, add the client config.
<h2 id="making-sure-things-are-working"><a href="#making-sure-things-are-working" aria-hidden="true" tabindex="-1"></a>Making sure things are working</h2>
Server
<code>nc -l 8000</code>
Client
<code>nc 10.0.0.1 8000</code>
<h2 id="source"><a href="#source" aria-hidden="true" tabindex="-1"></a>Source</h2>
The <a href="https://wiki.archlinux.org/title/WireGuard">excellent guide</a> on ArchWiki. Specifically the section about VPN.

Server setup
apt update
apt install wireguard-tools

\# On the peer that will act as the "server", enable IPv4 forwarding:
sysctl -w net.ipv4.ip_forward=1

\# write the server config to /etc/wireguard/wg0.conf
vim /etc/wireguard/wg0.conf
wg-quick up wg0

\# To bring down the network
\# wg-quick down wg0


Client set up
MacOS - install the Wireguard app, add the client config.
Making sure things are working
Server
nc -l 8000
Client
nc 10.0.0.1 8000
Source
The excellent guide on ArchWiki. Specifically the section about VPN.
