• a.bout
  • t.witter
23 Mar 2019 c.e.
10 Years of Twitter

I signed up for Twitter on March 5, 2009. I can't remember why or how I heard about Twitter, or who thought it'd be a good idea for me to get on it. I was in college at the time, and worked as a TA for a professor in the school of Management Information Systems. I strongly suspect she asked me to get on to help promote events that she was running and the like, but it's hard to say. One of the most defining memories I have of it was tweeting from the bus via flip-phone SMS messages to a short code.

Anyway. I've been on the platform, off an on for the last decade. In honor of a decade of tweeting, I downloaded my archive and compiled some stats about how I've used the platform since then.


I downloaded the entirety of my Twitter archive from the settings page on March 19th, and then used this python script to sort through things. It's not really well stitched together, but if you want to try it out, here's a general set of commands that I run to get it going:

$ python3
>>> exec(open('script.py').read())
>>> tweets = load_tweets('tweets.csv')
>>> tweets[-1]['text'] # this is your first ever tweet!


Ok, so here's some basic @niftynei tweet stats, from Mar 5, 2009 to Mar 19, 2019:


    Total tweets: 18,780
    Retweets: 2,334
    'Self authored' tweets: 16,446

    Total characters: 1,187,164
    Average characters per tweet: 72
    Total 'words' tweeted, excluding retweets: 186,316


    Replied to: 1,448 accounts
    Most replied to: myself (3,532)
    Top five replied to, links are to first reply ever:
      @zmagg (245)
      @vgr (189)
      @jc4p (169)
      @turtlekiosk (130)
      @lchamberlin (108)

    Longest tweet (by characters): 302 chars, 278 w/o link     Longest tweet (by words): 56 words, 271 characters

    Longest gap between tweets: 319 days, 1:17:08
    Second longest gap: 138 days, 4:36:53
    Shortest gap between tweets: 0 days, 00:00:00
    Median gap length: 0 days, 0:06:21
    Average gap length: 4:01:30

    Most tweets in a day, with retweets: 120


I also found some 'collections' of tweets that I did, based on hashtag. Here's a set of 'quotes' from 'Bob Moses, Software Project Manager' I wrote in 2015, right after reading Caro's Robert Moses book.

  • Jill who'd you tell about our plans to shut down that API? Well, Tim Cook just Slacked me about it #BobMosesSoftware
  • We've already invested two weeks. If we cut it now, it'd be a waste of developer, server, and your time #BobMosesSoftware
  • Make the button blue? Impossible. #BobMosesSoftware
  • These AB test numbers have links to private interests. #BobMosesSoftware
  • Look, the PM who sponsored this feature was a pitiful excuse for a person, and a crank. #BobMosesSoftware

Here's a collection of #words, some with more meaning than others:

  • ex pose say #words
  • bingo buzzchain bandit #words
  • concurrently battling an ur-reductive mental trip #words
  • physical manifestation at the hilt of representation #words
  • strong bold memories of Europe sunshine in the spring #words
  • It's a trinidadian dance funk kind of afternoon of the likes only Pitbull can satisfice #words
  • axiotic dimensional #words
  • "There were 6 right answers but I only knew one" #words
  • some trips you don't come back from 🍃 #words
  • dispatches from the hallowed halls of productivity theater #words
  • Vinculated to the predicated #words
  • What would it mean? To never know the joy of driving a nail, firmly flush with the wood top of your coffin. #words
  • deliciously voyeuriste #words
  • Skewed perceptions: a relativistic model #words
  • Cognoscenti is probably the best word. #words
  • The opiate for the masses. The opiate for the masters. The opiate for the missus. #words
  • tragi-comic #words
  • sliding swiftly into the obdurate past #words
  • spinning dystrophies of inalienability #words

Expanding on the literary theme, here's a series of sentences that might make good starts to novels:

  • I can't stop thinking about the silver Mary I saw at the Sacre Couer #novelstarts
  • And thus began my long love affair with the Q train. #novelstarts
  • She started pointedly: my guilty pleasure is stalking you. #novelstarts
  • It was all the things I had not done yet that kept me awake, instead of all the things that I had that put me to sleep. #novelstarts
  • all of our conversations were just lines of this screen play i was unwittingly writing called You & Me #novelstarts
  • The saddest sadist you'll ever meet lives ... #novelstarts
  • Character assassination was the strategy. Twitter bots, the chosen methodology. #novelstarts
  • "I just want to spend the rest of tonight at disco karoke with Hotline Bling on repeat," she said, breaking into a slow robot. #novelstarts
  • El Doctor te veras. #novelstarts

Heat of the Tweet

Finally, here's a 'heat map' of tweeting from the last 10 years. And yes, I did swipe the formatting from the Github repo's heat map.

2009 Mon Wed Fri
2010 Mon Wed Fri
2011 Mon Wed Fri
2012 Mon Wed Fri
2013 Mon Wed Fri
2014 Mon Wed Fri
2015 Mon Wed Fri
2016 Mon Wed Fri
2017 Mon Wed Fri
2018 Mon Wed Fri
2019 Mon Wed Fri
#twitter #blogging #stats
9 Feb 2019 c.e.
A Taxonomy of Lightning Nodes

It never ceases to amaze me how little the general crypto population knows about how the lightning network works, so I thought I'd write down something that's been quite obvious to me for a while, with the hopes of influencing others to see it my way.

Lightning is a network of node operators. Each node has a wallet with funds, that are then apportioned amongst a set of channels to other nodes. Each channel that is opened has a balance, and each node in the channel has the right to spend a certain amount in that channel. This "right to spend" gives every channel a directionality to it. In other words, which direction the funds can move at any given moment depends on which side has the right to spend them. For this reason, the Lightning network is a directed graph.

Every payment that moves through the system changes the balance of payments in every channel that it flows through. As payment volume grows, managing the 'flows' and ability to send payments from one node to another will become an important and non-trivial management task.

Drawing Lines Between Nodes

A key to understanding how these flows will affect ability to make payments is to understnad that not every Lightning node has the same 'goal'.

In fact, you can classify these nodes into three distinct groups. Each of these groups represents a different policy on liquidity in their channel balances. As such, the actions they will each regularly perform on their channel balances will be distinct. A channel balance is only useful if it allows you to do what you need to on the network, and each of these three actors will have different goals.

Theses three node groups are:
- consumers
- vendors
- liquidity providers


This is probably the most intuitive group to understand, since it's every one of us. A consumer is a net supplier of funds to the Lightning network. On a whole, they spend more money over Lightning than they receive. There is a certain amount of exchange that happens among nodes of this type, but this amount is dominated by their outflow to Vendor nodes. Typically, their payments will be to a relatively closed set of repeated contacts.

Generally, the actions a consumer takes will be one of:
- Adding more money to their wallet/outgoing channel balances
- Sending payments to vendors
- Creating new channels to pay new vendors

The apps that these users use are typically mobile wallets and web browser extensions. They're generally interested in centralized/custodial services. Probably not running their own node unless it is their mobile client or they've invested in a small home node.


This is the Amazons and Blockstream stores of the network. A vendor is a net drain of funds on the Lightning Network -- they receive more payments in than they send out. They are typically receiving inflows in exchange for a good or service, which means that they'll be withdrawing funds from their channels to cover their costs.

Generally, the actions a vendor takes will be one of:
- Withdrawing money from their channels
- Opening channels with liquidity providers, to get inbound capacity
- Originating invoices

The apps and infrastructure that these vendors use will generally be a bit more intensive and always on than consumers. Their ability to transact will be closely tied to their ability to reliably source inbound capacity. Backups and watchtowers are of a bigger concern to these users than to consumers.

Liquidity Providers

This is the HODLers, people who have a chunk of crypto that they want to put to work but aren't interested in spending it and don't really have much of anything to sell. They've got the time, know-how, and resources to set up a more 'industrial strength' node than the general 'consumer' population. They're interested in writing custom algorithms that can help them figure out how to price their liquidity and are willing to spend the time and energy (generally speaking) to figure out what configuration of channel balances and flows will bring them the best return on their node setup, in terms of routing fees. They earn money by providing liquidity between consumers and vendors.

Generally, actions a liquidity provider will take are:
- Opening new channels to vendors, to provide inbound capacity
- Advertising liquidity
- Rebalancing their channels between vendor + consumer accounts
- Network analysis to discover lucrative avenues to open/create new channels

In Exitus

It's my understanding that the Lightning Network needs all of these types of nodes to function. Providing a visible market for liquidity will make these roles even more apparent. I'm incredibly excited about the inclusion of liquidity advertising in the 1.1 spec, as it will give another lever for liquidity providers and vendors to make decisions about how to most effectively allocate channel balances across the network, in a decentralized and transparent manner.

#lightning #markets #liquidity #taxonomy
28 Jan 2019 c.e.
Reflections on the Art of JSON in Golang

Last month, I put a good bit of time into writing a little library to help bridge the gap between the requirements of JSON-RPC's spec and Go.

The Go standard library provides functionality for the version 1.0 of JSON-RPC. There is no standard library implementation for the 2.0 spec, but there's plenty of other implementations, some of which seem to get pretty close to the idioms that I landed upon for my version of it. I ended up writing my own library for a few reasons. First off, I wanted some practice implementing a spec. The work I'm looking to do for lightning over the next few weeks is basically spec writing and implementation; it seemed like a good idea to get some practice following a very simple and well documented spec like the JSON-RPC 2.0 spec.

Secondly, my motivation for needing a JSON-RPC implementation is that I was looking to write a 'driver' for the new plugin functionality that Christian Decker has been adding to c-lightning. c-lightning's plugins have a few very specific needs[1] that would likely require modifying another JSON-RPC implementation. Plus there's the overhead of figuring out how another person's library works.

I leveraged the json encoder/decoder from the Go standard library as much as possible. The trickiest bit was getting a good idiom put together for how parameters are declared and marshalled into command calls. There's a lot more that went into putting the whole plugin/rpc thing together, but I think for this post it'd be the most delightful to just walk through the design decisions that I made for the way the params parsing works.

Problem Statement

Let's talk a bit about what's going on during a JSON-RPC command message exchange. The general gist is that there's a client who wants to execute a method on the server. In order to do this, we need to tell the server what method we'd like to call (by providing a name) and then also passing in any and all of the information that the method needs (these are typically called 'parameters' or 'arguments'. The JSON-RPC spec terms them params).

Our job then is to provide an interface such that the client can smoothly call a method and then receive a response from the server. The ideal interface for such an interaction would look identical to any normal method call. For example:

func hello(greeting string, to User) (string, error) {
    // magically invoked on the server
    return "result", nil

Go provides a json marshaler/unmarshaler, a package called encoding/json. The problem is that the marshaler works on structs, not method signatures.

Instead, jrpc2 takes the tack of asking users to write their method calls as structs. Here's how the hello method that we saw above would be rewritten as a struct.

type HelloMethod struct {
    Greeting string `json:"greeting"`
    To *User        `json:"user,omitempty"`

Each of the method parameters is now represented as a public struct field. When we send this across the wire, we'd expect our library to generate the following json:

            "last_name": "Neal"

We need a way to signal to our library that this is in fact a 'method' that our jrpc2 library knows how to marshal into a valid command request object. We do that with an interface, that defines a single method, Name(). Any struct that implements this method will be considered ok for sending over the wire to the server.

func (r *HelloMethod) Name() string {
    return "hello"

We still need a way to pass this method call to the server, but from a client perspective that's all we need in terms of defining a new method.

On the Server End

c-lightning's plugin framework requires your app to serve as both a JSON-RPC client and server, since users can invoke method calls from c-lightning that are then passed to your plugin to execute. Server RPC method objects are mostly the same as above, with two additional methods added to the interface, New and Call.

When the server receives a request from a client, it 'inflates' the json request into a ServerMethod object. The New method gives you the ability to do any initialization or setup needed for the new instance. If there's state that needs to be shared between instances of the ServerMethod, you can pass them along here. Here's an example of where you want a New version of the GetManifestMethod to have access to the plugin object.

// definition
type GetManifestMethod struct {
    plugin *Plugin  // private so it's not mistaken for a method parameter

func (gm *GetManifestMethod) New() interface{} {
    method := &GetManifestMethod{}
    method.plugin = gm.plugin
    return method

This is nice because it lets you share state between method calls. Then there's the actual Call part of the ServerMethod, which obviously is where you do work. Since the 'inflated' struct is 'passed in' as the object of the call (i.e. the whole (gm *GetManifestMethod) declaration, you have access to all of the parameters that were sent by the client.

func (gm *GetManifestMethod) Call() (jrpc2.Result, error) {
    // example of using the plugin obj
    for i, sub := range gm.plugin.subscriptions {
        // ...
    return result, err

If you return a non-nil error from the Call, the server will ignore the result and send the client back an error object. As a final note, if you want your Result to be formatted for json correctly, you'll need to add good json annotations for its fields. We use the default encoding/json package to marshal and umarshal everything over the wire.

A Few Things on The Way to the Forum

The trickiest part of the whole jrpc2 mechanism is the custom marshalling for the param struct. The JSON-RPC spec defines two different ways that params can be passed from the client: either as an ordered array or as a named dict. i.e.

// As an ordered array 
"params": [1, 2, "hello", {"inner":"object"}]

// As a named dict
"params": {"first": 1, "second": 2, "greeting":"hello", "extra":{"inner":"object"}}

Basically, we're wrapping client calls in an outer object, with the 'method struct' being serialized into the params. jrpc2 includes methods to serialize calls as either an ordered array or a named dict, but defaults to the named dict when used as a client. It's worth noting that the order of appearance of fields in a method struct is how they'll appear in the array. If you re-arrange the ordering, and have switched it to use 'vectorized params' (aka an ordered array) then they should be switched in the param call.

Reflection Dragons

In order to do this correctly, I ended up digging in pretty hard to the reflect library. There's a bunch of nuance around deflating and re-inflating objects from json that I really struggled to find good resources on. Most golang articles on reflection stop and start with Rob Pike's article on the Go Blog, The Laws of Reflection, but it doesn't dig in much beyond the basics.

Re-creating a new version of the method struct is fairly straight forward, you can just call the New method. However, for any param that is a pointer on the method struct, we have to allocate a new 'extra' object and then run the json Unmarshaler on it. There's a few steps to this.

First, we need to determine what type of object we should be inflating. We can use the method struct's field declaration to determine what type of new struct to inflate.

When you 'inflate' a new object from a field type, it initially comes to you without a pointer address, because no memory has been assigned to it yet.

Short aside. Originally, method structs on the server didn't have a New command, instead I inflated it directly. Figuring out how to do this took me some amount of time. Unfortunately, I replaced it with the New method, as I wanted a way to be able to share objects across every method call, and then I completely (and accidentally) destroyed my git repo and lost my commit history so I can't show it to you exactly but, it involved inflating a new copy from an existing one and then figuring out a way to get it assigned to an address space so that we'd have a pointer to pass around. This isn't such a problem for sub-fields on the struct, since creating a new one allocates space for all of its member fields.

The only place that you need to do allocate a new object is for a field on a struct that's a pointer. Here's a short example.

// Method struct to inflate
type IdkMethod struct {
    Clue *Clue

When we're serializing this to json, we'll pass the Clue object as serialized json (if the pointer exists) or pass null if there is nothing assigned. On the server side, we need to 'inflate' this back into a Clue object, with a pointer that we can assign to the new IdkMethod object. Here's how we do it.

if fVal.IsNil() {

We use reflect.New to create a new version of the type of field. We have to use Type().Elem() because the type is a pointer -- we want to create a new struct of the type of the element that the pointer is pointing at, not a new 'pointer to element'. reflect.New returns a pointer to the new object that it has just allocated, which we can directly set the value of that field (e.g. fVal) to.

Another short aside, I don't know how you're supposed to figure out how any of this more complex pointer magic works if you've never dealt with pointers on a fairly intimate level. Language level abstractions are great ...until you fall into the pit of object marshalling.

There's a lot of other little neat things that I ended up needing to figure out to do, like filling in a slice or map. Briefly, here's the code for inflating a set of map objects:

    // the only types of maps that we can get thru the json
    // parser are map[string]interface{} ones
    mapVal := value.(map[string]interface{})
    keyType := fVal.Type().Key()
    for key, entry := range mapVal {
            eV := reflect.New(fVal.Type().Elem()).Elem()
            kV := reflect.ValueOf(key).Convert(keyType)
            err := innerParse(targetValue, eV, entry)
            if err != nil {
                    return err
            fVal.SetMapIndex(kV, eV)

You can find all of these great things and more at work in the innerParse function of the jrpc2 library. Currently it lives here.

In Exitus

I'm half-convinced there's a construction of param parsing where you only need to declare the method, and you can somehow 'shadow compose' the request objects that I settled on above. But! After using the library for making a few plugins plus the RPC object for c-lightning calls, I think there's a nice balance between declarativeness and flexibility. Particularly, while at first it seemed a bit redundant, having an explicit Name() function hook for the Method objects nicely decouples the declared method name from whatever is the nicest way to express it in Go.

By way of example, there's an RPC method on c-lightning called dev-rhash. With the Name() idiom, it's easy to handle this:

func (r *DevRhashRequest) Name() string {
    return "dev-rhash"

Under the 'more syntactically sugarful' and also imaginary (because I'm not entirely certain you can do it) way that I've been imagining, you'd have to write the Go method like this:

func dev-rhash() string {

And then every place you wanted to use it, you'd have all kinds of ugly dev-rhash() calls. Say nothing of the fact that Go uses upper and lower case letters on functions and objects to denote the 'visibility' of a method -- as written you wouldn't be able to call this method outside of the containing package, which for a library function renders it quite useless. Anyway, I think the API that I landed on is a decent one, for this reason alone, almost.

[1] The c-lightning plugin to c-lightning relationship is a bit complicated. A plugin is both a 'server', in JSON-RPC parlance, and a client. For most of the commands and the like, a plugin plays the role of a server, providing methods that c-lightning can call out to. Notifications from c-lightning to your plugin take advantage of the client -> server notification framework that's included in the JSON-RPC spce. The one exception, so far at least, is that you can pass back logs from the plugin to c-lightning, such that plugin logs will appear in the getlogs command on c-lightning. In order to do this, your plugin sends a log notification to the c-lightning parent, which inverts the server -> client relationship.


I cobbled together info on how the more magique aspects of reflection works from a variety of places. Here's some of the ones that I found the most helpful.

How to create an object with Reflection via reddit
Writing into a slice via blog
The exhaustive list of reflection tests in the golang source golang.org
And of course the seminal "The Laws of Reflection" Go Blog

#json #golang #encoding #static #reflection
28 Dec 2018 c.e.
The Demo at 50: Looking Forward

December 9th, 2018 marked the 50th anniversary of Doug Englebart's Mother of All Demos. (You can watch the actual demo on YouTube or read about it on Wikipedia). To commemorate the occassion, Doug Englebart's daughter and some of his long time collaborators pulled together an all day symposium for the still surviving demo crew members and other early Internet luminaries. I, like all the other lumpenproletariat of the modern Silicon Valley, bought a ticket to attend.

The day's festivities were held at the Computer Science Museum down in Mountain View, about a forty minute drive from San Francisco early on a Sunday morning. My friend and I arrived early, which gave us time to grab coffee, almost front-row seats at one of the twenty or so ten-person tables that filled the hall that the day's lectures would be held in, ogle the paper signs on tall cocktail tables that marked where the in-person demos of similar tech projects would be held, and traipse down to the first floor museum exhibit, one of Google's prototypes for a self-driving car.

It was mostly a day of reminiscing, with a few more modern speakers talking about projects they're currently working on to make the Web a more annotated and sourceable place. The main drive of most of the projects seemed to be HyperLinking. Ted Nelson, the closing speaker and an early Web researcher, is still going on about how HyperLinks should have been bi-directional.

On the System Itself

There was a panel discussion from a few original ARC researchers. We had a hardware guy, a couple of software guys, and Doug Englebart's daughter, Christina Englebart. The hardware guy, Martin Hardy, had created a hypterlinked diagram to show us all how the original demo computer system had been constructed. The demo itself was held at a hall in San Francisco -- the actual computer mainframe lived in a research center in Menlo Park, south of SF by a few tens of miles. In the demo, the computer screen printout and video feeds from several different cameras are broadcast onto the screen so that we can see researchers in Menlo Park, as well as a camera feed pointed at Doug's face, on stage. In order to get these video streams to show, they had to pipe all the data back to the mainframe in Menlo Park, where the computer composed the stream to feed to projector. They used a microwave tower to beam the feeds, as the Internet hadn't been invented yet. It'd be a few decades until fast speed Internet was installed between here there and everywhere.

Once the reminiscing and story recounting was done, they had a little bit of time to ask to audience for questions. There may have been a few, but the only one I remember was from a man who wanted to know, definitively, what room the Demo had occurred in. Given the spirited debate that follows, it seems that the biggest controversy surrounding the event was the actual location that it happened at. Good thing we have a video recording of it, otherwise we may not be sure that it happened at all.

Another gizmo that came up during the day was the projector machine that the group developed that could stop a film strip on a single frame. You used to not be able to pause film projectors because the heat from the bulb would burn the frame that you stopped on. Anyway, somehow the ARC research group was able to build a projector that would let you stop the film at any arbitrary point. One day, someone was showing the presentation to a group that wanted to know more about the project and happened to stop the film exactly on a frame that showed the computer had crashed. In the middle of the Demo. If you watch the film, you may be surprised to hear this, as you'd know that during the Demo, the whole project works pretty flawlessly. Well, it turns out that it did, in fact, crash. The reason you can't see it when watching the film is one that the digitization process probably lost that exact frame and two that the computer system they built was so incredibly quick to come back online that it restarted without anyone noticing. Turns out that the computer system crashed so frequently that they tuned it to come back so that no one would notice it had even failed. It's hard to square that with how slow my laptop takes to start some days.

Web Researchers, Then and Now

There were a number of great panel discussions about web technologies from a host of different web pioneers. Even Alan Kay made an appearance -- they put him on one of those teleconferencing robots and he beamed in from his home. He got up a few times to get a thing; I wasn't sitting quite close enough to get a good look at the books on the bookshelf behind him.

I think the rowdiest panel was probably the one with Wendy Hall, a UK researcher who's been working on web hyperlinking technology projects since the Demo, and Peter Norvig, the chief researcher for search at Google. There was a strange amount of hostility in the room towards Silicon Valley Money, chiefly coming from the people, a majority in the room to be clear, who had spent their lives in academia and decidedly not made it rich on the Internet and Software boom that came to be after their demos. Unfortunately, I don't remember the exact issues that showcased Hall and Norvig's ideological differences, but I believed it turned around a responsibility to filter out fake news and propaganda. Wendy had done a lot of work on being able to easily show provenance for information, so it was interesting to see her in conversation with Norvig, big wig of Google Search. As an aside, I'm not sure where the line on authoritarianism comes down between censorship and the promotion of truth, but we definitely seemed to be flirting with it. Even Vint Cerf had some strong things to say about the quality of information on the Internet.

Yet another presenter put up on the screen a Mosaic listserve email from Marc Andreesen, one that talked about how he had hacked into the browser the ability to add annotations to any webpage and asking for beta testers.[1] On page annotations seemed to be one of the biggest wishes from the bevy of Internet luminaries we heard from. Well, that and a way to get rid of fake news. Dan Whaley from Hypothes.is was on a panel as well. It was interesting, to me, to see modern efforts to bring annotation to the web. I'm not sure what every website would be like with a comments section, but it seems that the effort to find out hasn't died out yet.


One thing that Doug's daughter really brought home for me was the question of what the impact and legacy was of the Demo. The company that bought the technology wasn't able to turn it into a successful product. That wouldn't happen until later, much later, after Microsoft and Apple got their introduction to the mouse and such at Xerox's Palo Alto Research Center. In fact, Doug's ARC project was largely dismantled after the team was bought by Tymeshare. It seems that he had worked hard to open up the lab to researchers from other projects and universities -- almost everyone who was alive and working in the field at the time had, at one point or another, been to the ARC lab to see the software system at work in person. I can't help wonder if it as the collaboration and openness with the lab that led to some of the technological marvels that the group demoed that day in '68 to actually getting out into the world, in some form or another. Sure there were plenty of other insights and research that the team had done, but the reality is that annotations and bi-directional hyperlinks don't have mass adoption in the same way that the mouse and graphical user interfaces achieved.

How much of this idea leakage was due to the work that Doug did to make their projects available to others outside of their group? How much of it was a result of the same researchers ending up at Xerox's PARC which then let Steve Jobs and Bill Gates inside to see what they had built? It's hard to say, exactly.

[1] I wasn't able to find the original email, but Marc himself uses the feature to explain his investment in Rap Genius

#mother-of-all-demos #impressionism #conference-swag
27 Dec 2018 c.e.
Explaining Replace By Fee

I apologize in advance to those readers of mine that have zero interest in Bitcoin. I'm personally quite absorbed with the project, and am hoping that by writing about it incessantly, I might be able to convince you to at least appreciate the project for its vast complexity, if not for the riches it might make you, if only you invest at the right time.

I'd like to spend some time today writing out everything I know about a small corner piece of the Bitcoin puzzle, a transaction replacement protocol colloquially termed "Replace By Fee", or RBF for short.

A short description of the problem space

In order for a Bitcoin transaction to be considered valid, you must first have it included in a block by a miner. Normally, the way that would happen is as follows:

  1. You compose and sign a valid Bitcoin transaction. I'm leaving the details out here, but think of it like a HTTP packet that is ready to be sent out across the network, if that's helpful.
  2. You broadcast your transaction out from your wallet, onto the Bitcoin network.
  3. Other Bitcoin nodes on the network see your transaction and add it to their 'mempool'. This is the set of all Bitcoin transactions that have not yet been included in a block. They are candidates for inclusion.
  4. A miner receives your block. The miner finds a winning hash that makes its block a block. Your block is now mined.
  5. The newly mined block is transmit from the miner's computer to all the other computers on the Bitcoin network.
  6. Upon receiving this block, the Bitcoin node evicts all of the now-mined transactions from its mempool.
  7. Rejoice. Your Bitcoin is Spent!

You may remember that the topic we're discussing today is known as 'Replace By Fee'. When, you might ask, in this sequence of events might you want to replace your Bitcoin transaction?

The answer is sometime between steps 3 and 4 above. After you've broadcast your transaction, there is a chance that it will be seen and mined by a miner. Once your transaction has been mined, you can no longer broadcast a new version of that transaction, as the inputs to it have now been marked as spent.

There are a few cases, however, where your transaction might get trapped or evicted from the mempool without being included in a block. One common case for when this might happen is when the number of transactions that are looking to be included in a block (ie the mempool size) is larger than the available blocksize. In this case, transactions tend to be processed or mined based on the feerate per kilobyte that they offer to pay the miner for their inclusion.

If you've broadcast a transaction with a low feerate, and suddenly the mempool fills up with a lot of transactions that are looking to be included in a block, you may want to update your transaction to provide a higher feerate, so that your transaction will be confirmed in the next available block.

There's currently two mechanisms that people use to try to get their transaction included. The first is what we'll be talking about more in depth here, Replace By Fee. The basic gist of Replace by Fee is that you're rebroadcasting a previously broadcast transaction, but with a greater fee paid than the prior transaction.

The other strategy that wallets use to get transactions included in full blocks is called Child Pays For Parent, or CPFP for short. It involves issuing a new transaction, one that spends the earlier, still unconfirmed transaction. This second, child transaction will pay a larger feerate than it might on its own, with the hope that the now pair of transactions' total feerate will be high enough to merit inclusion in the next block. CPFP only works if the transaction you broadcast has an output that you can spend.

RBF: The Existing Algorithm

Replacing By Fee replaces the earlier transaction that you broadcast in other node's mempools. That's where the replacing happens. There is a set of rules governing whether or not a transaction is eligible for being evicted from the mempool and replaced by a new one. Here's a few things that the 'accept into mempool' code checks...

  • The transaction that you're attempting to replace has flagged itself as eligible for replacement. This is flagged at a transaction level, but is retroactive for any as yet unmined inputs that you're spending. If any of a transaction's inputs, or it's input's inputs, are flagged as replaceable, then this current transaction is also considered eligible for replacement. If a transaction or any of its parent inputs are not marked as replaceable, any transaction with an input conflict (that is they'd be spending the same inputs) is rejected with the error "txn-mempool-conflict".

  • Requires that all inputs already exist in the UTXO set. No currently unmined inputs are allowed in a replacement transaction. This is a tighter rule than the desired one, which is to check that the replacement doesn't require 'low fee junk' to be mined first. You can avoid this by rejecting any replacements that aren't using already mined inputs.

  • A replacement candidate must pay more in fees than all the transactions it replaces. The rationale for this is that sending transactions across the network consumes bandwidth. The higher feereate of the new transaction, in theory, pays for its increased usage of bandwidth: once for the original broadcast and then again for every subsequent replacement. Note that the nodes keeping and broadcasting this transaction don't get paid -- only miners do. In that sense the fee is more of a social justice than a net payment to every node that sees the transaction.

Note that this is in total fees, not fee rate. Any replacement transaction must pay more in total fees than the entirety of any and all transactions that the replacement would displace from the mempool. There's the potential that you'll be replacing an entire "package" of umined transactions, a parent-child chain of transactions that are looking to be mined. If you're a small transaction and you're trying to replace another who has an extremely large sized child also in the mempool, your effective fee rate (roughly calculated as the fee paid per byte of transaction that is included in the block) will be much higher than the original as you need to cover a larger amount of fees with a smaller number of bytes.

  • Finally, if the 'package' of transactions that you're looking to replace numbers greater than 100, your transaction replacement won't be added to the mempool. In other words, if someone has attached 99 transactions onto the transaction you'd like to RBF, you're shit out of luck. You'll have to wait until there's enough room in a block for your original to be mined.

Proposed Changes

Russell O'Connor published a proposal to change how the RBF rules work, at least two of them. The proposal would update the total fees rule. Instead of a replacement needing to beat the absolute fee amount of all transactions that it would be replacing (aka the "package" of transactions), it'd only need to be beat the effective feerate of the original. Additionally, the proposal would amend the 4th rule, such that the fee on your replacement is at least as much as the minrelayfee on the total package you're looking to displace from the mempool.[1]

Why is minrelayfee used as a minimum? A transaction that's replacing a larger set of transactions removes already transmitted bytes from the mempool. This rule change makes sure that the replacement transaction 'pays' for the cost of relaying those removed bytes.

Ok this is all pretty tedious. Let's take a look at some examples.

Miner Incentives, A Consideration

There's two cases that we should consider: a larger transaction wants to replace a smaller transaction (small txn -> larger txn) and that of a smaller transaction replacing a larger set of transactions, or package (large package -> small txn).

Current Rules

small txn -> large txn: Rule 3 stipulates that the total fees must be greater, with no regard to fee rate. In practice, no replacement is accepted if it lowers the total feerate of the mempool. (source). In practice, this shouldn't happen anyway. The motivation for RBF'ing a transaction is that the block inclusion feerate cutoff has spiked -- replacing one transaction with another larger one with a lower fee rate makes it less, not more, likely that your transaction will get mined in the next block.

large package -> small txn: The smaller transaction must pay more total fees than the existing package. The miner doubly wins: they're making the fees of a large transaction in a smaller byte footprint.

Proposed Rules

small txn -> large txn: Miner's choice strictly improves. The fee rate per byte that they're including has increased and the net fee of the new, larger replacement transaction is greater. This is no change from the current scheme.

large package -> small txn: Miner's choice also improves. Although the total fee that they will make for mining the smaller replacement transaction is net-net smaller than the fees the entire large package would have earned them, given a competitive environment for blockspace (ostensibly why the RBF was triggered in the first place), the smaller transaction with the higher per byte fee rate is more likely to be mined than the larger, lower fee per byte package it's replacing. The incentives of the miner (highest fee per block byte) and the RBF'er (having the transaction confirmed for the lowest reasonable fee) align.

Wherein We Contemplate a Word Problem

Let's take a closer look at the large package -> small txn case, as that's clearly the one where the proposed rule change has the greates impact.

A 1ksipa size transaction with a 10ksipa sized child transaction is in the mempool. The current feerate on the block is 2 satoshis / sipa[2]. The total fees that these two transactions, or package, pay is 2ksat + 20ksats = 22ksats.

Under the current scheme, a replacement transaction of size 1ksipa would need to pay at least 23k satohis, a feerate of 23 satoshis / sipa. This is an 11.5x increase in feerate from the original package's rate of 2 satoshis / sipa.

Under the proposed scheme, a replacement transaction of size 1ksipa would need to pay 12k satoshi in fees in order to replace a set of transactions of size 11ksipa. The effective feerate on the replacement transaction is 12 satoshis / sipa, a 6x increase in feerate above the package it's replacing.

The proposed ruleset strictly improves the feerate of the mempool, while lowering the fee ceiling for replacing a large or weighty transaction.

Notus Commentarius

RBF mechanics closely resemble that of an auction, where the rules for replacement are actually the next price that the auctioneer will accept a bid at. The current rules set the floor for the next bid to be extortionately high if the number of bytes you're looking to replace is quite large. Russell's proposed rule change lowers the bid floor to a more reasonable metric.

One of the largest arguments against changing the replacement fee rules, as far as I can tell, hinges on the argument that without a fee hike, anyone could spam the network with RBF requests, creating mempool churn and eating up network bandwidth. I'd argue that any RBF mechanism leaves an opening for this style of DoS attack on a node. The difference between these two proposals is not the mechanism, but merely the floor cost for waging such an attack -- at some point your transaction will be mined and the fees you've offered up will be paid. Further, the only case where this attack would be truly expensive is in the case where they're looking to replace a large number of bytes in the mempool -- perhaps that truly is the most likely DoS attack vector, however.

Thanks for sticking with me! Hope you enjoyed learning more about how mempool transaction replacement works! I left a few things off, but the main gist of how RBF works is all here.

[1] Russell O'Connor's proposed RBF rule changes (source Bitcoin ML) vs BIP125, the current RBF rules.
[2] A sipa is a byte/weight measurement. For simplicity's sake you can consider a sipa to be a byte.

#rbf #bitcoin #explainers
26 Dec 2018 c.e.
Blockchains Against Evil, Impressions

Takeaways from a blockchain ethics conference I attended earlier this month, Blockchains Against Evil

I attended a day-long conference/seminar earlier this month, that pulled together a bunch of people in the 'blockchain' space to talk about trends in the industry, especially around security and lawlessness.

The Event, Specifics

The event itself was held in a rented conference space off Divisidero, in San Francisco. There were about 30 people in attendance, if I had to guess. Most everyone who attended worked or invested in the 'blockchain' space. There was a good mix of job types and roles: programmers, investors, company-runners, cypherpunks, non-profit directors, etc. I knew a few people from the Internet, but most were new faces.

The day was split up into a bunch of round-table talks. I honestly can't remember most of the themes. I took notes, but I've since misplaced the notebook. I'm planning to write up a longer piece on the insights the discussions gave me that specifically related to privacy and secrecy and how cryptography and the state interplay in this, but that piece is far more ambitious that I have the time or inclination to reason through now. Much like my lost notebook -- it'll be dug out later.

Themes and Thematics

Instead, I'll leave you with a short overview of the most salient points that were discussed. Most of these are a paraphrasing of other's points and ideas. I take credit for only the spotty transcription.

  • Crypto has provided a secure mechanism for ransomware makers to get paid. The global nature of the web plus Bitcoin's ubiquitous reach[1] mean that ransomware is truly a viable attack for anyone who's got access to a Bitcoin wallet. This is all of you. Another lens to put on this one is that it's put a premium on securing networks of valuable data. If your data being inaccessible makes your work impossible, it's likely only a matter of time until you're a target for a ransomware play.

  • While ransomware has placed a bounty on your databases, Bitcoin and other Proof of Work currencies have placed a directly calculable value on a computer's CPU cycles. Previous hacking rings have focused on skimming credit card numbers[2]; the past decade has seen more and more viruses that aim to steal compute power rather than credit cards or identities. That's because they can make money by stealing computation cycles and your power to mine crypto. I'd be curious to see stats on how the rise of ASICs has affected the profitability of botnet miners. Bonus points for an analysis that includes the impact of the recent price drop on said profitability.

  • Personal security is hard to measure. There've been several high profile cryptocurrency and 'blockchain' project attacks recently that involved getting a phone company to port a target's telephone number to a new SIM card, giving the attackers access to their SMS two-factor authentication backup codes. The general advice for avoiding this sort of problem is to ask your phone company not to port your number without being provided with a secondary PIN number or the like; others at the conference had switched to Project Fi, Google's phone service, for the express reason that they don't have a customer support telephone number. (Personally, I already use Project Fi). More generally speaking, there seemed to be generally an interest in hiring a hacker to do a personal security audit. If you or someone you know runs this kind of a service, let me know. I'd love to hear more about what kind of people you work with and what your price point is for an individual investigation.

  • Demand for decentralized services historically has been rather complex, if not a bit on the weak side. Often, they crop up as alternatives to more centralized services when a core user group is pushed off of the more centralized services (i.e. music and film piracy, right-wing punditry, and most recently sex work with SESTA/FOSTA[3]). As difficult as it is, it's pretty wild to imagine existing in a fully decentralized world, one where no one has the power to deplatform anyone else. It's hard to imagine a world where everyone runs their own decentralized server, a la the Urbit dream. Curation and searchability seem like they'd be particularly high value services in this kind of world. It definitely would be heading into 'pure free speech' territory, of the likes we only dream of currently but also remember folks that while speech may be free, slander is still illegal.

  • Personal anonymity. What right do you have to decide who and what can see where your money is going? I've got a lot of unfinished thoughts on this that I'm hoping to put up later in a separate piece. If and when I do, I'll update this to link to it.

  • Closely related to that, do anonymous payment networks breed demand for dark market goods? I'm talking about child pornography and buying hitmen for untraceable cash. I think the recent Epstein revelations[4] points towards no, vice isn't necessarily driven by access to invisible money. Honestly, if anything it's moving illicitness from the cash economy to the digital economy. Cash is largely untraceable. If you lose it in a fire, it's gone. In some ways, this is oddly similar to problems with keeping private keys and wallets safe for digital cash. But I digress. To what extent has a traceable money supply kept people exercising base desires that a lack of traceability now enables? Again, I think this is smaller than we suspect, but maybe I'm wrong. If anything, I think dark money and dark Internet (Tor) have made buying illegal drugs and child pornography much easier than they were in the past, but does ease of use drive volume? These things are still illegal. I'd love to read a study on the impact of digital darkness on illicit good trade, though I imagine hard numbers on this are hard to come by.

In Exitus

Digital money has created huge new opportunities for criminals and privacy lovers alike. I feel like the cat's largely out of the bag with the existence of digital money systems such as Bitcoin and Zcash (and Grin soon!). I'd love to see personal and institutional privacy and security become both more widely understood and practiced -- though at its core this problem involves an even greater investment into even basic computational understanding.

Will we, as a society, be able to educate ourselves fast enough to protect our systems and selves against the rising tide of spying nation states and exploitative hackers? I guess we'll find out.

I really enjoyed spending a day hearing about the in's and out's of blockchain ethics. I'm really grateful that there's people in SF who want to have these conversations, and went so far as to organize a space where we could discuss them. Huge <3 to all the organizers and other attendees that made the day incredibly worthwhile.

[1] By Bitcoin I really mean any value-acknowledged cryptocurrency.
[2] See the story of The Iceman
[3] A lot of this discussion hinged on the stuff John Backus has been digging up lately, I really like his article on Music Piracy
[4] The man basically ran a prostitution ring for wealthy and well-connected men, from a cadre of underage women that he developed. Miami Herald has the story.

#blockchains #conference-swag #impressionism
28 Nov 2018 c.e.
Getting AMPed Up or Reflections on Lightning post Adelaide

I've recently been thrust head first into my first open source software ecosystem. I love it; I also feel like I'm struggling to contribute anything worthwhile because I've been spending so much time just getting up to speed -- the particular subsystem of software that I've landed in is incredibly complex and has a bit of scattershot documentation, spread across a couple of mailing lists and two enormous projects.

I want to give some meta commentary on the mechanics of getting involved in a new, active space, and then give a more nuts and bolts overview of the considerations that are shaping the edge of Lightning at the moment. I'm sure I've left things out, so know that my list is just a subset of all the things.

Finding Active Edges

There's a difference between getting up to speed and active in a currently evolving field versus learning a topic or subfield that's pretty much static. By way of example, I'd largely consider calculus and functional programming, as fields, to be pretty static, i.e. there's interesting stuff happening at the margins, probably, but there's not a lot of paradigm shifting research going into how to describe functionalism or what a second derivative is. As a field and practice, the borders of meaning and scope have largely been well defined.

'Active' spaces are different. They have action, or people actively working on new approaches or building out software and new ideas. The presence of people and the messiness of definition and conversation are beacons to what the interesting and new things the future will hold.

Arriving at an edge or beehive of activity where there are people working is like descending into a bit of chaos. In an active field, there's usually a lot of independent research and motivations and interests that keep the actors on this edge a bit spread out. Figuring out where the edges lie is difficult because the definition of the edge is its lack of a roadmap. Sometimes you can find artifacts that strictly define at least a subset of those edges -- the wiki tracking decisions made at the Lightning Summit in Adelaide two weeks ago is one such example.

I was lucky with Lightning, in a lot of ways. The biggest one is that due to the team I joined, I have a lot of direct access to people that have been working on the edge of the space basically since the beginning (h/t to cdecker). The other is that I joined just in time to attend the latest spec update meeting. These meetings are rare -- the last one happened over two years ago in Milan for the first lightning spec.

I'm not going to talk directly about what happened at the meeting; if you're interested check out the lightning mailing list, where we're currently in the process of hashing out the decisions made at the summit (which you can see here), or take a look at the PRs currently in progress on the lighting-rfc Github project.

Rather, I'd like to give some really meta impressions of what kind of thinking it takes to get involved in a project like Lightning -- hopefully this metaness will give you a portrait of what kind of conversations you need to be having or questions you should be looking to get answered when getting involved in a new field.

First off, it's hard to contribute to a field if you don't really understand the underlying system that it's operating on top of. Sure, this is easy enough to say, but just figuring out the contours of the system that define the problem space can be tricky. A lot of the stickiest problems that Lightning developers deal with, especially when looking to expand the protocol or improve the experience, are either limitations in the underlying Bitcoin protocol or a self-imposed mandate for privacy. If you don't have a good grasp on the goals of Lightning with regard to privacy (keep it, as much as possible), or a pretty deep knowledge of how Bitcoin itself works, you're not going to be able to contribute much to the conversation around Lightning -- mainly because you're going to struggle to even understand, let alone communicate with, people who are already working in the space.

I'm an incredibly quick study, but still relatively new to the Bitcoin and Lightning space. My largest contributions to date can mostly be summed up as asking clarifying questions. This may seem trivial, but I've come to see that it's an important contribution nonetheless -- comprehensibility is an incredibly important aspect of a system that needs and wants newcomers to both feel welcome to the space and able to contribute. And Lightning definitely could be more comprehensible!

Into the Deep

With an eye to making the Lightning space a bit less opaque, I'd like to run through a few of the higher level considerations that seemed to come up with some frequency during the weeks leading up to and at the summit itself. I think it's safe to say that these themes will be continuing problems and on-going discussions in the Lightning ecosystem.


Bitcoin protocol limitations come in a variety of flavors. Here's a quick, condensed (and definitely contains omissions) rundown of things in Bitcoin that hold up or complicate Lightning feature development:

  • Fees. Lightning is a 'second layer' protocol, sure, but at some point it has to publish transactions on the Bitcoin blockchain. Lightning's security mechanisms (ie your ability to successfully pull your money out of a channel) rely on the ability to get a transaction into a block within a reasonable amount of time. Lately, this hasn't been a problem, but if and when fees spike, there's a lot of potential to run into trouble if your transactions aren't able to get confirmed. Fees are complicated by the fact that 1) there's two parties involved in creating and spending all the transactions, 2) commitment transactions are usually composed, signed and stored long before you might actually need them, 3) economic incentives mean that you're probably looking to pay the smallest fee possible to accomplish what you want, but this means that you're probably in a bad position in terms of being able to get your transactions on chain in a fee spike event. Lightning as a protocol would like to move away from the business of needing to know what the fees should be, but that means we're going to run into another corner case of the Bitcoin transaction ecosystem...

  • RBF and CPFP. If you're not deep in the Bitcoin wallet management weeds, there's a good chance you've never heard these acronyms before. Briefly speaking, these are two mechanisms that the Bitcoin protocol provides for getting a transaction through that has largely been pushed to the back of the queue for being included in a block (mines/confirmed etc) because of a fee spike. RBF stands for Replace By Fee, whereby you basically re-issue a new copy of a transaction, but one with more fees per sipa[1]. CPFP means Child Pays For Parent. It takes advantage of the chained nature of Bitcoin transactions, and attempts to 'sweeten the deal' for miners such that they'll mine your first, low fee transaction in order to also be able to mine a high fee child transaction. The parent plus child chain is typically termed a 'package'.

  • Schnorr. What is Schnorr? Schnorr is a proposed change to multiparty signature composition. Including it in Bitcoin will require a revision of the signature verification mechanisms.In addition to more compact and easier to verify signatures, Schnorr unlocks a certain amount of obfuscation and script burying. Schnorr can make Lightning channel openings invisible on chain (right now they're a bit easy to spot[2]). There's a few other nice things that Schnorr signatures enable, that I don't exactly remember the details of, but they'll Lightning to send payments in parts more easily and securely[3]
  • Script Sighash Flags. Christian Decker's been spearheading an effort to update the way that Lightning balances are enforceable on chain. (The updated protocol is called Eltoo, you can read more about it in this high level article I wrote, or the paper itself, if you want something a bit more in depth.) This requires a change to Bitcoin script, specifically the addition of a new sighash flag called SIGHASH_NOINPUT[4][5]. Work on the new, improved state management protocol is basically stalled until this gets merged into the Bitcoin reference implementation. On another note, there's some other boutique, existing sighash flags that will probably start being utilized by Lightning transactions as part of the attempts to dodge the fee problem. Watch this space.
  • Transaction malleability. This is an ancient problem now in Lightning land, as it was resolved when SegWit landed. If you're going to be doing Lightning, you should know how SegWit works, as that's the only type of transaction protocol that Lightning wallets speak. As a historical note, transaction malleability basically refers to how fixed the transaction hash is. Lightning, in its current form, requires the guarantee that the hash of a signed transaction can't be changed (by a miner or the other party etc). SegWit fixed this -- it's practically never mentioned now. In other words, this problem has moved off the edge, largely because it's settled.


This feels like one that's taken for granted more than most things, but it largely informs a lot of architectural decisions that get made. Maintaining privacy is important, and it manifests itself in a bunch of ways. Here's a short list of things that privacy considerations impact.

  • Error handling. How do you know who bungled your payment?
  • Payment correlation / decorrelation. Can an observer figure out if payments being sent over different channels or the same payment over different time periods, routes, are the same?
  • Getting a clear picture of current network health. It's hard to a payment success rate if the payments themselves are localized and unreadable
  • Autopilots. How much information should nodes reveal, to help other nodes figure out who to connect to?
  • Anything that might leak private or proprietary information including but not limited to: channel balances, node wallet UTXOs, payment origination, payment destination

Other assorted things

  • Liveness. Payments can get stuck if nodes along the route aren't responding. This is particularly bad if a payment has to 'go to chain', ie be finalized via the blockchain.
  • Liquidity. Lightning payment capacity is a constantly mutating DAG. Channels' total value is known, but the balance of funds within that channel is often kept secret (see Privacy, above). This makes it hard to predict which routes will fail until you try it -- the advertised channel capacity may be pointing in the wrong direction. This is exacerbated by the fact that channel funding is one-sided at the moment. Splicing and dual-funding will help this problem.
  • How important are receipts? This deserves a much longer post and honestly I need to do more research around it; I won't get into it here.

In Exitus

I'm having a great time.

[1] A sipa is another term for a kiloweight, which is a Bitcoinic way of weighting bytes in a transaction to calculate the fee rate of a transaction. As a general rule, miners prefer transactions with the highest fee rate per byte. If a fee rate spike is happening, you're going to want to up your transaction's effective rate.
[2] As an aside, we green lighted work on a different signature scheme (some 2 party single ECDSA sig algorithm) that can let private channels remain invisible on chain. Nice because it doesn't rely on Schnorr.
[3] There's been a lot of discussion around AMP (base AMP, OG AMP, low versus high AMP). This deserves a longer discussion, but know that Schnorr sigs will provide a way to do split-payments with fewer drawbacks than any of the current proposals. In fact the coming of Schnorr is a background vibe underpinning a lot of the discussion, as it makes the timeline question more important.
[4] I believe the final name is settling somewhere near SIGHASH_NOINPUT_UNSAFE for #reasons.
[5] What's a sighash flag you ask? Briefly, it's a bit that's added to a transaction signature that tells the verifier what fields in the transaction that the signature signed. You can read more about them here.

#lightning #bitcoin #oss #edges
22 Nov 2018 c.e.
A Brief Love Letter to XOR

I'm taking an online crypto class[1] right now, and it's been forcing me to get more intimate with the bitwise operator XOR. On top of being incredibly lightweight, there's a few really cool things that XOR can do.

In the spirit of the Thanksgiving season, here's a brief love letter to my favorite little boolean operator, XOR.

What is XOR?

XOR stands for 'eXclusive OR', where 'or' refers to the boolean logic operation. What does that mean, a boolean logic operation? Briefly, it's what conclusion you draw from two truth values. It's kind of like a predetermined agreement mechanism. Boolean logic is a rule that you apply to two results, to resolve those two results to a single true or false.

A simple example is probably helpful. Let's say that we've got two voters, and we're trying to take their two votes (either YES or NO) and return a single decision for the 'election'. How these two imaginary voter's votes are counted is the role of the Boolean logic operator.

There's two, fairly common boolean operations that you might have heard of before: and & or. The decision for 'and' is fairly intuitive: if both voters vote YES, then the result is YES. Otherwise, the result is NO. We'll only get a final YES vote if both of the people we're asking say YES. If either voter votes NO, the final result from the boolean operation will be NO. The 'and' decision framework requires 100% agreement.

'Or', on the other hand, says that if either voter says YES, then we'll take the result to be YES. The 'or' decision framework requires only one single 'voter' to say YES in order to return a YES.

So what is XOR? XOR only returns true if the voters disagree. If both voters say YES, 'xor' will return NO. Same thing if both voters say NO: 'xor' will still return a NO. It's only when one 'voter' has chosen YES and the other NO that XOR resolves to a YES. It doesn't matter which voter says YES and which one NO, as long as the voters disagree XOR returns YES.

Why is it called exclusive or? Great question. I have no idea, but you can probably find out on the Internet.

XOR As Your Encryption Friend

XOR does some pretty fancy things. If you take a series of bits and XOR it together with another series of bits, the original series of bits can be retrieved out of the resulting string, but only if you know what the second series of bits was. It's almost impossible to tell what the original bit series was. Here's a quick example, to show you what I mean.

// If I take the bit series 0101 and XOR it with 1010  
0101 xor 1010 =  1111

A result of 1111 doesn't tell you what bits belong in which of the strings that you xor'd together. You could have xor'd 1111 with 0000. Or 1100 with 0011. But! If you do happen to know one of inputs, you can easily extract the other.

// If I know 1010 and the result, 1111, I can extract the other input   
1010 xor 1111 = 0101

This is incredibly useful in cryptography. If you take a message and XOR it with a 'secret key' (a random series of bits) as the same size as your message, viola, your message is now encrypted. If your 'secret key' is a random enough series of bits, then it will be practically impossible for anyone to know what the original message bits were. To decrypt this message, all you need is the encrypted message and the key that was used to encrypt it.[2]

// How to encrypt a message   
message xor key = encrypted_message  

// How to decrypt a message  
encrypted_message xor key = message

Other XOR Magic

XOR has a little bit of 'magic' that happens when you use either a set of all 0's or all 1's to XOR against.

You can 'bitflip' any series of bits by XOR'ing it with a series of 1's.

// Flip a bit set!
111000 xor 111111 = 000111

XOR'ing by a set of 0's is an 'identity fucntion' -- it'll return the same series of bits as what you originally XOR'd in. It's probably not a good idea to use a set of 0's as your encryption key -- it'd be like putting your message behind a piece of glass. XOR'ing by 0 is transparent!

// Show me the same!
111000 xor 000000 = 111000

In Exitus

The next time you use encryption to send a message with a friend over the Internet, give a little thanks for your crypto workhorse bestie, XOR.

[1] Dan Boneh's Crypto I on Coursera
[2] This method of encryption is generally called the One Time Pad encryption, as the key is as long as the message. So long as you never reuse the same key on a different message and your key is a random stream of bits, this method of encryption (xor'ing the message with a key) has what's known as perfect secrecy. The biggest, practical problem with this method of encryption is that the person decrypting your message needs to know the key. You'd need a secure way to send them the key, as anyone who gets the key can then decrypt the message. The key is as long as the message though! If you have access to a secure method of communication that can transmit something as long as the message, you should just send the message itself over that secure communication channel. It's just as long, and your chatting partner won't have to decrypt it. This equal length key problem is why they say that perfect secrecy is practically impractical.

#xor #boolean #love-letter