• a.bout
  • t.witter
19 Jan 2020 c.e.
The Low Down on Liquidv1, Blockstream's No.1 Sidechain

Let's talk about Liquid, Blockstream's semi-proprietary sidechain[1], lightning and how they interop with bitcoin. If you're anything like I was a year and a half ago, you may be fairly comfortable with bitcoin at a high level, but not quite understand where and how Liquid and Lightning fit inside of that. This brief article is for you, past self.

From a super high level, Liquid and Lightning are software projects that operate on top of pooled bitcoin.

Both of these projects have very similar underlying principles, but different ways for accounting for the money in the 'bitcoin pools' that they create.

I consider lightning to be the simpler case of this, since lightning 'channels' only concern two parties. The basic idea is that two people who hold bitcoin decide to put that bitcoin into a contract, or bitcoin transaction, such that a shared pool is created. The money in the shared pool can only be spent if both parties agree to spend it. A parties' agreement is demonstrated by signing a transaction that spends the jointly pooled funds. The transaction that creates the 'pool'[2] is published to the bitcoin blockchain. Any transaction that spends from that shared or pooled transaction will require both parties' signatures. If you're a party to this contract, you don't sign any transaction spending the pooled money unless you agree to how much that transaction pays you. Ideally, you've been keeping track of how much of the money in the shared pool belongs to you, so you can check that when someone asks you to sign a transaction that pays out from the pool that you're getting the appropriate share.

Lightning Fast Lunchbox Poker

Ok, so lightning is two people with some bitcoin who want to pool it together. What do these people do with this pool? Well, they basically create a 'napkin ledger' that records how much of that pool belongs to each person. Let's use a card game with a buy-in as an example. At a casino, you give your money to a cashier and they hand you a bunch of chips. You then take those chips to a poker table, and play poker. At the end of each round of play, the chips get redistributed to each party at the table depending on how they did that round.

Lightning is like a round of poker between two friends, but instead of chips you put the money into a lunch box that your friend Joe keeps track of. Joe will only distribute the money from the lunchbox when he gets a napkin with both of your signatures on it, saying who gets paid what amount from the money in the box. So the two of you play a very boring two player game of Texas Hold'em. At the end of each round, you get a new napkin, write down the updated balance, sign it, and then tear up the last napkin (so that neither of you can take an old napkin to Joe and get more than is currently fair).

Lightning channels work like this, except instead of Joe's lunchbox we use the bitcoin blockchain, and instead of napkins we use unpublished bitcoin transactions, and instead of tearing the napkins up there's something called a revocation token that makes it such that if anyone publishes an 'old napkin', the other party can go and take all of the lunchbox money for themselves, as penalty for you cheating.

Welcome to the Liquid Casino

Liquid is more like the casino where there are chips and instead of there being Joe with a lunchbox, there's a whole office with a cashiers window and multiple tellers that will change your hard money into red and green and black chips. In this case the cashiers are the members of the Liquid federation. They run special hardware boxes with special keys that make them part of a hardcoded signature set. Anyone can pay money into liquid, but you have to put it into a contract where the only people that can spend it again is by going to a cashier's window to cash out[3].

Instead of using napkins to keep track of who owns what and what their balance is, in this casino of pooled money, Liquid uses its own blockchain. Every movement of chips between players at the casino[4] is recorded in a transaction on this secondary chain. Just like at a casino, none of the chip movements really matter to the larger pool of cash outside of the Liquid casino, except that everyone in the casino knows that you can go to a cashier and get money for them (bitcoins in this case), so it really is as good as cash. Unless you try to buy a sandwich with them, in which case, well good luck.[5]

Since Liquid is a blockchain, and it's not bitcoin there's a few differences in how exactly the chain gets built or made. In bitcoin, you need a bunch of people running proof of work algorithms to figure out what the next block will be. In Liquid, since there's only a few cashiers at the windows, each of the cashiers takes a turn deciding what the new block will be. If you're a cashier and it's your turn to make a block, you get to pick what transactions go into it. You just bundle them up and then sign it, and send it out to the other cashiers who look at who's turn it is to make a block, check that the signature belongs to that person and that it builds upon the last block in the chain, and add it to their copy of the liquid chain. Then the next cashier does the same, etc. Blocks in liquid are issued every minute; this is a convention rather than a probability function like it is for the bitcoin chain.

In Liquid vocabulary, "putting money into the casino and getting chips" is known as "pegging-in". Taking your casino chips to a cashier and getting paid bitcoin back is known as "pegging out". When you understand that the liquid money pool is the set of transactions in the bitcoin blockchain that have all paid to a smart contract that can only be unlocked with a certain threshold of the cashier's (federation member's) signatures, suddenly it makes sense why anyone can peg-in but you have to talk to a cashier to peg out.

(In case it's not obvious, you peg in by creating a bitcoin transaction to a special address. Anyone who owns bitcoin can do this, thus anyone can peg in to liquid.)

Other Liquid Features

Liquid also does other fancy things like confidential transactions and allows you to issue new assets[6]. Assets on Liquid aren't pegged-in in the same way that bitcoin is, so who has them and how you get more of them and how many there are of them might just be a Liquid network thing, but it also might be a bigger wider thing, like the Tethers on Liquid that are part of a larger ecosystem. I'm not really going to talk about these here because they're not really part of the pooled bitcoin stuff. If you want to learn more about how liquid actually works, I'd recommend Blockstream's technical overview. If you want to run a liquid node, the Elements project tutorials are where you'll want to end up. The open-source software that can run a liquid sidechain is called "Elements" for reasons[7]. In theory, anyone can run their own federated chain using the Elements project, Liquid is just one of the existing options. I'd recommend this simple config file for an elements node if you decide to run on the liquid chain.

In Exitus

That's it bitches. We've pretty much covered lightning and liquid's relationships to the bitcoin blockchain. Seem simple? This is one of those things that I'd call very 'simplex', because it's simply complex.

[1] Disclaimer: Blockstream pays me salary.
[2] The way you do this is simple: you create a 2 of 2 multisig outputs. The only way to spend a 2 of 2 output is by first getting both party's signatures.
[3] There are plans to make it so you anyone can cash out at will, but right now you have to go through a cashier. This is for security purposes, and makes sense when you think about it.
[4] I imagine, if there were a Liquid Casino, their tagline would be "let's get liquid!" but who knows. Hard to say, really.
[5] BTCPay server just added Liquid to its set of things that it will accept, so the casino is expanding quite rapidly. Reddit discussion
[6] Watch this space for BitchCoin. #BitchCoin2020
[7] I seem to remember someone telling me it's because the Elements Project is a collection of building blocks, or elements, for a coin ecosystem. This is unverified.

#liquid #blockchains #bitcoin #sidechains
16 Jan 2020 c.e.
How to Be Perfect

Git is a collection of commits, a story about a history of changes that you made to a codebase. I've written before about the importance of this history, but I realized I didn't really put much time or effort into talking about how to manipulate your history. If you can re-write history, why not re-write it to make you look perfect?

Being perfect is expensive. It's not easy. Going back and attempting to make yourself look perfect after the fact is even more tedious and difficult. My understanding is that most people get into programming because they love automating rote things; what I'm about to show you is the opposite of automated -- it's a bit of painstaking work.

Why would you do this tedious thing? Because it makes you look fucking perfect.

Before we get too far into this, I should offer up a small caveat that editing your git history is pretty dangerous/risky. The absolute best thing to do before trying a rebase like this is to make a new branch that you can attempt to rebase from. That way, if something goes absolutely haywire, you'll have the original branch hanging around still, and you can start over from there, if you want. Or call it quits like the imperfect human that you are and roll forward from there.

Another final note, I consider myself a pretty basic git user as the number of commands that I use on the regular are fairly small, but as you'll see shortly, this technique does require a certain level of understanding of how git's model of commit history and rebasing work. If you don't really grok how git plays and then replays commits during a rebase, maybe reading through my weird way of mangling history will make it clearer. One can hope, at any rate.

Ok, so preliminaries out of the way, what am I even talking about by making a git commit history look perfect? Well, if you're anything like me, you might have a bad habit of checking in slightly broken code, discovering how it's broken later, and then writing a small update to fix the brokenness. One good example of this in c-lightning is allocating an object off of the wrong thing -- I don't find out about it until I've gotten some good test coverage with valgrind running.

Fixing Commits, The Easy Way

The easiest way to fix an imperfect commit is immediately after you've committed the imperfect code; when the commit you want to correct is the first commit in your history.

You can easily update the last commit using the --amend flag. Basically, how this works is that you update the code and stage the changes, as you would normally. When you go to commit it, you merely add the amend flag. This applies the staged changes to the last commit.

$ git commit --amend

It will re-open the text editor with the commit notes in it, so you can make any edits you'd like before saving and closing. Once you close the commit message document, git will amend the last commit to include the new changes.

Fixing up Commits, The Hard Way

Ok, let's say that you've made some commits, realized those commits had errors much much later, made some fixes, checked those fixes in and now are at the end of your project and want to go back make everything look incredibly neat.

Who does this? Me, I do this. It's fixable, but it's also painful. I'll walk you through an example of me fixing up some commits I made to the dual-funding branch on c-lightning, before pushing it up for review. In this particular case, I've committed two fixups into the same commit, however the original errors were made in two separate commits. I need to go back and split the fixup commit into two, before I can then squash each of the fixup up commits into their respective originating commit.

Note that just by giving you this information ("the fixup commit's errors belong in two separate commits"), I'm cheating a bit. Normally, you'd have to go back and figure out that this is what happened yourself. We'll still walk through how to find the originating line change, however, which is basically the same process.

Ok. Let's make some perfectionism happen!

Example of an over-eager fixup

Alright, here's the fixup commit. I've pulled it up by running git show 8280a3a.

commit 8280a3a54efa1dcbb767f5f499e2ad86fde460d3
Author: lisa neigut <niftynei@gmail.com>
Date:   Mon Dec 2 13:13:03 2019 -0600

fixup: pass correct `am_funder` value

diff --git a/openingd/openingd.c b/openingd/openingd.c
index 5b37dd7d4..d848cbac6 100644
--- a/openingd/openingd.c
+++ b/openingd/openingd.c
@@ -1170,9 +1170,9 @@ static bool send_receive_funding_tx_info(struct state *state,
        u16 num_ins, num_outs;
        int type;

-       msg = opening_negotiate_msg(tmpctx, state, false);
+       msg = opening_negotiate_msg(tmpctx, state, role == OPENER);
    if (!msg)
-           return NULL;
+           return false;

        type = fromwire_peektype(msg);
        switch (type) {
@@ -1434,7 +1434,7 @@ static u8 *funder_finalize_channel_setup2(struct state *state,
                       &our_sig.s, NULL);

    sync_crypto_write(state->pps, take(msg));
-   msg = opening_negotiate_msg(tmpctx, state, false);
+   msg = opening_negotiate_msg(tmpctx, state, true);
    if (!msg)
        return NULL;

Notice that this commit contains two separate line changes: one at line 1170, the other at line 1434.

It's easy to see why these were checked in together -- they're fixing a problem of the same class, namely the wrong value being passed in to opening_negotiate_msg's final parameter.

In most cases, as a one-off commit, this is a perfectly acceptable way to do things. But we want to make it look like we never even messed this up in the first place. We're erasing our mistakes. We can do this because this is still in a branch -- the commit where the original mistake took place hasn't been committed to master yet.

This is distinction is import. We're going to re-write history using rebase to make it look like this little whoopsie never happened. You can't (shouldn't rather) rewrite the history that's on master, but branches that you're about to push up as a PR are fair game.

To do this, we need to first find the original commit where the line we're changing got added to the codebase. Then we'll use rebase's fixup to merge this commit in with the original, no one will be the wiser to our original mistake. This will reduce your commit count, but that also reduces the number of commits that a reviewer needs to look at and makes the change more 'atomic'. Atomic commits are easier to revert or port elsewhere if need be, and make your history easier to understand because every commit has decent context for the change.

So I already cheated and told you the answer to this, but we need to confirm that they both originated in the same commit. If not, we'll need to break them into two smaller commits, and then fixup those patches individually into their respective originating commit. Spoiler: these definitely belong to different commits.

Let's start by finding the commit where the first line was added, the one that we modified. We need to find the originating commit for the line we're changing. Usually, I use git blame to look for clues as to where it might have been originally added. This can be kind of tricky, and is probably the hardest part of this whole endeavor. If you already know what commit the change originated in, you can skip this part and go down to where we split it up into parts.

Looking just at the first change in the commit, you'll notice that it's in a method called send_receive_funding_tx_info. I'm going to search in git blame for this method.

@@ -1170,9 +1170,9 @@ static bool send_receive_funding_tx_info(struct state *state,
        u16 num_ins, num_outs;
        int type;

-       msg = opening_negotiate_msg(tmpctx, state, false);
+       msg = opening_negotiate_msg(tmpctx, state, role == OPENER);
        if (!msg)
-           return NULL;
+           return false;

Now we run git blame on the file that the change is contained within. I've used vim's find utlity to locate the method.

$ git blame openingd/openingd.c

7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1091)
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1092) static bool send_receive_funding_tx_info(struct state *state,
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1093)                                        enum role role,
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1094)                                        struct input_info ***local_inputs,
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1095)                                        struct output_info ***local_outputs,
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1096)                                        struct input_info ***remote_inputs,
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1097)                                        struct output_info ***remote_outputs)
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1098)
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1099) {
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1100)       struct channel_id id_in;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1101)       bool complete;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1102)       size_t i;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1103)       u8 *msg;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1104)
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1105)       /*
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1106)        *  BOLT-3f9f65d3ad5a21835994f9d9226ed9e0e4066662 #2
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1107)        *
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1108)        * - if is the `opener`:
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1109)        *   - MUST send at least one `funding_add_input` message
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1110)        *   ...
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1111)        * - if is the `accepter`:
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1112)        *   - MAY omit this message
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1113)        */
...

This is a long method, so I'll then search for opening_negotiate_msg, which leads me to this code block.

7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1168)               struct input_info **inputs;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1169)               struct output_info **outputs;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1170)               u16 num_ins, num_outs;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1171)               int type;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1172)
12953b897b openingd/openingd.c          (lisa neigut      2020-01-16 20:03:39 -0600 1173)               msg = opening_negotiate_msg(tmpctx, state, role == OPENER);
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1174)               if (!msg)
12953b897b openingd/openingd.c          (lisa neigut      2020-01-16 20:03:39 -0600 1175)                       return false;
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1176)
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1177)               type = fromwire_peektype(msg);
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1178)               switch (type) {
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1179)                       case WIRE_FUNDING_ADD_INPUT:
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1180)                               if (!fromwire_funding_add_input(state, msg, &id_in, &inputs))
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1181)                                       peer_failed(state->pps, &state->channel_id,
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1182)                                                   "Parsing received funding_add_input %s",
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1183)                                                   tal_hex(msg, msg));
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1184)                               check_channel_id(state, &id_in, &state->channel_id);
7cd71d3128 openingd/openingd.c          (lisa neigut      2019-11-21 17:48:36 -0600 1185)

In git blame, the first column is the commit hash where that line was last changed. You'll notice that most of these lines were introduced/touched in commit 7cd71d3128, but two of them have been updated in 12953b897b. 12953b897b is the commit we want to fold into the other one. I'm going to guess that the commit we want to fold into is 7cd71d3128. I'll confirm this by running git show 7cd71d3128, and looking for the originating lines.

$ git show 7cd71d3128
commit 7cd71d31281fc23ba7d63d057c631c95591ee4d3
Author: lisa neigut <niftynei@gmail.com>
Date:   Thu Nov 21 17:48:36 2019 -0600

    df: port to new input/output dance

    updates to use the newer funding_add messages.

diff --git a/channeld/channeld.c b/channeld/channeld.c
index 172660c6a..b0085ebc2 100644
--- a/channeld/channeld.c
...

For the updated commits, we'll need the first commit message for this, df: port to new input/output dance. I'm not going to walk through it, but I'll do the same for the other commit now. (It originates in a commit who's tagline is ``).

Once I know where these changes belong and that they're not in the same commit, we'll split the single fixup into two commits, which we can then apply to the two places where the problem originated.

We're going to want to edit this commit in the rebase history. Typically I do a rebase -i, mark the offending commit as an edit.

$ git rebase -i 8adbd4d31eac37449d40a6a3ee207d4a4de3e188

This opens up a list of commits from HEAD to the passed in commit hash. My git editor is set as vim, so it'll open in vim. This a worksheet where you can interactively edit your commit history. Be careful here -- any lines that you delete will be lost from your branch, and will be pretty hard to get back if you lose/don't remember the commit hash.

Here's the relevant section of my rebase list.

pick 8bfcd941e fixup: missing bracket
pick 3be3df51d df: convert accepter_sigs to new pattern, add input check
pick 8280a3a54 fixup: pass correct `am_funder` value
pick f5d7f1d0b fixup! send_receive_funding_tx_info, use pointer
pick ba715505e df: save incoming peer messages until we return
pick 3dfeafa9d df: fixup, need to pass a not null in to to/from wire
pick 1001fa1da Revert "lightningd/bitcoind: remove unused 'get_output' function"
pick 29fb84ec0 bitcoind: Add typesafe version of gettxout
pick e665d91f0 dual-funding: confirm scriptpubkey matches what's onchain
pick 9ba12df8f fixup! patch up feature addition of v2 fundchannel

The commit we want to update is the the third one, fixup: pass correct 'am_funder' value. The bottom of the interactive rebase screen lists our options for how to change up this commit.

# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit

We'll use fixup later. For now, we want to stop to edit this commit. All we need to do is update pick to edit. Like this.

pick 8bfcd941e fixup: missing bracket
pick 3be3df51d df: convert accepter_sigs to new pattern, add input check
edit 8280a3a54 fixup: pass correct `am_funder` value
pick f5d7f1d0b fixup! send_receive_funding_tx_info, use pointer
pick ba715505e df: save incoming peer messages until we return
pick 3dfeafa9d df: fixup, need to pass a not null in to to/from wire
pick 1001fa1da Revert "lightningd/bitcoind: remove unused 'get_output' function"
pick 29fb84ec0 bitcoind: Add typesafe version of gettxout
pick e665d91f0 dual-funding: confirm scriptpubkey matches what's onchain
pick 9ba12df8f fixup! patch up feature addition of v2 fundchannel

This will stop on this commit and allow us to make any changes we want to. Save and exit the interactive rebase document, and it'll execute the commands you've given it. When it reaches the edit command, you'll end up with a prompt like this one:

niftynei@granito:lightning$ git rebase -i 8adbd4d31eac37449d40a6a3ee207d4a4de3e188
Stopped at 8280a3a54...  fixup: pass correct `am_funder` value
You can amend the commit now, with

  git commit --amend '-S'

Once you are satisfied with your changes, run

  git rebase --continue

When we run log we'll see that HEAD has stopped just after the commit that we wanted to update. We've basically gone back in time to that commit and can make any changes we want to the history. Once we play git rebase --continue, git will continue applying the rest of the patches listed in the logs until we reach back to where we started, at the branch tip.

Here's what my git log looks like.

commit 8280a3a54efa1dcbb767f5f499e2ad86fde460d3 (HEAD)
Author: lisa neigut <niftynei@gmail.com>
Date:   Mon Dec 2 13:13:03 2019 -0600

    fixup: pass correct `am_funder` value

commit 3be3df51d80351b14120b2e6534bdd88d7d21e20
Author: lisa neigut <niftynei@gmail.com>
Date:   Mon Dec 2 13:11:42 2019 -0600

    df: convert accepter_sigs to new pattern, add input check

    we're now using funding_signed2 and a commitment_signed instead
    of the accepter_sigs method

commit 8bfcd941ed988ca15e64068ce78bb45125f51119
Author: lisa neigut <niftynei@gmail.com>
Date:   Mon Dec 2 13:09:29 2019 -0600

    fixup: missing bracket

commit a5a14cb76a59a38110c854a96610e9065ea30756
Author: lisa neigut <niftynei@gmail.com>
Date:   Mon Dec 2 13:09:05 2019 -0600

Ok. We want to split the top commit into two different commits. To do this, I'm going to reset to the commit before it. That'll roll the project back to the previous commit, and move any changes in that commit back to unstaged.

$ git reset 3be3df51d80351b14120b2e6534bdd88d7d21e20
Unstaged changes after reset:
M       openingd/openingd.c

Alright, let's add the first change back. I'll use git add -p for this, as it'll let me precisely select the patch I want to stage for commit.

niftynei@granito:lightning$ git add -p
diff --git a/openingd/openingd.c b/openingd/openingd.c
index 5b37dd7d4..d848cbac6 100644
--- a/openingd/openingd.c
+++ b/openingd/openingd.c
@@ -1170,9 +1170,9 @@ static bool send_receive_funding_tx_info(struct state *state,
                u16 num_ins, num_outs;
                int type;

-               msg = opening_negotiate_msg(tmpctx, state, false);
+               msg = opening_negotiate_msg(tmpctx, state, role == OPENER);
                if (!msg)
-                       return NULL;
+                       return false;

                type = fromwire_peektype(msg);
                switch (type) {
Stage this hunk [y,n,q,a,d,j,J,g,/,s,e,?]?

The answer is Yes, I want to stage this, so I hit Y. Then we quit, with q. We only want to check this first change in. Now when we run git status, you can see that some changes have been staged, but that there's still more to add. This is great and exactly where we want to be.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   openingd/openingd.c

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   openingd/openingd.c

I'm going to commit these now, with the info about which commit it belongs to.

$ git commit -m "fixup! df: port to new input/output dance"

Now I add the other commit.

$ git add -p 
$ git commit -m "fixup! dual-funding: wire up opener side of dual funding"

Ok. We're done breaking that commit into two. Now let's finish off this rebase, so we can run a second rebase to apply the commits to the correct fixup. If we add the autosquash flag, it'll automatically place these commits in the correct place, and change the rebase action from pick to fixup.

$ git rebase --continue
$ git rebase -i 8adbd4d31eac37449d40a6a3ee207d4a4de3e188 --autosquash

Here they are moved to the correct part of the file, with their action updated.

pick 83efa2bc8 dual-funding: wire up opener side of dual funding
fixup 9f5c3acc2 fixup! dual-funding: wire up opener side of dual funding
pick 30f6a536c fundchannel_complete: return 'txid' and 'remote signed' tx
pick 70d61f8db df: verify that funding output not included
pick 7cd71d312 df: port to new input/output dance
fixup 4f1e3d908 fixup! df: port to new input/output dance
pick 2183a2b42 df: include input info on witness data
pick 6f829ad20 df: add remote's witnesses to unreleased tx object

All we need to do is save and close so the rebase can run. Viola! Mistakes successfully disappeared! Here's what the commit log looks like:

83efa2bc8 dual-funding: wire up opener side of dual funding
30f6a536c fundchannel_complete: return 'txid' and 'remote signed' tx
70d61f8db df: verify that funding output not included
7cd71d312 df: port to new input/output dance
2183a2b42 df: include input info on witness data
6f829ad20 df: add remote's witnesses to unreleased tx object

As I stated before, don't fuck this up

I'd rate this as fairly advanced Git abuse. It's normal and expected that you might run into conflicts when doing this. If you're extremely surgical and tactical, you'll probably do pretty well (i.e. one line change per fixup commit), but from time to time you'll run into situations where moving commits around incurs conflicts with later patches.

The easiest thing to do in this case is to abort. Fucking up your git history during a rebase isn't impossible to come back from, but there's a good chance you'll lose some work or create a bigger mess to clean up than when you started. If it's really important not to lose something, git rebase --abort will end a rebase and reset your history back to before you started. As I stated at the beginning, the absolute safest way to do this is to checkout a new branch to do the rebasing and fixups in. You can compare the two branches at the end to guarantee that you haven't lost any changes.

If for some reason these warnings didn't save you and you do royally fuck your rebase up, get to know git reflog, the savior of git quagmires.

In Exitus

Pefection is attainable. Don't fuck it up.

#git #rebasing #tips
10 Nov 2019 c.e.
Ch-Ch-Ch-Chia!

tl;dr: A "Proof of Space" algorithm doesn't actually have anything to do with storing user files

I've been curious lately about other 'crypto-system' projects. A friend of mine sent me this nice writeup they did of IPFS; I thought it'd be fun to do something similar for what I thought was another 'file coin' protocol - Chia's Green Paper. Turns out that I was wrong about it being a 'file coin', but it was interesting so here's a quick review of what the paper contained, plus some open questions that the paper doesn't address.

What is Chia?

Chia is another cryptocurrency play. It's aiming to be a 'better' and 'greener' competitor to Bitcoin. As such, it's got a lot of the features that Bitcoin has[0]. It's got blocks, it's got 'miners' (more on this in a bit). It's rumored to have a scripting language not unlike Bitcoin's Script. It's not launched yet, testnet is expected to go live before the end of the year.

[0] I've culled this discussion from both the Green Paper and the public Keybase group.

Mining Chia

The biggest single difference between Chia and Bitcoin is how they secure the network. Bitcoin uses a hash-function to act as a random lottery for all mining participants. Each lottery round is about 10 minutes long and involves you, and every other lottery participant, hunting for the winning combination of bits that will produce a hash of a small enough size. Your chance of winning this lottery is directly proportional (statistically, at least) to how many times you can call the 'hash' function in that ten minutes, relative to the number of times every other lottery participant can also do this. "How many times you call the 'hash' function" is also commonly referred to as 'hashpower'. If you control a lot of hashpower, relative to other miners/lottery players, that means that you're able to run that hash function a lot more times than your other lottery competitors. Over time, if you have 10% of the total hashpower, you can expect to 'win' 10% of the blocks.

You've got to be running computers the whole time to play in this lottery, and those computers consume electricity to run. So Bitcoin uses up electricity to mine; the larger the number of people playing in the miner lottery and the more powerful their machines, the more power that is used on every lottery round, or block mining.

Chia is trying to be 'greener', which means that reducing the amount of electricity you need to participate in the lottery of 'choosing' the next block that is mined. There's a couple of ways that different crypto projects choose to do this. Common ones are variations on Proof of Work or Proof of Stake.

Chia's approach is a combination of two different 'proof' systems, a lottery that requires disk space to participate and a 'timing delay' proof system, that is required to rate-limit the speed with which new blocks can be officially 'verified', as well as a filtering mechanism for picking a 'winner' from the disk-space lottery.

The paper itself names each of these participants in the block creation: there are Farmers, the ones with blocks and blocks of disk space that are being used to 'farm' a lottery share; and there are Time Lords, who use a fixed-step squaring function to rate-limit how fast a farmed block signature can be added to the chain.

The farmers are participating in a 'Proof of Space' algorithm; the Time Lords run a Proof of Time function (also known as a Verifiable Delay Function, or VDF).

Farming it Out

Proof of Space works by filling terabytes of space with junk data and calling it a 'farm plot'. Ok, that's a little rude; the data isn't junk necessarily. But don't think you're going to be able to use that space for anything else.

A farm plot is a set size, let's say 10G of disk data. So 100G of disk space you can fit 10 'farm plots'. I'm probably going to get the details a little bit wrong here, but the general idea is that each farm plot has it's own public/private key pair. Every time a new block is available to be mined, that farm plot signs the block with its private key; the signature scheme is unique[1], which means that one private key and block can only ever produce one signature.

In a previous post, I talked about how the ECDSA signatures that Bitcoin uses are malleable. This is because Bitcoin ECDSA sigs involve a randomly chose value, r. You can generate lots of valid signatures for the same message just by using a different r value (but you really shouldn't publish them, as I believe that this can eventually leak information about your private key).

BLS signatures, on the other hand, are not malleable. Which means for any given message and a single private key, you can generate at most one valid signature. In Chia, this means that one farm plot only ever equals one signature (per message, which is a block in this case).

Ok, so one plot means one signature. These signatures have a 'quality' (measured as the number of leading zeros of their serialized form). A good 'quality' signature has a high probability of being used to finish mining the block by the time lords (we'll come back to this shortly).

This boils down to each plot being a single lottery ticket for every block reward. In our example above, where 10G of space is one plot, and we've got 100Gs that we're 'farming' with, this means that you've got 10 lottery tickets that you can submit for every block that gets mined. If there's 10 other farmers on the network, each with the same 100G of 10 plots, that means that you'd expect to have the best (or smallest) signature about 10% of the time.

In Bitcoin a lottery ticket is a 'hash' that you've computed. You generate additional lottery tickets by computing new hashes. In Chia, a lottery ticket is a farm plot (of disk space). To get more lottery tickets, you need to commit more disk space to farming.

Pretty cool huh? This turns disk space into lottery tickets, instead of CPU time and electricity.

[1] Chia uses BLS signatures to do this; they've got an implementation of it in C++ up on Github. Spec for the BLS signatures here.

Seeding Plots

But, what's in these plots? How does having disks filled with 'junk' give you anything resembling a lottery ticket? Why does the signature depend on this disk space?

The answer has to do with the 'algorithm' that's being used to sign blocks with. Signing a block requires taking squares of large numbers. This is computationally intensive and takes time to do if you compute it from scratch each time. Instead, you can generate lookup tables ahead of time that significantly cut down the amount of computation required to create signatures. A lookup table basically consists of pre-computing the squares of a bunch of combinations of numbers, and then storing them on disk until you need them.

These lookup tables take up a lot of space though. So you've got to store them on disk. This disk space that you fill up with lookup tables is your farm plot. It's required for being able to sign, since without them you'd need to run the computationally intensive calculation every time. This is why the algorithm is a 'Proof of Space' -- you wouldn't be able to competitively submit block signatures (in an attempt to get the 'best signature') if you didn't have gigabytes of data filled up with lookup tables.

Actually, generating these plots is computationally intensive, as you'd imagine, so the startup cost to creating a plot is non-zero. This is good for farming stability, in my opinion, as it makes joining the farming lottery costly. Ideally you'd use a high powered CPU machine to build your lookup tables, and then transfer them to cheap storage being run by a slow or low-powered CPU machine. I/O is more likely to be a bottleneck for farmers than CPU.

Searching For A Good Sig

Why don't farmers just try to sign a bunch of different blocks to get the best signature? Great question. Chia actually allows for this. They call the constant k, it's the number of different blocks at a single chain height that are considered valid. Bitcoin has a k value of 1, which is to say that there's a single chain tip that (honest) miners all attempt to extend. In Chia, they allow farmers to attempt to extend 3 simultaneous chaintips. Which means that for every chain link, your farm plot would sign up to three blocks, or have three chances at winning the block lottery.

There's no real discussion of where blocks come from (who's building them) or what the data that the blocks contain, so there's conveniently no discussion of how the spendable UTXO set is calculated amongst these three-heads of simultaneous block creations. Ostensibly, if a transaction is in one block it wouldn't be valid to be included in another one? Or maybe it has to be mined in 3 blocks before it's considered valid? The paper didn't talk at all about the data structures that are stored in the Chia chain, so it's hard to know what the plan is here, exactly.

Actually, that's another interesting point about how Chia blocks work. The data portion of the blockchain isn't included in the 'stem' of the chain. Instead, the data is added to 'foliage' blocks which are bound to the mainline mined blocks by signatures. This is done to make the signatures non-malleable (because the inputs don't change). There weren't good details exactly how these foliage blocks are composed in the paper, either.

[2] The Chia paper has a good discussion around the implications of different k values.

Implications of Plotting

There's two ways that people can 'cheat' the Proof of Space requirements. One is that you figure out a short cut for running the signature generation step that makes it feasible to calculate the squared products on the fly instead of relying on lookup tables. This effectively turns Proof of Space into a Proof of Work. Since table look ups are constant time though, you'd really need something magical to beat lookup tables on speed.

Two, some form of compression that allows you to store lookup tables in a fast to access/decompress manner. I'd imagine that the Chia team has investigated this, or has even built a compression algorithm into the plot storage data structures. Even if not, table lookups are fast and hard to beat, so again kind of a stretch to imagine that compression will keep you competitive.

The Lordiest Time of All

Using 'high quality' farmer signatures is just the first half of the block 'confirmation' process in Chia. We need something to rate-limit how fast blocks get mined. Rate limits are good because they allow time for blocks to propagate to signers, which gives everyone (in theory) an opportunity to sign the block (and therefore participate in the lottery for it).

Just having farmers sign blocks isn't very rate-limity though. To overcome this problem, Chia tacks on another 'proof' system, a secondary algorithm or 'proof puzzle', whose answer takes a non-variable amount of time to calculate. This class of algorithms is called a 'Verifiable Delay Function' or VDF for short. They're called that because of the 'verifiable' delay that they introduce into a system; arrival at the answer means that you've run through a deterministic number of steps.

The solution to a VDF typically has the property that it is easy to verify but difficult (or time consuming) to calculate. Chia uses a squaring function to achieve this, where the number of required squaring operations is determined by a parameter t. t varies based on the 'difficulty' of the chain at the moment and the 'quality' of the farmer signature that it's attempting to prove. (Farmer signatures are one of the inputs to the VDF.) This is where being a lucky farmer and getting a 'good quality' signature for a block works out for you -- a good quality signature means that the Time Lord's VDF function will finish faster than any other farmer's signature. The first one to finish is most likely to be the highest 'quality' and therefore accepted as the chaintip to build blocks off of next.

The rate at which blocks are verified (and thus mined) is limited then by the difficulty factor which determines the t that is used to calculate the VDF result. A lot of good quality farmers (so lots of farm plots), will result in the difficulty rising; a decrease in 'farmer signature quality' results in the VDF increasing, which would bring down the difficulty at the next adjustment cycle. This is similar to Bitcoin.

Time Lord Incentives

One thing that I'm not super clear on is the incentives for Time Lords. Incentives on a whole aren't well covered in the Chia paper; it mainly focuses on the technical details of how to marry VDFs with 'space proofs'. The Chia chat on Keybase briefly touched on the idea that Time Lords won't be compensated for their time, only farmers who are participating in the sig-lottery will receive the block reward for each block mined.

In fact, the network doesn't even really need that many time lords, just enough to make sure that someone is sorting through all of the available signatures and then working towards the next proof that time has elapsed.

Is Chia Actually 'Green'?

In some senses, Time Lords in Chia aren't much different from hashing miners in Bitcoin -- it takes computation power and time and calculate VDFs. I think that this might explain why there's a decided emphasis on devaluing the labor of Time Lords in the system -- adding incentives to the Time Lord function might create another mini-race to compute similar to Bitcoin. Such an arms race would greatly undermine Chia's ability to call itself a 'greener' alternative.

Instead, the protocol will initially pay out block rewards to lucky farmers, instead incentivizing users to fill their spare hard disks with 'junk' in return for a chance to play the Chia lottery.

I don't think this is a terrible way to approach a chain, and I might be wrong about the exact incentive payout structure for Time Lords -- there wasn't any mention of payout schemes in the paper. I will say that I think it opens up the possibility for perverse incentives, or some amount of cheating incentive, in that Time Lords might only attempt to confirm their owner's plots, as that's the surest way to get paid. I can imagine a world where every big-time farmer basically runs their own Time Lord as well, so small independent farm vassals are at a bit of a disadvantage to big, corporate farms that have their own special-purpose built Time Lord machines that can crank out VDF proofs.

Bald Patches

The Green Paper focuses pretty exclusively on the marriage of the two Proofs: of Time and Space, but leaves out a lot of details.

Specifically, there's no discussion of the format of data in the foliage blocks. This includes the UTXO set and transaction aggregation. Where do these blocks come from? Who composes them? It didn't seem like farmers would be the ones organizing and creating blocks to sign, but it's possible I missed something here that is stated.

I didn't really do a fair treatment of the implications of having a multi-headed chain, but to be fair neither did the paper. A lot of my questions on this are tightly tied to the UTXO set questions that I mentioned in the above paragraph, but mainly -- if you've got a 3-tip chain head, where does your transaction data need to be in relation to these chain-heads in order to be considered confirmed? How do you prevent a UTXO from being 'double spent' in two blocks at the same depth? I'm sure that the Chia designers have thought through this, as they seem like fairly obvious questions but it would have been more complete to have some discussion of the problem in the paper itself.

My biggest concern with Chia, however, stems from their lack of complete treatment of the incentive structure amongst the varying requirements and actors within their system. Take storage of the blockchain, for example. They're estimating 500kb blocks[3] every 5 minutes, or about 6MB per hour. Where exactly is this blockchain going to be stored? If you've got disk space to spare, it seems like it'd be in your best interest to turn it into farming plots, not a place to hold the chain's past transaction data. There's an assumption that 'nodes' will exist that host the Chia blockchain and participate in block propagation. On its face, this is analogous to how Bitcoin nodes work. There's a lot of nodes on the network that hold bitcoin, or act as live wallets, that don't participate in mining. If Chia does grow to be as widely used as bitcoin, I can imagine this being a valid use case but in the early startup stage, I'd imagine it being difficult to bootstrap willing node hosters who aren't also interested in using their spare disk space to farm for lottery tickets.

One obvious way around this would be to make the minimum plot size larger than the expected blocksize in the near term. It's also possible that I'm grossly overestimating the total anticipated chain size, and this isn't actually as much of a conflict of interest as I surmise.

[3] I'm unclear what exactly constitutes a 'block' in this example; is that all 3 chain-tip blocks plus their associated foliage, or is it just a single foliage block?

In Exitus

It's really interesting to see the benefits of the bitcoin hash function lottery laid out through the foil of someone else's attempt to replace it. Bitcoin is beautiful because of its simplicity -- its whitepaper is short yet well considered. I understand that it's likely Chia is relying on standard assumptions that were coined by the Bitcoin paper (and hence excludes them), but overall there is no denying that Chia as a system is more complicated and lacking in a few key details as to the practical interaction of its parts.

Nevertheless, I'm looking forward to the testnet launch in the next few weeks.

#blockchains #bitcoin #chia #explainers
15 Aug 2019 c.e.
Quick Note on Malleability

tl;dr: There are a bunch of potentially valid signatures for the same transaction hash and public / private key pair. Read on to find out why.

Problems

I'm working on getting my dual-funding implementation working. I've written a bunch of code, and now I've got to make sure it all works as expected. To do this, I've been writing 'protocol' level tests using a framework one of my teammates put together a few weeks ago. It allows you to specify wire calls that the test runner will send to the program under test, and then verifies that it receives the correct result back.

It's been a really great tool for debugging all the small programming errors that I've got in the code I wrote. Recently I ran into a verification error that stumped me.

The test files that I'm using are generated using a script that calls bitcoind. It builds a 'funding transaction' and all the required signatures, and then outputs that into a test case, that I can run the test runner against.

Part of what it generates is the expected signature data that the c-lightning node will produce for the signature. When c-lightning sends back its signature data, the test runner verifies that the bytes exactly match what it's expecting.

The problem I ran into is that the bytes aren't matching. The signature that my test-script maker produces is not the same as the signature data that c-lightning generates.

What's equally confusing, at least to me, is that the transactions are identical. So are the keys that they're signing with. How are they getting different signatures then?

Transaction they're both signing (they're signing the first input)

02000000023f05da71fb470a53807eb01db4a9220941e3737fa44bccd1fb00f1968c98396c0000000000ffffffff3f05da71fb470a53807eb01db4a9220941e3737fa44bccd1fb00f1968c98396c0100000000ffffffff01a0e2160000000000220020c46bf3d1686d6dbb2d9244f8f67b90370c5aa2747045f1aeccb77d818711738200000000

Witness data that I get from bitcoind:

3044022078afb02e8c8802f65fe92096eb851c623a3bfb631cc8e41878728f35fc94448202203958fb2ed3d58fcd30d1bdd7ab90e1e2dfb524d8423c457719065e7ea5bf98c001
02d6a3c2d0cf7904ab6af54d7c959435a452b24a63194e1c4e7c337d3ebbb3017b

Witness data that I get from c-lightning:

3045022100e4ac2f9b298df16be6c287f7b452468ca09d79816f89674ea4fbe0999a3ef6b80220179186b253e1df02714a01b2c19144efa141a7072049e51cddd299572a0cbbc801
02d6a3c2d0cf7904ab6af54d7c959435a452b24a63194e1c4e7c337d3ebbb3017b

You'll notice that the second line of these is the same 02d6a3c2d0cf7904ab6af54d7c959435a452b24a63194e1c4e7c337d3ebbb3017b, but the first line is different.

I thought that transactions in SegWit were supposed to not be malleable? Why are my signatures different?

Short Explainer on ECDSA Signatures

Turns out that every ECDSA signature includes a nonce, called r. This is a randomly chosen number. The data that is sent in a 'bitcoin signature' includes this r value and the actual signature s. You use r and the public key to verify that the signature is correct for the transaction.

In fact, the second line that was the same above, 02d6a3c2d0cf7904ab6af54d7c959435a452b24a63194e1c4e7c337d3ebbb3017b, is the compressed public key. We know that both c-lightning and bitcoind are using the same public / private key pair to sign it. The 'problem' is that the r values are different.

This isn't actually a problem. Both signatures are valid. We can dissect the signatures to get the r values out.

c-lightning signature, dissected:

3044 02 20 
    // r   78afb02e8c8802f65fe92096eb851c623a3bfb631cc8e41878728f35fc944482
     02 20 
   // s   3958fb2ed3d58fcd30d1bdd7ab90e1e2dfb524d8423c457719065e7ea5bf98c0
01

bitcoind signature, dissected:

3045 02 21
    // r 00e4ac2f9b298df16be6c287f7b452468ca09d79816f89674ea4fbe0999a3ef6b8
     02 20
    // s 179186b253e1df02714a01b2c19144efa141a7072049e51cddd299572a0cbbc8
01

The r value for the bitcoind signature is 78afb02e8c8802f65fe92096eb851c623a3bfb631cc8e41878728f35fc944482. The r value for the c-lightning signature is 00e4ac2f9b298df16be6c287f7b452468ca09d79816f89674ea4fbe0999a3ef6b8. Clearly not the same value.

Which is why the signature data is different. Sigh.

What Happened to Malleability?

SegWit is known for having made transactions non-malleable. This does have something to do with the signatures, but not in the way that I thought.

Prior to segwit, you could take a transaction, change the transaction enough to change the transaction id, and still use the same signature. Which is to say that one signature would be valid for different versions of the same transaction. This is bad for lightning, in particular, because we rely on transaction ids staying the same, as we build transaction chains for commitments and the like.

This kind of malleability has to do with the range of transactions that any given signature is valid for. Basically, it means that one signature is only good for that one version of the transaction. If you have a signature, you can't change the transaction and publish it and still have the signature be valid.

It says nothing about the range of valid signatures for that one transaction.

A short diagram might be kind of nice here.

One Signature -> Only valid for one transaction.

One Transaction -> Lots of valid signatures! (with different `r` values)

So, are my test cases ok? Yes they're perfectly fine. What's the solution here? Probably not to verify the signature data that the node under test sends back. I should really just check that the public key works, and do a few modifications so that the node under test has to have their published transaction be accepted by bitcoind in order to continue (currently the test runner also broadcasts the transaction).

In Exitus

Malleability only applies from the "I have a signature, what transactions is it good for" sense. Not "I have a transaction, what signatures are valid for it?".

#bitcoin #sigs #malleability
3 Aug 2019 c.e.
The Importance of History

I work on c-lightning and one thing that I spend a lot of time doing, as a new c-lightning contributor, is making my git commit history as perfect as possible. What is a perfect git commit history? There are two things that our team uses to judge whether a Pull Request's commits are good for committing: how bisectable it is and how readable it is. Bisectability is a concern for the machines, or keeping your commit history interoperable with a useful tool, git bisect. Readability has to do with documentation, keeping your commit history interoperable with other humans that are maintaining the codebase.

Bisect

Tooling. It's important. It's especially important that you keep it in a state where you can continue to use it effectively. git bisect is one of those tools that is incredibly powerful, but only if you keep an incredibly high bar of 'cleanliness' in your commit history. Basically what this boils down to is that every commit must, at a bare minimum, build. At its core, this has to do with how git bisect works.

For the uninitiated, git bisect is a built-in git utility that helps to identify the patch that introduced a bug. In order for it to work effectively, however, you need to be able to build (and hopefully run the tests) on every commit.

To start a git bisect session, you run git bisect start. Next, it expects you to mark a good and a bad commit. If the current HEAD is bad, you'd say

$ git bisect bad

Assuming you know at least one commit that was good, you then tell git bisect what the 'good' commit is. As an example, here's how you'd mark a commit hash good.

$ git biset good 9a7b7a8e

From there, git bisect will binary search through the commits between the known good and known bad points until you've identified the commit where the error takes place. It automatically checks out commits, and then you run your tests. If the test fails, you mark the commit is bad (with git bisect bad); if the test passes, you mark it as good (git bisect good). When you're finished, return to head with git bisect reset.

Since git bisect picks 'random' (actually they're halfway between the last marked good + bad) commits, if any commit in your project doesn't build and you land on it with git bisect, you're not going to be able to figure out if it's actually a good or a bad commit. Same for breaking tests as well.

To summarize: non-buildable commits break your abilty to 'easily' git bisect through a codebase, and build at any given commit.

Readability

The other thing that c-lightning reviewers ask for is readability in commit messages. This means that each commit should only contain a single 'logical' change, and that the commit message should accurately describe it. Further, if you submit a set of commits in a pull request, the ordering of the commits should be such that they organize a 'narrative' around what you're changing. If you need to make changes to helper functions, or pull something out so that you can use it elsewhere, that would go before the commit where you use it in the new thing. Same for renaming of things!

Ideally a reviewer will be able to step through the commit set on a PR and be able to understand how, step by step, you got to the final goal that the PR represents.

In Practice

I find all of these things hard to do in practice, but I've been working on it. One reason for this is that I tend to change a bunch of things at once. Another is that often times when you start in on a project you don't have a good idea of the scope of the change that's necessary to accomplish what you're looking to do. So you start in one place, only to discover that there's a number of other things that need to change in order to move forward. I tend to end up with a huge heap of unrelated changes staged in git.

git add -p

There's a couple of ways to get around this. One is to commit more frequently. You make a small change, or rename something, or pull up a method. Then you stop and check it in. Another good tool is to use git add's -p flag. This lets you 'pick' what changes you'd like to add.

$ git add -p

This starts an interactive program that presents you with every edited 'hunk'. You can add it to the staging area (y), leave it out (n), or if it's too big (you want to add a few lines but not the whole hunk) you can edit it, with 'e'. There's a lot more options, you'll see them if you try this. Try them out!

I find myself using 'edit' more than I probably should. I like how fine-grained a control it gives you for adding things to a commit.

If you accidentally stage a 'hunk', you can similarly unstage hunks with the same tool using git reset HEAD -p.

git rebase -i

Once you've made your commits, you can reorder them, merge commits together, or execute a commands against them using git rebase. I use this all the time to add more things to other commits with fixup, to re-arrange the order that commits come in, or to re-write a commit message. A word of caution about re-ordering commits: this is a really great way to end up with conflicts if you're not careful. If you end up with a mess in the middle of a rebase, you can always abort with git rebase --abort.

Typically when I start on a thing, I'll have 20+ commits that will get whittled down to 2/3rds or half of that number for the PR.

I've never done this, but you can confirm that git bisect works by running git rebase -i with the -exec flag. Kamal[1] has a great blog post on how this works here. Typically what you'd exec is the build and check make commands, and also your linter. For c-lightning this would probably be something along the lines of make check check-source.

git rebase is great. Score one for neater commit history and preserving bisectability!

In Exitus

Yes, I spend a lot of time these days massaging commits.

[1] Hi Kamal!

#git #history #bisect
23 Mar 2019 c.e.
10 Years of Twitter

I signed up for Twitter on March 5, 2009. I can't remember why or how I heard about Twitter, or who thought it'd be a good idea for me to get on it. I was in college at the time, and worked as a TA for a professor in the school of Management Information Systems. I strongly suspect she asked me to get on to help promote events that she was running and the like, but it's hard to say. One of the most defining memories I have of it was tweeting from the bus via flip-phone SMS messages to a short code.

Anyway. I've been on the platform, off an on for the last decade. In honor of a decade of tweeting, I downloaded my archive and compiled some stats about how I've used the platform since then.

Methodology

I downloaded the entirety of my Twitter archive from the settings page on March 19th, and then used this python script to sort through things. It's not really well stitched together, but if you want to try it out, here's a general set of commands that I run to get it going:

$ python3
>>> exec(open('script.py').read())
>>> tweets = load_tweets('tweets.csv')
>>> tweets[-1]['text'] # this is your first ever tweet!

Stats

Ok, so here's some basic @niftynei tweet stats, from Mar 5, 2009 to Mar 19, 2019:

Totals

    Total tweets: 18,780
    Retweets: 2,334
    'Self authored' tweets: 16,446

    Total characters: 1,187,164
    Average characters per tweet: 72
    Total 'words' tweeted, excluding retweets: 186,316

Replies

    Replied to: 1,448 accounts
    Most replied to: myself (3,532)
    Top five replied to, links are to first reply ever:
      @zmagg (245)
      @vgr (189)
      @jc4p (169)
      @turtlekiosk (130)
      @lchamberlin (108)

    Longest tweet (by characters): 302 chars, 278 w/o link     Longest tweet (by words): 56 words, 271 characters

    Longest gap between tweets: 319 days, 1:17:08
    Second longest gap: 138 days, 4:36:53
    Shortest gap between tweets: 0 days, 00:00:00
    Median gap length: 0 days, 0:06:21
    Average gap length: 4:01:30

    Most tweets in a day, with retweets: 120

'Collections'

I also found some 'collections' of tweets that I did, based on hashtag. Here's a set of 'quotes' from 'Bob Moses, Software Project Manager' I wrote in 2015, right after reading Caro's Robert Moses book.

  • Jill who'd you tell about our plans to shut down that API? Well, Tim Cook just Slacked me about it #BobMosesSoftware
  • We've already invested two weeks. If we cut it now, it'd be a waste of developer, server, and your time #BobMosesSoftware
  • Make the button blue? Impossible. #BobMosesSoftware
  • These AB test numbers have links to private interests. #BobMosesSoftware
  • Look, the PM who sponsored this feature was a pitiful excuse for a person, and a crank. #BobMosesSoftware

Here's a collection of #words, some with more meaning than others:

  • ex pose say #words
  • bingo buzzchain bandit #words
  • concurrently battling an ur-reductive mental trip #words
  • physical manifestation at the hilt of representation #words
  • strong bold memories of Europe sunshine in the spring #words
  • It's a trinidadian dance funk kind of afternoon of the likes only Pitbull can satisfice #words
  • axiotic dimensional #words
  • "There were 6 right answers but I only knew one" #words
  • some trips you don't come back from 🍃 #words
  • dispatches from the hallowed halls of productivity theater #words
  • Vinculated to the predicated #words
  • What would it mean? To never know the joy of driving a nail, firmly flush with the wood top of your coffin. #words
  • deliciously voyeuriste #words
  • Skewed perceptions: a relativistic model #words
  • Cognoscenti is probably the best word. #words
  • The opiate for the masses. The opiate for the masters. The opiate for the missus. #words
  • tragi-comic #words
  • sliding swiftly into the obdurate past #words
  • spinning dystrophies of inalienability #words

Expanding on the literary theme, here's a series of sentences that might make good starts to novels:

  • I can't stop thinking about the silver Mary I saw at the Sacre Couer #novelstarts
  • And thus began my long love affair with the Q train. #novelstarts
  • She started pointedly: my guilty pleasure is stalking you. #novelstarts
  • It was all the things I had not done yet that kept me awake, instead of all the things that I had that put me to sleep. #novelstarts
  • all of our conversations were just lines of this screen play i was unwittingly writing called You & Me #novelstarts
  • The saddest sadist you'll ever meet lives ... #novelstarts
  • Character assassination was the strategy. Twitter bots, the chosen methodology. #novelstarts
  • "I just want to spend the rest of tonight at disco karoke with Hotline Bling on repeat," she said, breaking into a slow robot. #novelstarts
  • El Doctor te veras. #novelstarts

Heat of the Tweet

Finally, here's a 'heat map' of tweeting from the last 10 years. And yes, I did swipe the formatting from the Github repo's heat map.

2009 Mon Wed Fri
2010 Mon Wed Fri
2011 Mon Wed Fri
2012 Mon Wed Fri
2013 Mon Wed Fri
2014 Mon Wed Fri
2015 Mon Wed Fri
2016 Mon Wed Fri
2017 Mon Wed Fri
2018 Mon Wed Fri
2019 Mon Wed Fri
#twitter #blogging #stats
9 Feb 2019 c.e.
A Taxonomy of Lightning Nodes

It never ceases to amaze me how little the general crypto population knows about how the lightning network works, so I thought I'd write down something that's been quite obvious to me for a while, with the hopes of influencing others to see it my way.

Lightning is a network of node operators. Each node has a wallet with funds, that are then apportioned amongst a set of channels to other nodes. Each channel that is opened has a balance, and each node in the channel has the right to spend a certain amount in that channel. This "right to spend" gives every channel a directionality to it. In other words, which direction the funds can move at any given moment depends on which side has the right to spend them. For this reason, the Lightning network is a directed graph.

Every payment that moves through the system changes the balance of payments in every channel that it flows through. As payment volume grows, managing the 'flows' and ability to send payments from one node to another will become an important and non-trivial management task.

Drawing Lines Between Nodes

A key to understanding how these flows will affect ability to make payments is to understnad that not every Lightning node has the same 'goal'.

In fact, you can classify these nodes into three distinct groups. Each of these groups represents a different policy on liquidity in their channel balances. As such, the actions they will each regularly perform on their channel balances will be distinct. A channel balance is only useful if it allows you to do what you need to on the network, and each of these three actors will have different goals.

Theses three node groups are:
- consumers
- vendors
- liquidity providers

Consumers

This is probably the most intuitive group to understand, since it's every one of us. A consumer is a net supplier of funds to the Lightning network. On a whole, they spend more money over Lightning than they receive. There is a certain amount of exchange that happens among nodes of this type, but this amount is dominated by their outflow to Vendor nodes. Typically, their payments will be to a relatively closed set of repeated contacts.

Generally, the actions a consumer takes will be one of:
- Adding more money to their wallet/outgoing channel balances
- Sending payments to vendors
- Creating new channels to pay new vendors

The apps that these users use are typically mobile wallets and web browser extensions. They're generally interested in centralized/custodial services. Probably not running their own node unless it is their mobile client or they've invested in a small home node.

Vendors

This is the Amazons and Blockstream stores of the network. A vendor is a net drain of funds on the Lightning Network -- they receive more payments in than they send out. They are typically receiving inflows in exchange for a good or service, which means that they'll be withdrawing funds from their channels to cover their costs.

Generally, the actions a vendor takes will be one of:
- Withdrawing money from their channels
- Opening channels with liquidity providers, to get inbound capacity
- Originating invoices

The apps and infrastructure that these vendors use will generally be a bit more intensive and always on than consumers. Their ability to transact will be closely tied to their ability to reliably source inbound capacity. Backups and watchtowers are of a bigger concern to these users than to consumers.

Liquidity Providers

This is the HODLers, people who have a chunk of crypto that they want to put to work but aren't interested in spending it and don't really have much of anything to sell. They've got the time, know-how, and resources to set up a more 'industrial strength' node than the general 'consumer' population. They're interested in writing custom algorithms that can help them figure out how to price their liquidity and are willing to spend the time and energy (generally speaking) to figure out what configuration of channel balances and flows will bring them the best return on their node setup, in terms of routing fees. They earn money by providing liquidity between consumers and vendors.

Generally, actions a liquidity provider will take are:
- Opening new channels to vendors, to provide inbound capacity
- Advertising liquidity
- Rebalancing their channels between vendor + consumer accounts
- Network analysis to discover lucrative avenues to open/create new channels

In Exitus

It's my understanding that the Lightning Network needs all of these types of nodes to function. Providing a visible market for liquidity will make these roles even more apparent. I'm incredibly excited about the inclusion of liquidity advertising in the 1.1 spec, as it will give another lever for liquidity providers and vendors to make decisions about how to most effectively allocate channel balances across the network, in a decentralized and transparent manner.

#lightning #markets #liquidity #taxonomy
28 Jan 2019 c.e.
Reflections on the Art of JSON in Golang

Last month, I put a good bit of time into writing a little library to help bridge the gap between the requirements of JSON-RPC's spec and Go.

The Go standard library provides functionality for the version 1.0 of JSON-RPC. There is no standard library implementation for the 2.0 spec, but there's plenty of other implementations, some of which seem to get pretty close to the idioms that I landed upon for my version of it. I ended up writing my own library for a few reasons. First off, I wanted some practice implementing a spec. The work I'm looking to do for lightning over the next few weeks is basically spec writing and implementation; it seemed like a good idea to get some practice following a very simple and well documented spec like the JSON-RPC 2.0 spec.

Secondly, my motivation for needing a JSON-RPC implementation is that I was looking to write a 'driver' for the new plugin functionality that Christian Decker has been adding to c-lightning. c-lightning's plugins have a few very specific needs[1] that would likely require modifying another JSON-RPC implementation. Plus there's the overhead of figuring out how another person's library works.

I leveraged the json encoder/decoder from the Go standard library as much as possible. The trickiest bit was getting a good idiom put together for how parameters are declared and marshalled into command calls. There's a lot more that went into putting the whole plugin/rpc thing together, but I think for this post it'd be the most delightful to just walk through the design decisions that I made for the way the params parsing works.

Problem Statement

Let's talk a bit about what's going on during a JSON-RPC command message exchange. The general gist is that there's a client who wants to execute a method on the server. In order to do this, we need to tell the server what method we'd like to call (by providing a name) and then also passing in any and all of the information that the method needs (these are typically called 'parameters' or 'arguments'. The JSON-RPC spec terms them params).

Our job then is to provide an interface such that the client can smoothly call a method and then receive a response from the server. The ideal interface for such an interaction would look identical to any normal method call. For example:

func hello(greeting string, to User) (string, error) {
    // magically invoked on the server
    return "result", nil
}

Go provides a json marshaler/unmarshaler, a package called encoding/json. The problem is that the marshaler works on structs, not method signatures.

Instead, jrpc2 takes the tack of asking users to write their method calls as structs. Here's how the hello method that we saw above would be rewritten as a struct.

type HelloMethod struct {
    Greeting string `json:"greeting"`
    To *User        `json:"user,omitempty"`
}

Each of the method parameters is now represented as a public struct field. When we send this across the wire, we'd expect our library to generate the following json:

{
    "jsonrpc":"2.0",
    "method":"hello",
    "params":{
        "greeting":"hola",
        "user":{
            "first_name":"Finn",
            "last_name": "Neal"
        }
    }
}

We need a way to signal to our library that this is in fact a 'method' that our jrpc2 library knows how to marshal into a valid command request object. We do that with an interface, that defines a single method, Name(). Any struct that implements this method will be considered ok for sending over the wire to the server.

func (r *HelloMethod) Name() string {
    return "hello"
}

We still need a way to pass this method call to the server, but from a client perspective that's all we need in terms of defining a new method.

On the Server End

c-lightning's plugin framework requires your app to serve as both a JSON-RPC client and server, since users can invoke method calls from c-lightning that are then passed to your plugin to execute. Server RPC method objects are mostly the same as above, with two additional methods added to the interface, New and Call.

When the server receives a request from a client, it 'inflates' the json request into a ServerMethod object. The New method gives you the ability to do any initialization or setup needed for the new instance. If there's state that needs to be shared between instances of the ServerMethod, you can pass them along here. Here's an example of where you want a New version of the GetManifestMethod to have access to the plugin object.

// definition
type GetManifestMethod struct {
    plugin *Plugin  // private so it's not mistaken for a method parameter
}

func (gm *GetManifestMethod) New() interface{} {
    method := &GetManifestMethod{}
    method.plugin = gm.plugin
    return method
}

This is nice because it lets you share state between method calls. Then there's the actual Call part of the ServerMethod, which obviously is where you do work. Since the 'inflated' struct is 'passed in' as the object of the call (i.e. the whole (gm *GetManifestMethod) declaration, you have access to all of the parameters that were sent by the client.

func (gm *GetManifestMethod) Call() (jrpc2.Result, error) {
    // example of using the plugin obj
    for i, sub := range gm.plugin.subscriptions {
        // ...
    } 
    return result, err
}

If you return a non-nil error from the Call, the server will ignore the result and send the client back an error object. As a final note, if you want your Result to be formatted for json correctly, you'll need to add good json annotations for its fields. We use the default encoding/json package to marshal and umarshal everything over the wire.

A Few Things on The Way to the Forum

The trickiest part of the whole jrpc2 mechanism is the custom marshalling for the param struct. The JSON-RPC spec defines two different ways that params can be passed from the client: either as an ordered array or as a named dict. i.e.

// As an ordered array 
"params": [1, 2, "hello", {"inner":"object"}]

// As a named dict
"params": {"first": 1, "second": 2, "greeting":"hello", "extra":{"inner":"object"}}

Basically, we're wrapping client calls in an outer object, with the 'method struct' being serialized into the params. jrpc2 includes methods to serialize calls as either an ordered array or a named dict, but defaults to the named dict when used as a client. It's worth noting that the order of appearance of fields in a method struct is how they'll appear in the array. If you re-arrange the ordering, and have switched it to use 'vectorized params' (aka an ordered array) then they should be switched in the param call.

Reflection Dragons

In order to do this correctly, I ended up digging in pretty hard to the reflect library. There's a bunch of nuance around deflating and re-inflating objects from json that I really struggled to find good resources on. Most golang articles on reflection stop and start with Rob Pike's article on the Go Blog, The Laws of Reflection, but it doesn't dig in much beyond the basics.

Re-creating a new version of the method struct is fairly straight forward, you can just call the New method. However, for any param that is a pointer on the method struct, we have to allocate a new 'extra' object and then run the json Unmarshaler on it. There's a few steps to this.

First, we need to determine what type of object we should be inflating. We can use the method struct's field declaration to determine what type of new struct to inflate.

When you 'inflate' a new object from a field type, it initially comes to you without a pointer address, because no memory has been assigned to it yet.

Short aside. Originally, method structs on the server didn't have a New command, instead I inflated it directly. Figuring out how to do this took me some amount of time. Unfortunately, I replaced it with the New method, as I wanted a way to be able to share objects across every method call, and then I completely (and accidentally) destroyed my git repo and lost my commit history so I can't show it to you exactly but, it involved inflating a new copy from an existing one and then figuring out a way to get it assigned to an address space so that we'd have a pointer to pass around. This isn't such a problem for sub-fields on the struct, since creating a new one allocates space for all of its member fields.

The only place that you need to do allocate a new object is for a field on a struct that's a pointer. Here's a short example.

// Method struct to inflate
type IdkMethod struct {
    Clue *Clue
}

When we're serializing this to json, we'll pass the Clue object as serialized json (if the pointer exists) or pass null if there is nothing assigned. On the server side, we need to 'inflate' this back into a Clue object, with a pointer that we can assign to the new IdkMethod object. Here's how we do it.

if fVal.IsNil() {
    fVal.Set(reflect.New(fVal.Type().Elem()))
}

We use reflect.New to create a new version of the type of field. We have to use Type().Elem() because the type is a pointer -- we want to create a new struct of the type of the element that the pointer is pointing at, not a new 'pointer to element'. reflect.New returns a pointer to the new object that it has just allocated, which we can directly set the value of that field (e.g. fVal) to.

Another short aside, I don't know how you're supposed to figure out how any of this more complex pointer magic works if you've never dealt with pointers on a fairly intimate level. Language level abstractions are great ...until you fall into the pit of object marshalling.

There's a lot of other little neat things that I ended up needing to figure out to do, like filling in a slice or map. Briefly, here's the code for inflating a set of map objects:

    fVal.Set(reflect.MakeMap(fVal.Type()))
    // the only types of maps that we can get thru the json
    // parser are map[string]interface{} ones
    mapVal := value.(map[string]interface{})
    keyType := fVal.Type().Key()
    for key, entry := range mapVal {
            eV := reflect.New(fVal.Type().Elem()).Elem()
            kV := reflect.ValueOf(key).Convert(keyType)
            err := innerParse(targetValue, eV, entry)
            if err != nil {
                    return err
            }
            fVal.SetMapIndex(kV, eV)
    }

You can find all of these great things and more at work in the innerParse function of the jrpc2 library. Currently it lives here.

In Exitus

I'm half-convinced there's a construction of param parsing where you only need to declare the method, and you can somehow 'shadow compose' the request objects that I settled on above. But! After using the library for making a few plugins plus the RPC object for c-lightning calls, I think there's a nice balance between declarativeness and flexibility. Particularly, while at first it seemed a bit redundant, having an explicit Name() function hook for the Method objects nicely decouples the declared method name from whatever is the nicest way to express it in Go.

By way of example, there's an RPC method on c-lightning called dev-rhash. With the Name() idiom, it's easy to handle this:

func (r *DevRhashRequest) Name() string {
    return "dev-rhash"
}

Under the 'more syntactically sugarful' and also imaginary (because I'm not entirely certain you can do it) way that I've been imagining, you'd have to write the Go method like this:

func dev-rhash() string {

And then every place you wanted to use it, you'd have all kinds of ugly dev-rhash() calls. Say nothing of the fact that Go uses upper and lower case letters on functions and objects to denote the 'visibility' of a method -- as written you wouldn't be able to call this method outside of the containing package, which for a library function renders it quite useless. Anyway, I think the API that I landed on is a decent one, for this reason alone, almost.

[1] The c-lightning plugin to c-lightning relationship is a bit complicated. A plugin is both a 'server', in JSON-RPC parlance, and a client. For most of the commands and the like, a plugin plays the role of a server, providing methods that c-lightning can call out to. Notifications from c-lightning to your plugin take advantage of the client -> server notification framework that's included in the JSON-RPC spce. The one exception, so far at least, is that you can pass back logs from the plugin to c-lightning, such that plugin logs will appear in the getlogs command on c-lightning. In order to do this, your plugin sends a log notification to the c-lightning parent, which inverts the server -> client relationship.

Resources

I cobbled together info on how the more magique aspects of reflection works from a variety of places. Here's some of the ones that I found the most helpful.

How to create an object with Reflection via reddit
Writing into a slice via blog
The exhaustive list of reflection tests in the golang source golang.org
And of course the seminal "The Laws of Reflection" Go Blog

#json #golang #encoding #static #reflection
More...