In chapter 1 I'm assuming this is a typo: 981 should be 983 as former is not a prime?

A Finite Field of order 981

It says after "You might have noticed that the field has a prime order every time."

I am new to python and the double underscore is a little bit confusing to me. I have implemented the addition method for two field elements as below :
def add(self, other):
if self.prime != other.prime:
raise RuntimeError('cannot add fields of different primes')
num = (self.num + other.num) % self.prime
return self.class(num, self.prime)

but while executing print(a+b==c) I am getting the following error
TypeError: unsupported operand type(s) for +: 'instance' and 'instance'

what is happening here? and how do i run the add method on two field elements?

PS: I am not sure if github issues are the right place to ask such doubts, but let me know if there is any other discussion forum where these types are being discussed.

Thank you.

In the „Parsing the payload“ and „Getting headers“ section, the image shows the command as a whole and then each item separate together with its meaning. However, the first items (protocol version) are different. In the full command it's 7f110100 and in the description it's 72110100.

Hi @jimmysong
I propose extending the clock model for negative numbers, for both the dividend and divisor:

For negative dividend, it is like turning the clock counterclockwise, as you have described:
-4 % 12 = 8

For negative divisor, it is like the clock hours are negative, and ordered counterclockwise:
-4 % -12 = -4

Cheerio!

I have read the same comment in Andreas Antonoupolus book and I think the bitcoin security implications are not really right.

The public key is hashed into 160 bits. So I think that means that many private keys end up producing the same bitcoin address many times (2^256 / 2^160 = 7.9x10^²28 times).

So the security of bitcoins in a bitcoin address is not 10^77 but aprox 10^29.

Am I wrong?

Thanks for your book.

After y is derived from x, you need to verify that (x,y) is a valid point of the curve

https://github.com/bitcoin-core/secp256k1/blob/master/src/field_impl.h#L130

in the "Why Bitcoin isn’t Turing Complete" sidebar: the second paragraph contains the sentence "Anyone can create a Script program that every full node on the network executes this program." which might read better like "Anyone can create a Script program that every full node on the network must execute."

In the ipynb, the first exercise is missing (to implement op_hash160), so I implemented as follows:
(my implementation deleted) (appears fixed with commit a333e9e)

In the "Longer than 75-byte elements" sidebar, the phrase "There are specific 3 specific OP codes for this" could lose the first "specific".

Just under that last sidebar, the reference link for bitcoin script could be: "https://en.bitcoin.it/wiki/Script"

While explaining the scriptSig of p2pk, it reads "In the case of p2pk, the ScriptSig is just the signature." and I wonder if it would not be more exact to say "In the case of p2pk, the ScriptSig is just the signature, which is comprised of the DER formatted (r,s) signature, and a sighash byte. The sighash byte will later be used during sig-ops to specify how to construct z, that is, what parts of this serialized transaction must be signed"
Similarly, the 01 sighash byte might be colored separately from the length and DERsig.

In "Problems with p2pk", I don't quite understand this sentence: "Later on, Pieter Wuille discovered that the compressed SEC format was existed inOpenSSL, did their use in Bitcoin become more common."

No tests for the last two exercises but I attacked them like:
`

Exercise 1

hex_scriptPk = '767695935687'
hex_scriptSig = '52'
stream = BytesIO(bytes.fromhex('07' + hex_scriptSig + hex_scriptPk))
print(Script.parse(stream))
`
and

`

Exercise 2

hex_scriptPk = '6e879169a77ca787'
hex_scriptSig = '0051'
stream = BytesIO(bytes.fromhex('0a' + hex_scriptSig + hex_scriptPk))
print(Script.parse(stream))
`

Regarding the Pinata (I call them puzzles), this comment is mostly about my own bewilderment. Are these not unsafe? I mean, is not the very best strategy for every node, to look for someone spending these as the tx is relayed, swap out the outputs and swap-in their own output to claim the prize... and so on right up until the miner does it themself? How was the sha1 puzzle claimed if a miner was not helping the one who solved? Don't these fail the isStandard() test for relaying? I wonder if a better pinata might be built with p2sh, where redeem scripts pass recent isStandard() rules and where they are signed by a key which is the solution... then published somewhere online so others can learn about them instead of scanning the utxoset. Your thoughts?

Hi.

In chapter 3 this expression is incomplete & confusing.

uG + vP = z+re) / ((z+re)/kG = kG = R

Thaks for your book.

The solution uses the parse() method:

>>> envelope = NetworkEnvelope.parse(stream)

However, this method doesn't exist yet. It is created in the next exercise.

 def __pow__(self, exponent):
    	n = exponent % (self.prime - 1) (1)
        num = pow(self.num, n, self.prime)
        return self.__class__(num, self.prime)

Make the exponent into something within the 0 to p-1 range.

Actually the range here is 0 to p-2 is it not, as its modulo p-1?

Also this is something I couldn't intuitively follow and would prefer a little more elaboration. So far we have only multiplied by n^p-1 to deal with negative exponents and used modulo p to map integers into the field of p and personally I couldn't make the jump to mow applying modulo p-1.

Hi.

If I am correct fee calculation has changed with segwit.
Weight is used instead of bytes length.
Unfortunately nothing is said about that in the book.
Are fees calculated the same (with weight) for all 4 segwit scripts (p2wpkh, p2sh-p2wpkh, p2wsh, p2sh-p2wsh) or just for the "pure" ones (p2wpkh, p2wsh).

Calculating fees is for me a terrible problem cause they are deciding when you add inputs & outpus but after that comes the signing, adding redeem scripts, etc.
Fees is something that has to be explained in detail.

On the image of the merkleblock message the size of the flag bits field (varint) is not specified.

Hello, I noticed a typo in Chapter 1 .asciiidoc at "Coding Exponentiation in Python":

Coding Exponentiation in Python
We method need to define the exponentiation for FieldElement, which ...

in the Solution for Exercise 6 the variable that contains the number of hashes is called „num_txs“:

class MerkleBlock:
...
    def parse(cls, s):
        ...
        num_txs = read_varint(s)
        hashes = []
        for _ in range(num_txs):
            hashes.append(s.read(32)[::-1])
        ...

Something like „num_hashes“ would make more sense..

While explaining 'version':

• the first paragraph has the phrase "...that's a version number that's very differnt than Windows 8 or Windows 10." where "different" is missing the second "e".
• the last paragraph has a sentence "It actually is when the interpreted as a little-endian integer..." which has an un-needed "the".

Just after the 'varint/compact_size' sidebar, the phrase "Each input contains 4 fields, the first two which point ot the previous transaction..." has "ot" where "to" was intended.

In the paragraph following the bullet list of tx_in fields, the sentence "Thus, we need to define exactly which output within a transactio that we’re spending." spells "transaction" in bold without the last "n".

In the last exercise which tests Tx.fee(), you've implemented TxIn.fetch_tx() which relies on a TxIn.cache
attribute which was never setup in TxIn.init(). if self.cache = {} is not setup in init(), maybe at least a hint "fetch_tx() was added after we started coding TxIn, feel free to initialize any required attributes that it needs." would be helpful. (appears fixed with break-out of TxFetcher class in commit 30a5007)

We take this ScriptPubKey and put that in place of the empty ScriptSig.

This is only for "bare" scripts like P2PK and P2PKH. For P2SH, it is redeemScript instead of scriptPubKey

The chapter uses several times the „pre/post“-prefix for different BIPs. For example:

Pay-to-witness-script-hash as seen by pre-BIP0141 software

However, post-Segwit nodes will now have encountered the special Script sequence for p2wsh ...

Technically „pre“ / „post“ refer to the period before/after a specific event. So „post-SegWit nodes“ actually means every node after the activation of SegWit. But not all nodes supported Segwit after its activation. So a better way to phrase it could be „Segwit-aware nodes“ or „Segwit nodes“.

However, I guess that virtually all readers would interpret the „post/pre“-prefixes in the intended way. So this is probably a bit nit-picky...

The line "=== Stardard Scripts" should be read "=== Standard Scripts"

Hi.
In chapter 3 in the following example:

>>> from ecc import FieldElement, Point
>>> prime = 223
>>> a = FieldElement(num=0, prime=prime)
>>> b = FieldElement(num=7, prime=prime)
>>> x1 = FieldElement(num=192, prime=prime)
>>> y1 = FieldElement(num=105, prime=prime)
>>> x1 = FieldElement(num=17, prime=prime)
>>> y1 = FieldElement(num=56, prime=prime)
>>> p1 = Point(x1, y1, a, b)
>>> p2 = Point(x2, y2, a, b)
>>> print(p1+p2)
Point(170,142)_223

The second pair x1,y1 should be named x2,y2 (I think).

Thanks for your book.

In helper.py: At first glance, I think of a base58 value more as an array of chr ie: a string, than I think of it as an array of bytes. therefore:
perhaps BASE58_ALPHABET is more clearly expressed as a normal string and not bytes?
Similary, in encode_base58():
prefix is encoding the leading zeros, so I see it more like BASE58_ALPHABET[0] * count
and result could just as easily be initialized as the empty string '',
and built like result = base58_ALPHABET[mod] + result
and returned like return prefix + result
Related, encode_base58_checksum() would no longer need to .decode('ascii') and decode_base58() would not need to .encode('ascii'), but... big but, what else would break? I have no clue yet.
More related: asciidoc changes would also be needed as encode_base58() and encode_base58_checksum() are discussed in chapter 4.

I know this is rather nit-picky, what really caught my eye was the expression prefix = b'1' * count which made think about 0x31 instead of hinting me straight towards index-0 of the base58 alphabet, then I realized that base58 results are all being treated like bytes instead of strings while in helper.py.

Graph for A + (B+C) looks wrong. First reflection (red line) is done from a point not on the graph.

Hi.

I found the wiki in https://en.bitcoin.it/wiki/Main_Page

In particular de list of opcodes is in https://en.bitcoin.it/wiki/Script

In Exercise 2, either the file name (helper.py) is missing or the word „in“ is redundant.

Write two functions in h160_to_p2pkh_address and h160_to_p2sh_address that convert a 20-byte hash160 into a p2pkh and p2sh address respectively.

Will you also be covering LN programming?

On Figure 1 the transaction index in the example transaction is 01000000. But in the field description it is 00000000

https://github.com/jimmysong/programmingbitcoin/blob/master/ch03.asciidoc#elliptic-curves-over-finite-fields

I believe the image listed under the title "Elliptic Curves over Finite Fields" is actually still displaying the "Elliptic Curves over Reals" image. It should be a different image and is confusing to the reader.

Ch 13 needs an introduction to Bech32, and directs readers to BIP173

The terms „block“ and „block header“ are sometimes used interchangeably. For example:

A valid proof of work is a hash of the block which, when interpreted as ...

I wonder if it wouldn't be (pedagogically) better to use the correct technical terms to avoid potential confusion?

Pg. 13

Think of Finite Fields as something that you could have learned in High School
instead of Trigonomotry, just that the education system you’re a part of decided that
Trigonomotry was more important for you to learn.

Trigonometry misspelled

...Elliptic Curve Cryptography. That, in turn is essential for…

Should be a comma following in turn

Pg. 14

Let’s unpack each of the following.

Should be “above” instead of “following” (since its referencing the list above) and maybe followed by a colon

We name the elements 0, 1, 2, etc because they’re convenient for our purposes.

etc should have a period at the end, "etc."

Pg 15.

If not, we got an invalid Field Element

Change got to "have" or maybe "get"

Pg. 18

After all 11+f17=9 just doesn’t look right for most people because they’re not used to Finite Field addition.

Add a comma proceeding After all. Fix the finite field addition symbol.

Pg. 21

The difference here is that the exponent is not a field element, so has to be treated a bit differently

Insert "it" after so

Pg. 25

We want, for example something like this to work:

Insert comma after for example

Pg. 26

In thes chapter we learned about Finite Fields and how to implement it in Python. We’ll be utilizing thes in Chapter 3 for Elliptic Curve Cryptography.

"This" mispelled as thes in two instances

The part that describes the fields of the transaction in Figure 20 contains several items that are not in that transaction. It's actually the same description as that of Figure 21.

test_verify_input is missing

The only soft forks to utilize BIP9 have been BIP141 (segwit) and BIP91 (reducing threshold for segwit).

The CSV softfork (BIP68,112,113) was the first one activated by BIP9, using bit 0

BIP91 did not follow the BIP9 default parameters (80% over less than 3 days), and essentially no one except mining pools run BIP91. Maybe you could mention it as a sidenote, but this is not a good example of a normal softfork

nLockTime is ignored if nSequence of all inputs are 0xffffffff. The consequence could be fatal if this is overlooked.

By far, this chapter wiped me out. Here are my thoughts/experiences/troubles.

The writing was rather good, or Im not seeing typos/spelling mistakes because Im tired.

I kicked myself, having to struggle through the 2nd exercise Tx.verify_input() because Id written Tx.sig_hash() to indeed use a tx copy, but not deep. Specifically, sig_hash() tests passed and z was properly calculated, but Id blown away my real tx inputs so verify_input() couldnt possibly succeed. Perhaps unit-tests that check if the tx has changed from how it was originally created fromhex could fail during the sig_hash() tests and lead the coder to fix this likely common mistake. I dont recall enough of a warning in the written text but I already had this in the back of my mind as I started with sig_hash()... I just didnt go deep enough and was using the original TxIn instances.

I think I altered Tx.fee() which had an optional testnet boolean param however since the Tx is always initialized for mainnet or testnet, I didnt feel I should have to remember what Im doing when calculating a fee, rather it should just be passing self.testnet into TxIn.value().

I also started Tx.sig_hash() by setting script_sig values to b'\x00' however I needed to set these to Script([]) instead (at least when more than one input) because script_sig later needs to implement .serialize() which Script() does but bytes() does not.

When working on the last exercise, I found myself wanting a from_wif() function but couldnt find where that was implemented... so I cheated instead, but at the time I thought it would be handy.

I got lucky enough to find faucets that all spent from segwit utxos... I suspect the transaction parser was choking on these with the 2b offset between version and inputs, but I was getting strange error messages as if Script was running into unknown opcodes (212 in my case, read marker as no-inputs, the flag as 1 output, first 8b of my input prev-txid as the first tx amount, and then failed in Script.parse() somewhere in the middle of that first input's prev-txid)... so I think the cursor was out of place. I ended up sending regular tx's to the addresses created earlier and didnt have the same problem once I was spending from utxos that had no witness. I could see many others possibly having similar problems, unless there are specific faucets to select txs that will not have have witness fields. One thought might be for TxFetcher to know if segwit is supported and be able to pass this in the request to http://tbtc.programmingblockchain.com:18332/rest/tx/non-segwit/ and for that url to return old-style serialization... until the library supports segwit parsing.

I realized that Jimmy is hard at work, I think since the last chapter, as I couldnt match-up my local git copy with what I was reading online. I think Im working on a moving target, so I will take a break and see how far Jimmy gets. Im really enjoying this work, I suspect that Programming Bitcoin will be one of the 'must have' references for coders. Go Jimmy!

Hi @jimmysong,
Quick question regarding chapter 1: finite field division

From my understanding.
We derive Fermat: n^(p-1)%p=1 where p is prime

Then, we use Fermat's theorem to evaluate finite field division:
b^-1 = b^-1 ⋅ 1 = b^-1 ⋅ b^(p-1) = b^(p-2)

I don't follow this part: b^-1 ⋅ b^(p-1) = b^(p-2)
Here, exponents (-1, p-1) are added up, like in normal math. However, some of the exponents, such as -1, are negative and therefore represent division, which we, at this point are still in the process of defining.

Why is it possible to perform normal arithmetic on negative exponents?

Thanks in advance!

The calculation of the number of bits in __rmul__ is done as follows:

    def __rmul__(self, coefficient):
        bits = math.ceil(math.log(coefficient, 2))
        ...

However, this doesn't seem to work if the coefficient is 1, 2, 4, 8, 16... A simple fix could be:

 bits = coefficient.bit_length()

Btw, the (1) for this code line is also missing.

Hi.

I consider this part confusing:

The interesting thing about Scalar Multiplication is that at a certain number, we get to the point at infinity (remember, point at infinity is the additive identity or 0). If we imagine a point G and scalar multiply until we get the point at infinity, we end up with a set like this:

{ G, 2G, 3G, 4G, …​ nG }

It turns out that this set is called a Group and because n is finite, we have a Finite Group. Groups are interesting mathematically because they behave a lot like addition:

G+4G=5G or aG+bG=(a+b)G

When we combine the fact that scalar multiplication is easy to go in one direction but hard in the other and the mathematical properties of a Group, we have exactly what we need for Elliptic Curve Cryptography.

Why is this called the Discrete Log Problem?
You may be wondering why the problem of scalar multiplication is referred to as the discrete log problem.

We called the operation between the points "addition", but we could easily have called it a point "operation". Typically, a new operation that you define in math utilizes the dot operator (⋅). The dot operator is also used for multiplication, and it sometimes helps to think that way:

P1⋅P2=P3

When you do lots of multiplying, that’s the same as exponentiation. Scalar multiplication when we called it "point addition" becomes scalar exponentiation:

And then like out of the blue comes this part which I don't understand:

Any Elliptic Curve has to be defined with the following parameters:

We have to define a, b of the curve y2=x3+ax+b.

We also define the prime of the finte field, p.

===> We define the x and y coordinates of the generator point G

===> We also have the order of the group generated by G, n.

The brief mention of G in the 1st block doesn't prepare me to understand the 2nd block.
And I don't see where the n comes from.

Thanks for your book.

You may want to remind your readers that varint is not CVarInt, but CCompactSize in Bitcoin Core

Noticed that you used (1) etc..to correspond to your comments. Maybe use # instead?

NG : Points are equal iff they are on the same curve and have the same coordinates
OK : Points are equal **if** they are on the same curve and have the same coordinates

The check_pow() method uses hash256() to get the hash:

    def check_pow(self):
        '''Returns whether this block satisfies proof of work'''
        sha = hash256(self.serialize())
        proof = little_endian_to_int(sha)
        return proof < self.target()

I wonder if there is a particular reason why the hash() method that was created in a previous exercise isn't used here instead.

1. The definition of finite field should start with a nonempty set, to ensure that it contains 0. To be precise, there are also distribution laws, as well as commutativity and associativity of the two operations.

It's a fact, though not immediately obvious, that every finite field contains a prime power number of elements. Thus there is no finite field of order 981 = 3^2 * 109. (in integers modulo 981, the numbers 3, 109, and their multiples don't have multiplicative inverses).

The finite field of order 8 is not what you think it is. If you take the set of integers modulo 8, then 2, 4, and 6 do not have inverses so it will not be a field.

(Notice a pattern? In the integers modulo n, the number m has an inverse if and only if gcd(m,n) = 1. Proof: Euclidean algorithm).

The finite fields of exactly prime order (finite prime fields) are indeed what you think they are.

To get powers, you need to start with the prime field and add on some algebraic elements:
For example, to get the finite field of order 4, you start with F_2 = {0,1} and throw in for example a root of the polynomial x^2+x+1 (note that F_2 does not already contain a root). Call that root a. Then F_4 is {0,1,a,a+1}. This is somewhat analogous to how we construct the complex numbers from the reals by adding a root of x^2+1, this time with finite fields. Note that a+1 is also another root of x^2+x+1. Addition works by treating 1 and a as linearly independent elements and multiplication works by remembering that a^2 + a + 1 =0 i.e. a^2 = a+1. (Since -1 = 1).
To get F_8, you need to add on some more algebraic elements coming from polynomial roots. The polynomial x^4+x+1 will do the trick. If a root of it is b, then F_8 becomes {0,1,a,a+1,b,b+1,a+b,a+b+1}. You can check that this really does satisfy the definition of a field. For example the mult inverse of a is a+1 since a(a+1) = a^2 + a = a+1 + a = 2a+1 = 0a + 1 = 1.

Maybe you want to focus on finite prime fields (according to standard math terminology, the rational numbers is a "prime field" (the only non finite one), so there is a need to say "finite prime field" here). To make this more accessible, you could ignore the thing I said about prime powers and just mention that the integers modulo n is a field exactly when n is a prime (the proof of this should be accessible), but finite fields of prime power order exist and are not really used here. When n is not prime, inverses of non-zero elements wont always exist (and this looser structure is often called a ring).

@jimmysong Congratulations on the book deal! It would be pretty nice to have a Slack/IRC/Discord/something where would-be contributors could follow your uncommitted progress and discuss the project. Do you happen to have any plans for such an online community?

Problem: @jimmysong not responding to any issues related to this repo.

Repro steps: Create an issue.

In a 5-of-7, the 5th signature can be for any of the 7 public keys. The signatures do not have to be put in any particular order.

Not really. The order of signatures MUST be same as the order of keys. This is consensus critical, and it makes sure that m-of-n multisig will perform at most n ECDSA operations.

So in a 5-of-7, the 5th signature MUST NOT be for the first 4 public keys

see: https://github.com/bitcoin/bitcoin/blob/master/src/script/interpreter.cpp#L1025

While explaining closure, the line: "(a+b-n)G=aG+bG-nG=aG+bG-O=aG+bG" appears to have a capital-oh: Is that meant to be? what does O stand for, order? or should that be zero 0 for point at infinity?

While explaining commutativity and associativity in Ch03, the asciidoc uses P, Q, and R while the images in figure 8 and 9 switch to using symbols A, B, and C. (fixed in commit bf91399)

Under 'Defining the curve for Bitcoin':

• the second bullet-point mis-spells 'finite' with a missing 'i'.
• the sentence: "The curve was chosen, in part, because n is so close to P." uses a capital-P while 'p' for prime is printed in lowercase elsewhere.

In the side note for 'Why Double-sha256?', the last phrase has an additional 'is' and the word 'possible' seems redundant preceding the word 'potential'.

Hi.

In this code:

class Point:
    ...
    def __rmul__(self, coefficient):
        current = self  (1)
        result = self.__class__(None, None, self.a, self.b)  (2)
        while coef:
            if coef & 1:  (3)
                result += current
            current += current  (4)
            coef >>= 1
        return result  (5)

coef should be changed to coefficient (I think)

Thanks for your book.

In 2106, the block header will need to roll over or use some sort of soft-fork as the timestamp in the block header will no longer continuously increase (due to being reset to 0).

I don't think this is fixable with softfork, unless we do something like a soft-hardfork

transactions utilizing an OP code called OP_CHECKSEQUENCEVERIFY as defined in BIP0068

OP_CHECKSEQUENCEVERIFY is BIP112, which depends on BIP68. However, both BIP112 and BIP68 use version 2.

in preface.asciidoc:

search and replace (5 instances of) "programming-bitcoin" becomes "programmingbitcoin" unless it is intended that final github repo will indeed have the hyphen.

search and replace (1 instance of) "linke" becomes "link"