How to implement a Blockchain from scratch - tasks of a miner

As opposed to the common idea, the task of the miner is not only to calculate the hashes. Its task in a blockchain system is to create a new block with a set of valid transactions and state information. The most important tasks are the followings in an accoun/balance based blockchain system:

- Choose one of the top block of the different partly competing blockchains: as the top of the blockchain always forks, one task of the miner or validator is to pick one of the possible top blocks and start building on top. As these top blocks are competing with each other a heuristic has to be used to choose a blockchain that will most likely the longest one. Miners or validators are cryptoeconomically incetivized to pick the most probable longest block.  

- Pick a valid set of transactions from the transaction pool. If a transaction is valid or not depends highly on the transaction itself. The transaction has contain a valid signature, its from and to address should either contain valid account or the related account should be able to add to the transactions. Transactions for double spending or replay attacks must be avoided. There might be further considerations for picking the transactions, like there might be transaction fee included that has to be maximized and there can be a limit for the available transactions as well.  

- Apply transactions to the state. How it is exactly carried out is again highly dependent on the transactions. If it is a transfer transaction, than the related balances of the related accounts have to modified. If it is a state modifying transaction, the state of the corresponding account has to be modified.  

- As soon as the set of transactions is chosen and the state is calculated, merkle or patricia root both for the transactions and for the state has to be calculated. 

- As a last but one step, proof of work has to been done, meaning that nonce values have to be fine-tuned until the hash value will be less than the given difficulty. 

- As a last step, the new block has to be boradcasted to the system. 

How to implement a Blockchain from scratch - adding system transactions

Adding a system transaction to a blockchain system is actually quite easy. Let we imagine the situation that we have something as a number of privileged accounts - they might be related to nodes as well and we want to initiate a transaction to adding a new one to these privileged accounts: 

- There must be a transaction that contain as an address one of the address of the privileged accounts. 

- There must be a state information in the blockchain containing the available set of system accounts 

- There must be a validation rule that not only validate the signature but checks as well if the related account is in the system state among the possible addresses. 

- There must be a state transforming rule, which in case the transaction is valid adds the new account to the state. 

The system can be started by adding to the first node a system right and then the first node can attach further nodes to the system account state. 

Optionally, there can be other special accounts or activities in the state that can be carried out only by system transactions. 

Certainly it is an open question if such a system is more vulnerable against forking attacks.

Such a system could provide the basis for realizing on-chain governance. Let we imagine the situation that difficult is a variable in the system state that can be modified only by system transactions. There might be one model that every authorized account can modify the variable, however we imagine that each transaction makes just a kind of a vote for the new value. If the number of positive votes reach a certain value, the new difficult will be considered in the future.  

How to implement a Blockchain from scratch - apply transaction to the state

Mining in an account/balance blockchain system practically means finding a set of consistent valid transactions and applying them to the system state. For this activity we need two sub-functions: on the one hand the, transaction has to be validated against the state, on the other hand the transaction has to be applied to the state. For the sake of simplicity we assume that each block stored the whole state, which is copied one to one at the initializing the block.

For a simple transaction only setting data, the pseudo code is the following:

ValidateTransaction (Transaction t)

if t.signature is not valid

  return error signature not valid

  else

    foreach account a in the state

      if a.address == t.fromAddress

        if a.nonce != t.nonce +1

          return error replay attack

        else

          return transaction is valid

      else

        add new account to accounts with fromAddress 

For a transaction transferring money:

ValidateTransaction (Transaction t)

if t.signature is not valid

  return error signature not valid

  else

    foreach account a in the state

      if a.address == t.fromAddress

        if a.nonce != t.nonce +1

          return error replay attack

        else

          if a.balance < t.amount

            return balance is not enough

         else

          foreach account a in the state

            if a.address == t.toAddress

              return transaction is valid

           add new account to accounts with toAddress

      else

        return error from account must exist at transferring money 

For applying the state we have to iterate on all of the account check them if they are valid and after that apply the state change. It might vary again based on the exact type of the transaction. If it is a simple data transfer transaction:

ApplyState (Transaction t)

  foreach account a in the state 

    if a.address == t.fromAddress

      a.data = t.data

for the money transferring transaction it is a little bit more complicated

ApplyState (Transaction t)

  foreach account a in the state 

    if a.address == t.fromAddress

      a.balance -= t.amount

  foreach account a in the state 

    if a.address == t.toAddress

      a.balance = t.amount

How to implement a Blockchain from scratch - minimum wallet

Wallet is practically a set of keys that is stored directly or indirectly in the local node. It is important to note that this is a wallet functionality of a blockchain node, which might have similar functionalities as a mobile or web wallet, however the implementation might be a little bit different. 

In an UTXO based system, a wallet stores the keys and a set of unspent transaction outputs that provide the possibility to read out balance without initiating a full blockchain search. In an account balance based system, a copy of the locally managed accounts are practical to store, with the corresponding private keys. This provides the possibility to read out balances or data in an efficient way. On the other hand, locally created but still not with the blokchain accounts have to be managed as well. 

An account has the flowing data;

- Accounts: a list of extended account information is practical to be stored, meaning the basic account information with the private keys and some flag if they are fully synchronized with the blockchain. If the private keys are encrypted, further information might be required as well.

A minimum wallet should implement the further functionalities:

- CreateNewAccount: creating a new account locally, generating a new public private keys and calculating the necessary further information. The functionality might be implemented differently, if it is a random wallet, hierarchical wallet, or hierarchical deterministic wallet. 

- ImportAccount: importing an account based on the private key.

- CreateAndSignTransaction: creating a new transaction which can be any type of a transaction like value transferring or data setting transaction. The transaction is signed by the private key of the related account might vary based on the type of the transaction. A value transferring transaction is signed by the private key of the sender address. After a successful signature, the transaction is broadcast ed into the network. A value setting transaction is signed by the private key of the account which value has to be set. The functionality might be separated into several sub-functionalities, like creation transaction, signing an existing transaction, and broadcasting into the network as three separated tasks.    

- GetBalance: getting the balance of an account or of the whole wallet. 

- BackupWallet: backing up the locally stored account information, especially the private keys. 

- RestoreWallet: restoring the account based on the backed up information, especially on the private keys.  

How to implement a Blockchain from scratch - transactions

Transactions are responsible in every blockchain to create changes in the system. Considering an account/balance based system, they are actually simple statements saying transfer money from an account A to account B, or change the state of account A for a new value. It is important that a properties of a transaction should be set only once, practically at the beginning, to avoid possible hacking attempts.  A transaction should contain at least the following elements:

-   TransactionId, it is actually a hash of all the important values of a transaction. Practically the hash of all of the previous elements. TransactionId provided on the one hand as a kind of a primary key for the transaction itself, the transaction can be identified based on this Id. On the other hand, it might provide a kind of a hacking resistance consistency the transaction is only valid if the TransactionId is consistent with the other values. It is certainly a question if the TransactionId itself should be stored on the blockhain or if it is enough to generate it. If we generate the value, we might miss one consistency guarantee, on the other hand we might as well save storage space on the chain. At any case TransactionId is practical if we want to refer to created but still not signed transactions on the client side.  

- Nonce: the value should avoid replay attacks. It should be set by the wallet software as an incremental value of the account nonce. 

- Address: the address of the account that we want to modify, or from which address we want to transfer cryptocurrency. If the address is the public key, this field should contain the public key, if it is calculated value of the public key like with hashing or double hashing, than this calculated value should be here. 

- Signature: valid transactions must have a signature, which is the data of all relevant information in the transaction, signed by the private key. The signature is generated by the used cryptographical algorithm, like with the help of Elliptic Curve Cryptography. In case we have TransactionId as well, than this id should not necessarily be presented in the signature. The reason for that is that we might want to administrate valid but still not signed transaction on the wallet side. 

Depending on the exact transaction type we can have further properties as well. It is important to note that in a given system, we might as well several different kinds of transactions, like one for transferring money, and further ones for setting data, like in case of an identity management system. 

- ToAddress: if our transaction is a value movement transaction, we will need the address where we want to move the money. 

- Amount: in case of a value movement transaction, we will need the amount to transfer as well. 

- Data: if our transaction is meant to register data in the blockchain, we will need the new data value as well. 

Transactions need to have the following functionalities: 

- Create transaction: in a way that all of the important properties can only be set once.

- SignTransaction: with a private key and a given cryptography the signature of the transaction can be created. The signature should be in case as well to be able to set only once. 

- VerifyTransaction: based on the signature, exiting data parameters and public key, the signature can be verified. If the public key is directly the address, the signature is simple. If the address is derived with a hash function from the public key, the public key must be also given as an input.

Advanced scenarios might have further functionalities as well, like creating raw transactions, or partially sign transactions. 

  

How to implement a Blockchain from scratch - accounts and balances

There are two kind of a blockchain: 

1. UTXO based systems stores only transactions with inputs and outputs represented as practically coins, an output is practically spent, if there is a transaction which refers with an input to the output of the other transaction.

2. In state based systems there is an explicit representation of an account which contains a balances or other information as data.  

An account has the following data structure:

- Address : it is a public key or the hash value of the public key.

- Sequence : it provides a protection against a replay attack, in simplest case it is an integer, in more complicated scenarios it is a kind of a ring of hashes.

- Balance: if the blockchain solution implements cryptocurrency either externally or as an internal cryptoeconomical incentive, there must be a variable for that of a type double or float. 

- Data: if the platform implements something other that cryptocurrency, we can have some more data elements as well, like a string or strings for identity management or an array of key value - value pairs for a general smart contract system. 

- AccountId: it is questionable if account id is required, as the account address should identify the account as a primary key. The implementation of such an id is the hash of all or some data in the account. On the one hand, It provides some more consistency and hacking resistance, on the other hand however, the value should be recalculated at every new value assignment or balance change.

If the account of a wallet is not stored in the blockchain but in the local wallet, than further data and functionalities should be taken into consideration:

- Private Key: As the wallet creates and signs transactions if the account, private key of the account has to be stored as well, either in a plain or an encrypted version. It is important that private keys should somehow, directly or encrypted, be stored on the wallet side, but they must not be stored in the blockchain. 

- Syncronised with the blockchain: if we create a brand new account with the help of the wallet it might still not be added to the blockchain. It can be added to the blockchain at the first transaction (like at changing value of the account or at transferring money to that transaction), or it can be added with an explicit transaction. Independently if the account is added to the blockchain, at each round there might be a synchronizing round that synchronized the account values from the blockchain state to the local wallet account store.

As a functionality one should provide the following services on the wallet side related to the accounts:
- GenerateAccount: creating a brand new account, means creating a new private and public key with a cryptography and key generation mechanism, like eliptic curve cryptography. The private key should be given back to the user and stored in the wallet account possibly with a corresponding symmetric encryption mechanism. If the address of the account is not the public key directly, the address should be generated from the public key, like with a help of different hash algorithms. 

- ImportAccount: the account should be created with the help of an input private key. Public key and address should be generated based on the given cryptographical protocol. As this is not a brand new account, the data of the account must be synchronized from the blockchain.
- SyncAccount: the data, balance and sequence parameters of the wallet account should be updated based on the latest reliable values from the blockchain. 

From a conceptual point of view, accounts that are stored by a wallet are the accounts that are administrated by the wallet. In an account/balance based system, they are similar as storing the list of unspent transactions in an UTXO based blockchain platform.

There might be more than one different style of account in the system, similarly as externally owned and smart contract accounts in Ethereum. Typically one can be responsible only for storing the cryptocurrency balance, as others for storing data on the blockchain.

Block headers in a multi-hash blockchain system

From a practical point of view a multi-hash blockchain means a kind of a multi-header blockchain system. For each hash pointer or each pair of hash pointers we have to save a:
- nonce for each hash pointer
- difficulty for each pair of hash pointers
- general retention policy for each pair of hash pointers
- actual retention policy for a given hash pointers

At calculating the new hash of the hash pointers, the following data should be taken into account:
- all the information that is stored in the hash pointer and was previously mentioned
- the hash values of the previous hash pointers
- merkle root of the transactions
- merkle or patricia root of the state

The block of a multi-hash system contains:
- several hash links
- the transactions
- the new state

Types of blockchain governance

One of the critical part of every blockchain solution is governance. It is especially valid for blockchain platforms, like Bitcoin or Ethereum, but it is actually the same for blockchain applications that are realized on the top of a platform. Governance means carrying out changes or further development of the application. There are several models that has different advantages and disadvantages in the practice:

- Off-chain governance means that the governance model is totally independent from the blockchain itself. Changes of the platform is proposed by a community or trusted people and both the implementation and the implementation is carried out by these people as well. Such a model has the advantage that it is technologically less challenging, however it might cause a split in the community resulting a split or hard-fork in the platform itself as well. Such models are realized by Bitcoin and Ethereum where improvements are presented by improvement proposals, like Bitcoin Improvement Proposal (BIP), or Ethereum Improvement Proposal (EIP). The model caused several hard-forks in the above mentioned platforms, resulting for example Bitcoin Cash, or Ethereum Classic. 

- On-chain governance means that several elements of the platform is able to change by a community decision, and both the decision mechanism and the change implementation is supported by the platform itself. Such an example is the difficulty increase or decrease mechanism of the Bitcoin network which is hard-coded in the consensus mechanism and is practically relized by the voting of the miners or the voting of the stakeholders. Other platforms like Monero have much further developed on-chain consensus mechanism, as an example part of the transaction fee is collected into an online wallet as an improvement budget, which can be used for further development or even marketing projects based on the voting of the community. 

From another perspective blockchain governance can be categorized on the fact, who makes the decision. From this point of view there might be the following models: 

- Centralized governance: systems like Ethereum has a pretty centralized governance. From a practical point of view almost all of the changes are governed eventually by Buterin. It has certainly, the advantage of being able to adapt fast, however not necessarily truly open as participation. 

- Open governance: Systems like Bitcoin has a real open governance model. Practically everybody can initiate new changes in the system and the implementation of these changes require a pretty open consensus as well. Practically the core developers, the miners, the exchanges and the shops should all agree in order that a change is implemented successfully. As it is almost a pretty open democratic process, certainly it can be very slow and inefficient in the implementation phase.

- Governance by trusted actors: There can be model between the two previously mentioned examples as well, where several trusted (and sometimes well known) actors can initiate and implement changes in the platform. Such governance model will be implemented directly by Hedera Hashgraph. Similar initiative can be found like in Dash or Monero, where indirectly the stakeholders or the master node responsibles can be regarded as trusted actors.    

Wallet functionality in Blockchain full nodes

Most blockchain nodes contain a kind of a wallet functionality if they provide some kind of a functionalities for the end-users as well. This is actually not a coincident, both in account/balance and UTXO based systems, private keys self are not stored in the blockchain. As a consequence, for creating a new account which is practically a public private key pairs we need a local wallet functionality. Locally created addresses appear in the blockchain only if someone sends money for that address in an UTXO based system or if that account is explicitly added to the system in an account balance based blockchain. As a consequence a wallet has the following functionalities: 

- creating accounts: generating private public keys

- managing already created accounts that are still not added to the blockchain

- managing accounts already added to the blockchain, calculating balance, managing keys ...

Adding an account to the blockchain in a balance based system

In an account/balance based blockchain solution not only the transactions but the accounts and balances are stored in the blocks as states. It is an interesting question how such a system can be implemented. As the state is built up practically by the set of <account,balance> tuples a new account should be somehow added to the state. There might be the following possibilities for that:

- explicit transaction for adding an account to the state.

- implicit transaction for adding an account to the state, with a logic, like: if account not exist in state than add with the initial data.

- some kind of a tricky hash structure that indirectly stores or the possible accounts, but not store them directly. Certainly, it is questionable if such a thing can be realized with the structure consideration of the blockchain. 

A further question arises how such an account should be deleted from the state. A simple zero initialization is surely not enough as we are not sure if that is really an intention to delete. The clearest way is to have the possibility for something as an explicit delete transaction that deletes the given account from the state.