Grant Update March 2019

Technical Updates | April 5, 2019
Home » Company » Blog » Grant Update March 2019

Outputs and Outcomes

Our focus is to:

  • Improve the performance and reliability of the Protocol
  • Provide ability to monitor the exchange rate and make adjustments as needed
  • Maintain the Anchor Master Securing the Factom blockchain in the long term

Grants: Protocol Grant 009  |  Oracle Grant – 007   | Anchor Grant 008  | Protocol Grant 012  |   Oracle Grant – 010   | Anchor Grant 011

Status and Achievements

March was a good month for Factom development.  We saw the release of v6.2.1 on March 7th, which contained several months of accumulated updates.  There were more updates that were released than github displays.  There was also a release candidate for 6.2.2 which was tested on the testnet.  This fixed some unexpected issues that surfaced when some of the federated servers upgraded on mainnet with 6.2.1.  

March 9th also marked the end of the 3rd development grant (number 009).  This was the one which resulted from the Inter-grant proposal in December 2018.  The 009 grant is considered a success. It resulted in release of the massive number of updates that were talked about during the December grant proposal.  The individual grant goals are detailed later.

March also saw more onboarding of non-Factom Inc. developers.  We saw pull requests from these developers in March:

Who Soup from Factomize

Thomas Meier from Layertech

Adam Levy from Canonical Ledgers

There were two Sponsor meetings in March.  They were held on March 13 and March 27. The March 13 Sponsor meeting was recorded and is published here.  Sponsors in attendance this month were  Dominic Luxford, Nolan Bauer, and Nikola Nikolov (Factomatic).

In late February the Factom network experienced a pause, where network protected itself from forking by discontinuing forward progress until consensus failures were resolved.  The main after-action report was posted in the last monthly report, but a much more detailed report was generated. A senior protocol developer spend two weeks analyzing many gigabytes of logs and tracing through the code pathways to determine the failure mode and to see if the initial estimates of the failure were accurate.  It turned out that the initial guess as to the cause was accurate. The deep analysis is available here.

One of the takeaways from the late February network pause was the need to update the reboot infrastructure.  The bulk of that work was accomplished in March. The reboot infrastructure is now in a much more robust state than late February.  The infrastructure updates will continue through April as well.

March saw the release of the updated factom-walletd.  This contained two headline items. It made available disk encryption for protecting data at rest.  It also includes support for managing user identities. This forms one of the base methods of coordination that users of the protocol will use.  It is already in use by the FAT protocol and the Voting subsystems built on factom.

Progress was also made on the refactoring, as a precursor to Sharding.  There were two parallel efforts going on with refactoring. The first is converting the holding queue to a map.  The development was mostly completed in March, but will require more finishing work for development in April. It will also need extensive testing, but as a sharding precursor, it will be useful in production before the massive reconfiguration needed for sharding will be started.

The second sharding precursor is updating how the Directory blocks are generated.  The current method is a bit inflexible, where minor changes that would seem inconsequential introduced timing bugs that would produce incorrect results.  Much of this work was done in March, and will continue into April. This stability update will also be useful in production before the major refactoring begins.

This being the last report in the 009 grant period, we have accomplished all the goals laid out in the grant.

Goal #DescriptionOutcomeDetails
1Respond to any issues impacting the network (as priority) to ensure the network continues to operate.SuccessUnfortunately, this did occur a couple times during the grant period. A report is available here.
2Strengthen the software against consensus failures.SuccessThis was achieved in multiple small ways.  A big example of this is a fix of the minute 9 election bug, which was released with 6.1.1.
3Increase capacity of the network.SuccessThis was achieved and was detailed in a report here.  The report was presented at a sponsor meeting here.
4Add Network Diagnostic APISuccessThis was released and is live.
5Build infrastructure for automated governanceSuccessThe protocol and example code was released and the protocols are being used by FAT and the Voting Grant recipients.
6Support other Grantees, Authority Node Operators, Developers, UsersSuccessThis other core development grant report describes Factom, Inc as being very supportive.



New capabilities that were released

  • Syncing by minutes requires a block download – DBSig messages should be accepted when first seen
  • Optimizations to speed up loading a factomd database – Assorted optimizations for 2nd pass downloading and partial fix for a stalled 2nd pass
  • Update to golang 1.12 – factomd will keep up with the latest golang release: 1.12.
  • Add Brain Swap fixes and added simulation tests – Enhances the testing automation around ‘brain swap’ identity migration operations. This is a common technique used by all ANO’s when a new version of factomd is released. This testing allows us to more easily detect backward-incompatible changes to the protocol.
  • Expose identity functions through factom-walletd API – We are adding this functionality for two primary reasons:

– The Voting Grant being worked on by community members. They are waiting on Factom Inc to build an identity system that accommodates “Standing Parties” and the staking of FCT and EC addresses.

  • Add ability for factom-walletd to store identity keys – We are adding this functionality for two primary reasons:
  • Factom-walletd configuration fails with http authorization – factom-walletd should not return an error about “Invalid Params”
  • Encrypt database for factom-walletd – protect the factom-walletd private key database on the disk
  • Add Identity commands to factom-cli – Allow Identities and Identity keys to be easily managed from factom cli rather than just through curl requests.
  • Support CORS Domains for WalledD – allow a javascript web browser to access the walletd API if the wallet owner set it up as such.
  • Community Contribution – https support and speed improvements – factom lib side of FD-808 – This allows support of https in the enterprise wallet. It also speeds up refreshing for new transactions.
  • Register a SLIP-44 number for the identities – Use a reserved derivation type for identities for cross-compatibility.
  • Add new display options to factom-cli – Improves cli usefulness for scripting factom actions. This would be useful for us for QA and for developers using the factom-cli


Maintenance\QA\Debugging released

  • Message filtering issue caused excess holding and fail to transition to following by minutes. – Fix the message filters which were stopping factomd frown downloading the 2nd pass when catching up with the blockchain.
  • Factomd slows down 2nd pass with full holding map – Speeds up 2nd pass download when syncing the blockchain from the network
  • Brain swap activation height is off by 1 – Activate the brainswapped identity at the expected height.
  • Stop continually evaluating entries from the last block – reduce database traffic when running and loading a database with lots of entries. Also simplify how entries are selected for downloading.
  • Need user friendly error for method “identity-keys-at-height – Return a meaningful error message when there are problems with tracking an on-chain identity.
  • The api identity-keys-at-height returns server error for heights before chain creation – Return a meaningful error message when asking about an identity at a time before it was created.
  • Disallow re-activation of a retired Identity Key – Close attack vector in Identity management which allows a 3rd party to replay an old identity key update
  • Misc. Identity Fixes – update identity spec to be better handled by some 3rd party JSON libraries.  Also, add more restrictions to a key’s validity in identities.



In Progress

New capabilities

  • Enable ability to retrieve receipts at directory block level – New API that gives much clearer visibility to anchor transactions.
  • Factomd V2 API returns Eth txid – With ethereum anchoring enabled, we need a way for developers to ask factomd which anchors covered a given transaction.
  • Improve simulation by allowing fnodes to be added during tests – Allow us to do new kinds of testing where network topology of sim nodes can change dynamically during testing.
  • Update Database Integrity Checker – Useful for testing, for development, and as a sanity check for database changes.
  • Downloads dbstates in batches when syncing from the network – Increases efficiency when syncing from the network
  • Check for repeat ABlock signatures – Allow the database integrity checker to detect an edge case found in testing where there are multiple signatures with the same pubkey.  Not expected in production.
  • Add more simulation testing scenarios – Adding more extensive testing for other brain-swap scenarios will help catch more types of backward-incompatible changes from making it into the codebase.
  • Add more brainswap testing scenarios – Adding more extensive testing for other brain-swap scenarios will help catch more types of backward-incompatible changes from making it into the codebase.
  • Community Contribution – CrossBoot replay garbage collection never ends – let Garbage collector use less resources with cross boot replay filter
  • Inefficiency with falling multiple times – use less CPU when handling p2p peers
  • Community Contribution – Legibility Improvements Part 1 – Factomd code should be more legible when possible.
  • Add unit tests to ensure replay filters are populated correctly on boot – Ensure there are no regressions of populating the replay filter when booting.
  • Community Contribution – remove duplicate filenames of fourSegments26.txt – remove messy output from go mod or go build in any project that has any dependency within the factomd repo, directly or indirectly



  • Factomd not reading from CustomSeedsURL – In a custom isolated test network, allow followers to find peers with the CustomSeedsURL
  • Add DBSigs for top DB State to Database – Keeps signatures from the highest block in the database so that we can provide the block to other nodes
  • The simulation Test other than TestSetupANetork are not run on CircleCI – Expand the number of tests run on Circle CI to help with automated QA
  • Raw Object not found on transaction acknowledgement – Fix intermittent timing errors when using the raw API in factomd
  • When loading from a save state if the current dblock had FCT/EC transactions the balances were not loaded prior to evaluating the transactions. – Fix timing issue when evaluating FCT and EC balances.
  • Factomd Authority JSON marshalling should be usable – correctly export server identity coinbase address and efficiency fields in JSON unmarshaling.
  • Clean up logging of messages. Force filename to lowercase – Improves logging so we can better diagnose problems. Increases resolution of timing and standardize on output files.
  • Make the version and git version set-able from goland – allow goland debugger to show git commit hash and golang version in factomd and the control panel.
  • Inefficient DBState loading while syncing from the network – Better handle downloading the first pass of getting the blockchain
  • Logging improvements from old revisions – Add millisecond to times for better analysis. generally better log behavior. No crash on nil state logging.
  • Wallet database reinitializes – automatically clear the cache of factom-walletd when detected a new local test-blockchain.
  • Make factomd stall if elections try to replace more than half the federated served in a single block – Fixes a potential vulnerability in leader set where a bad actor could cause a coup and take over the blockchain.
  • Sort auth set only on block boundaries – Bug fix related to elections to allow them to resolve properly in some elections.
  • Local EOM or DBSig could be follower instead of leader executed  executed in rare cases and needed to be added to holding – keep around internal EOM messages even if in an odd edge condition.
  • Extra DBSTATE messages were generated when restoring from a savesate – Don’t create extraneous DBstates when loading from a savestate file.
  • Testing API to filter inbound and outbound messages – Add a message filter and an API to set it so that QA can simulate nodes disappearing from the network.
  • Fast track chain commit/reveals – Prioritize Chain creation over Entry creation to help prevent backed up queues
  • Make holding management be dependency driven instead of scan driven – Replace holding queue with a holding map to increase performance as well as to prepare for sharding
  • Refine some of the unit test code – Add unit tests for local commits and local wallet simulations
  • Fix a pokemon instance involving MessageBase – Catch a new style of Pokemon bug found on the testnet with MessageBase
  • Replay filter of entries is not populated on reboot – Allow a federated server to reject replayed messages which are also rejected by the rest of the network within the first hour of booting.


Future Plans

Protocol Grant

  • Ongoing Performance Improvements
  • Ongoing Maintenance corrections
  • Continue supporting the protocol and issues that arise

Oracle Grant

  • Ongoing Maintenance corrections

Anchor Grant

  • Ongoing Maintenance corrections


POSTED: April 5, 2019 BY Kevin Casper IN Technical Updates

Factom Inc's community manager, carrying a load of marketing operations experience from the enterprise tech and video gaming industries.