Grant Update January 2019

Technical Updates | February 4, 2019
Home » Company » Blog » Grant Update January 2019

Outputs and Outcomes

Our focus is to:

  • Improve the performance and reliability of the Protocol
  • Provide ability to monitor the exchange rate and make adjustments as needed
  • Maintain the Anchor Master Securing the Factom blockchain in the long term

Grants that expired in Dec: Protocol Grant Ph2   |   Oracle Grant Ph2   | Anchor Grant Ph2   

Factom is now operating under an interim, anticipated Continuity Grant. Details here:

Status and Achievements

January efforts in Factomd development were focused on QA of the upcoming Feb Release. The several difficult-to-find and subsequently fix bugs that were identified in December led to more blocker bugs in January. By mid January all that remained were bugs that could be worked around.  The bugs did not affect operations of nodes that have caught up with the network. Additional QA happened the latter half of January. Testing on the testnet also revealed some oddities which can be worked around, but testnet experimentations turned out to be very valuable. Thank you to those participating there for helping with the release process.

Another big effort completed in January was to quantify how much the factomd software has improved over the grant period.  The grant specified a target improvement of 50% capacity improvement and 80% improvement of missing message traffic. (missing message traffic is also related to capacity).  The testing showed that these goals were indeed achieved. The report was presented to the sponsors here.

As for emergency assistance, the long streak of stability was broken in January.  The early morning of January 19 (UTC) the factom network experienced a stall event.  There were two takeaways:

  • The restart mechanism worked, but it exposed some weaknesses in the restart script which needs to be refreshing.  With the availability of new vendor libraries that the restart mechanism can use, we decided to rewrite the restart mechanism to better account for error handling, logging, etc.  That work is in progress and is in development on a private devnet.
  • We know that the version of factomd being run in January has several problems related to faulting and determining which audit servers should take over.  As the servers would run for extended periods of time, these confusions would accumulate. After running for so long without an upgrade the errors had accumulated to a point where the network couldn’t recover like normal.  The upcoming release will correct many issues causing this type of confusion.

The 3 Factom, Inc sponsors continued with the 01/02, 01/16 and 01/30 meetings.  The sponsors Dominic Luxford, Factomatic (Nikola), and Nolan Bauer participated with Sponsor Emeritus Nic Robinette as a spectator.  The sponsor meetings are now being recorded and are viewable by general public.
Both videos are available here: January 16th, 2019 recording and January 30th, 2019 recording.

The Development Grant ended On Dec 10, but effort continues.  The deliverable goals laid out in the grants are planned for the next two releases.  The main deliverable of the grants is maintenance, bug fixing, and QA which have been the primary drivers in December and January.

Work in Progress

New capabilities pending in next release

  • Standing Parties (Protocol Grant)
    • Ensures no other standing party can lay claim to the same standing
    • Cannot be intercepted by an attacker before secured in the Standing Chain
    • Can be easily tallied to calculate the standing behind a decision in the Factom protocol
  • FactomD Recognizes Voting (Protocol Grant)  
    • The tracking of identities which can be used to track the standing parties and delegate votes to provide support for ANOs, Grants, etc.
    • Provide APIs to expose information about staked FCT to allow the 3rd parties to do calculations based on this data.
    • Provide APIs to expose information about staked ECs to allow the 3rd parties to do calculations based on this data.
  • New Network Diagnostic API (Protocol Grant) – Gives visibility into aspects of nodes to better understand their performance and what could have caused a stall.
  • Rate Change Alerts (Oracle Grant) – Keeps the exchange rate current with the market to ensure they stay near the intended constant rate.
  • Sept 17th Release Feature development complete but was pulled from MainNet due to issue. Will release once issues identified and QA is complete.  
    • Performance Graphing Tools – Collecting data is complete for required for new internal graphing tools that allow QA a better understanding of how changes affect performance
    • Timeouts Defined Merged – Improvement to simulation tests so when they fail they exit
    • Calculate System Status Always – Populated Control Panel Summary\Syncing by default
    • Upgrade Golang to 1.11 – Keep up with Golang development
  • Coinbase delay for Testnet – Extend delay between a coinbase to all more time to coordinate a test of the coinbase cancel on the testnet.  
  • Diagnostic Graphs – Detailed view of the internal system performance
  • Add Simulation Tests – Increased the amount of features tested during development unit testing
  • Simulation tests in factomd_test.go should support timeouts defined in blocks – Adds ability to limit the amount of time a simulation QA test can run before declaring it has failed.
  • Message Logging Clean up – Allows the clean up of messages to better diagnose issues
  • GoLand Build Tags – GoLand does not support dynamically setting build parameters so this added a prebuild script that overwrites the engine/version.go file to accomplish what is done in production using build tags.
  • Modify factomd API call ‘current-minute’ to also return ‘current-block-height’ – Allows you to make a single API call and returns the minute and block height together to make testing more reliable
  • Upgrade to golang 1.11 – Continue standards of staying current version
  • Community Contribution for Integration of CORS code into Factomd – Allows Javascript browsers to use the data that is provided by Factomd.
  • Investigate new theory of database head reversal – This fixed a problem observed in the great stall where the last DBState in the database was not loaded when restarting after a stall which in turn caused nodes to boot to different heights which blocked the network from making forward progress without a lot of elections
  • Encrypting the Factom-walletd Database – Exchange Requests that requires password security when using wallet
  • Simulation tests in factomd_test.go should support timeouts defined in blocks – Adds ability to limit the amount of time a simulation QA test can run before declaring it has failed.
  • Wallet for simulator to enable more complex tests – Code refactoring to enable more complicated simulation tests involving Wallet features
  • Log Holding and write script to check data – Debugging tool to allow investigation of holding queues behavior
  • Community Feature Contribution – Show hostname on control panel – Community Contribution: Allow control panel to self identify which node it represents.
  • Make logging reopen log files that have been deleted – Allows for more detailed logging on long running diagnostics.
  • Log debugAPI – Provides diagnostic value for application developers using the debug API.


  • Continue to reduce number of missing messages – Allow the ANO nodes to communicate more effectively, have less bandwidth overhead and less risk of failure due to peer to peer network deficiencies. (Protocol Grant)
  • Sept 17th Release Development complete but was pulled from MainNet due to issue. Will release once issues identified and QA is complete.  
    • Duplicate Message reduction – Performance Improvement by no longer sending messages that were previously received.
    • MMR Improvements
      • reduced the amount of messages asked for that are not needed
      • do not ask for same message multiple times without specific time lag
      • reduce RAM consumption for process lists during reboot
    • SimCtrl EC purchase method – Improvement that allows simulation to more accurate match observed purchase behavior
    • Election Failures – Improvement to the number of false fails from simulation tests when follower/audit nodes are not keeping up
  • Out of order execution of dbstates – Corrects a bug that could cause nodes to panic.
  • Load Balance on Restore – Fixes a stall condition that causes a panic on boot
  • Nil Hash Tracking – Added code to avoid crashing in some cases with pokemon bug
  • Confile File Name Path Issue – Fixes issue where on Windows in some configurations the config files were not found.
  • Factomd sometimes writes out bad savestate files – Update the way savestate files are created to reduce consensus failures under certain conditions.
  • Invalid DBSig Handling – allows old dbsigs to be removed from process list
  • Make system Status Changes available to the control panel – This change makes the sync status of the node be available in the Control Panel for the node even if logging is off which helps diagnose stalls and other sync related issues. The changes also makes faulting and bad messages logging be on by default which helps diagnose failures of a variate of crashes by recording data about bad messages and elections.
  • Need better EC purchase method because the current one doesn’t handle high loads – This is a simulator change to make the load generated during testing behave more like the pattern observed in the real network which improves the likelihood of revealing bugs during testing.
  • WSAPI Tests were dependent on prior WSAPI test having run – Allows execution of individual test instead of just the entire suite.
  • Logging for API level transactions – Assist in debugging interactions with applications
  • ProcessBlocks failure on long-running servers after 5.4.4 release
  • Renenable controlpanelsettings flag after bad merge – Enables logging on nodes that are having issues
  • Coinbase Canceling only takes Majority of Fed nodes, but also counts audits in tally – Allows it to require half of the Authority Set to cancel a payment.
  • Remove Replay Data From Save state – Corrects why SaveDBStateList and cause an increase in memory allocation.
  • Misc logging fixes – This improved the quality and level of detail in some log files and add millisecond time resolution to logged times to facilitate debugging.
  • Community Contribution Bug Fix to correct wallet-balances call in factom-walletd – Allows passwords to be used.  Needed by an exchange.
  • Election in minute 9 cause the leader who was voted out to only follow by blocks and never accept the updated authority set. – Allow a Federated Server which was faulted out during minute 9 to recognize that after the blockchain progresses.
  • Consider blocks loaded from database as valid – Allows leaders to boot to the same height as followers and audit servers
  • GetVirtualServers() could be called in minute 10 and panic – Allows panic in uncommon situations
  • Smarter limits in unmarshalling functions – This is a better fix for a critical bug/attack vector
  • Community Bug Fix: error string out of scope for entryblock panic –  Allows panics to print a message when encountering some types of errors.
  • Reboot issues if dbstate old – This fix is necessary should the network stall for more than an hour.  This allows a network to be rebooted, by allowing the processing of the DBSigs, which set the time frame for a new block.
  • ignore useless message entryblockresponse missingentryblocks – Simplify p2p network interaction by ignoring extraneous discontinued historical messages. These were initially thought to be a severe attack vector against individual nodes, but after fixing were determined to not to be particularly dangerous.
  • factomd panics indicating missing identity entry blocks – Reliably reboot after a partial prior boot

Work Completed

New capabilities

  • Development of 2 new capabilities completed and moved into testing for next release listed above


  • Development of a number of maintenance corrections completed and moved into testing for next release.



Future Plans

Protocol Grant

  • Enable the ability to count votes of standing parties
  • Ongoing Performance Improvements
  • Ongoing Maintenance corrections
  • Continue supporting the protocol and issues that arise

Oracle Grant

  • Improvements to Rate Change to make keeping rate accurate more user friendly
  • Rate change alerts

Anchor Grant

  • Ethereum Anchors
  • A monitoring tool to give better internal UI feedback on anchoring status


POSTED: February 4, 2019 BY Crystal Wiese IN Technical Updates

Crystal Wiese is the Director of Marketing at Factom. Working within the startup tech scene for the past 10 years she has become passionate about taking great ideas and building a strong comprehensive narrative. Graduating from the Art Institute of Portland with a degree in Design Management, Crystal has always seen the value in creating strong ties between the technical, creative and business.