Grant Update February 2019

Technical Updates | March 9, 2019
Home » Company » Blog » Grant Update February 2019

Outputs and Outcomes

Our focus is to:

  • Improve the performance and reliability of the Protocol
  • Provide ability to monitor the exchange rate and make adjustments as needed
  • Maintain the Anchor Master Securing the Factom blockchain in the long term

Grants: Protocol Grant 009  |  Oracle Grant – 007   | Anchor Grant 008   

Status and Achievements

February was a taxing month for Factom development.  February started with the attempted release of V6.1.1.  This release contained several months of accumulated updates which would have fixed many outstanding bugs in the core software.  It did not help that Factom, Inc was still operating in the reduced development effort period throughout February. This had been laid out in the Inter-grant Continuation plan earlier in December 2018.  There were two major issues that prevented wide deployment of 6.1.1. The first issue was known, but was considered tolerable, as the benefits of the upgrade exceeded the drawbacks.

Deployment on the Community Testnet showed that 6.1.1 was unworkable as a general release.  This turned out to be a great success, as the Testnet demonstrated a new failure mode with brainswapping (seamless server upgrading).  The Testnet is far more complex than the simpler testing environments normally used. It saved the Mainnet from what would have been a tumultuous series of crashes.

The limited development bandwidth in February was spent fixing the two blocker bugs.

  • The second pass stall bug took a couple different fixes to get fully working again.  By the end of Feb the bug was solved and was going through internal testing.
  • The brainswap bug prompted several new test suites to be setup to test swapping between different versions.  One of the tests is an automatic test which gets run anytime code is checked in.  A more complex test that compares across different versions was created as well. Thirdly, integration tests that operated on different docker containers were developed as well.  By the end of February and a couple different attempts, the bug was squashed. The Kraft release (v6.2.1) now brainswaps are compatible with previous versions.

The next version of Factomd (6.2.1) contains more changes to the codebase than Github displays (containing 342 changed files).  It is an extensive update to say the least.

The Grants release (v6.2.0) was also published and was deployed in February.  This brought the first quarter 2019 grants into reality. Thanks to Niels Klomp for authoring the code changes.

Sadly, during deployment of 6.2.0, it did not go smoothly.  A failed brainswap on Feb 22, 2019 stalled the network.  Luckily the stall happened at a point in time where most of the ANO network was awake and available.  The restart system had atrophied over the past year since it was set up. The Feb 22 stall showed that there was a lot of cleanup work that needed.  In February that cleanup process began. Part of the cleanup will be to work with ANOs to configure firewalls as part of the server changeover that began in 2018.

There were two sponsor meetings in February, attended by Sponsors Dominic Luxford, Nolan Bauer, Valentin Ganev (Factomatic).  The Feb 13, 2018 meeting was recorded and published.

February we also welcomed development activity from several new developers:

  • Who Soup from Factomize
  • Thomas Meier from Layertech
  • Niels Klomp from BIF

Factomd releases are now referred to by paper names.  Earlier they were referred to by version numbers, which changed during the development process.  They were also referred to by anticipated dates, which tended to slip. The next 4 releases after 6.1.1 are Butter, Kraft, Parchment, and Xuan.  Back in 2015, Factom was referred to as blank paper.  This new naming convention embraces that history.

Completed

New capabilities

  • New Network Diagnostic API (Protocol Grant) – Gives visibility into aspects of nodes to better understand their performance and what could have caused a stall.
  • Rate Change Alerts (Oracle Grant) – Keeps the exchange rate current with the market to ensure they stay near the intended constant rate.
  • Sept 17th Release Feature development complete but was pulled from MainNet due to issue. Will release once issues identified and QA is complete.  
    • Performance Graphing Tools – Collecting data is complete for required for new internal graphing tools that allow QA a better understanding of how changes affect performance
    • Timeouts Defined Merged – Improvement to simulation tests so when they fail they exit
    • Calculate System Status Always – Populated Control Panel Summary\Syncing by default
    • Upgrade Golang to 1.11 – Keep up with Golang development
  • Coinbase delay for Testnet – Extend delay between a coinbase to all more time to coordinate a test of the coinbase cancel on the testnet.  
  • Diagnostic Graphs – Detailed view of the internal system performance
  • Simulation tests in factomd_test.go should support timeouts defined in blocks – Adds ability to limit the amount of time a simulation QA test can run before declaring it has failed.
  • Modify factomd API call ‘current-minute’ to also return ‘current-block-height’ – Allows you to make a single API call and returns the minute and block height together to make testing more reliable
  • Upgrade to golang 1.11 – Continue standards of staying current version
  • Community Contribution for Integration of CORS code into Factomd – Allows Javascript browsers to use the data that is provided by Factomd.
  • Encrypting the Factom-walletd Database – Exchange Requests that requires password security when using wallet
  • SimCtrl API – Added SimCtl API to the debug API to allow scripting to perform testing procedures
  • Simulation tests in factomd_test.go should support timeouts defined in blocks – Adds ability to limit the amount of time a simulation QA test can run before declaring it has failed.
  • Wallet for simulator to enable more complex tests – Code refactoring to enable more complicated simulation tests involving Wallet features
  • Log Holding and write script to check data – Debugging tool to allow investigation of holding queues behavior
  • Community Feature Contribution – Show hostname on control panel – Community Contribution: Allow control panel to self identify which node it represents.
  • Make logging reopen log files that have been deleted – Allows for more detailed logging on long running diagnostics.
  • Log debugAPI – Provides diagnostic value for application developers using the debug API.

Maintenance

  • Continue to reduce number of missing messages – Allow the ANO nodes to communicate more effectively, have less bandwidth overhead and less risk of failure due to peer to peer network deficiencies. (Protocol Grant)
  • Sept 17th Release Development complete but was pulled from MainNet due to issue. Will release once issues identified and QA is complete.  
    • Duplicate Message reduction – Performance Improvement by no longer sending messages that were previously received.
    • MMR Improvements
      • reduced the amount of messages asked for that are not needed
      • do not ask for same message multiple times without specific time lag
      • reduce RAM consumption for process lists during reboot
    • SimCtrl EC purchase method – Improvement that allows simulation to more accurate match observed purchase behavior
    • Election Failures – Improvement to the number of false fails from simulation tests when follower/audit nodes are not keeping up
  • Out of order execution of dbstates – Corrects a bug that could cause nodes to panic.
  • Nil Hash Tracking – Added code to avoid crashing in some cases with pokemon bug
  • Confile File Name Path Issue – Fixes issue where on Windows in some configurations the config files were not found.
  • Factomd sometimes writes out bad savestate files – Update the way savestate files are created to reduce consensus failures under certain conditions.
  • Print system status summary to a log file so QA can use the performance graphing tools – Help QA process by creating a log file for use with the performance graphing tools.
  • Invalid DBSig Handling – allows old dbsigs to be removed from process list
  • Make system Status Changes available to the control panel – This change makes the sync status of the node be available in the Control Panel for the node even if logging is off which helps diagnose stalls and other sync related issues. The changes also makes faulting and bad messages logging be on by default which helps diagnose failures of a variate of crashes by recording data about bad messages and elections.
  • Need better EC purchase method because the current one doesn’t handle high loads – This is a simulator change to make the load generated during testing behave more like the pattern observed in the real network which improves the likelihood of revealing bugs during testing.
  • WSAPI Tests were dependent on prior WSAPI test having run – Allows execution of individual test instead of just the entire suite.
  • Logging for API level transactions – Assist in debugging interactions with applications
  • ProcessBlocks failure on long-running servers after 5.4.4 release
  • Move flags moved by merge into init – Properly read the home directory
  • Renenable controlpanelsettings flag after bad merge – Enables logging on nodes that are having issues
  • Coinbase Canceling only takes Majority of Fed nodes, but also counts audits in tally – Allows it to require half of the Authority Set to cancel a payment.
  • Remove Replay Data From Save state – Corrects why SaveDBStateList and Restore.save() cause an increase in memory allocation.
  • Misc logging fixes – This improved the quality and level of detail in some log files and add millisecond time resolution to logged times to facilitate debugging.
  • Community Contribution Bug Fix to correct wallet-balances call in factom-walletd – Allows passwords to be used.  Needed by an exchange.
  • Cleanup attack/bug vector in unmarshaling – Hardens the protocol against error and malicious attack.
  • Election in minute 9 cause the leader who was voted out to only follow by blocks and never accept the updated authority set. – Allow a Federated Server which was faulted out during minute 9 to recognize that after the blockchain progresses.
  • Consider blocks loaded from database as valid – Allows leaders to boot to the same height as followers and audit servers
  • GetVirtualServers() could be called in minute 10 and panic – Allows panic in uncommon situations
  • Smarter limits in unmarshalling functions – This is a better fix for a critical bug/attack vector
  • Community Bug Fix: error string out of scope for entryblock panic –  Allows panics to print a message when encountering some types of errors.
  • Reboot issues if dbstate old – This fix is necessary should the network stall for more than an hour.  This allows a network to be rebooted, by allowing the processing of the DBSigs, which set the time frame for a new block.
  • ignore useless message entryblockresponse missingentryblocks – Simplify p2p network interaction by ignoring extraneous discontinued historical messages. These were initially thought to be a severe attack vector against individual nodes, but after fixing were determined to not to be particularly dangerous.
  • factomd panics indicating missing identity entry blocks – Reliably reboot after a partial prior boot
  • Clone_entire_state – Allow blockchain to continue downloading when stopped partway through
  • NoReactivatingOldIdentityKeys – Close attack vector in Identity management which allows a 3rd party to replay an old identity key update

Awareness

Pending Release

New capabilities pending in next release

  • Syncing by minutes requires a block download – DBSig messages should be accepted when first seen
  • Optimizations to speed up loading a factomd database – Assorted optimizations for 2nd pass downloading and partial fix for a stalled 2nd pass
  • Update to golang 1.12 – factomd will keep up with the latest golang release: 1.12.
  • Add Brain Swap fixes and added simulation tests – Enhances the testing automation around ‘brain swap’ identity migration operations. This is a common technique used by all ANO’s when a new version of factomd is released. This testing allows us to more easily detect backward-incompatible changes to the protocol.
  • Expose identity functions through factom-walletd API – We are adding this functionality for two primary reasons:

– The Voting Grant being worked on by community members. They are waiting on Factom Inc to build an identity system that accommodates “Standing Parties” and the staking of FCT and EC addresses.

– FPT’s proof-of-concept application. Their team was struggling to quickly understand how to implement Identities with the bare building blocks of Factom (entries and chains), and since we anticipate this to be a fairly common hurdle, it would be easier if we created a standard API for Identity creation and management.

  • Add ability for factom-walletd to store identity keys – We are adding this functionality for two primary reasons:

– The Voting Grant being worked on by community members. They are waiting on Factom Inc to build an identity system that accommodates “Standing Parties” and the staking of FCT and EC addresses.

– FPT’s proof-of-concept application. Their team was struggling to quickly understand how to implement Identities with the bare building blocks of Factom (entries and chains), and since we anticipate this to be a fairly common hurdle, it would be easier if we created a standard API for Identity creation and management.

  • Factom-walletd configuration fails with http authorization – factom-walletd should not return an error about “Invalid Params”
  • Encrypt database for factom-walletd – protect the factom-walletd private key database on the disk
  • Add Identity commands to factom-cli – Allow Identities and Identity keys to be easily managed from factom cli rather than just through curl requests.
  • Support CORS Domains for WalledD – allow a javascript web browser to access the walletd API if the wallet owner set it up as such.
  • Community Contribution – https support and speed improvements – factom lib side of FD-808 – This allows support of https in the enterprise wallet. It also speeds up refreshing for new transactions.
  • Register a SLIP-44 number for the identities – Use a reserved derivation type for identities for cross-compatibility.
  • Add new display options to factom-cli – Improves cli usefulness for scripting factom actions. This would be useful for us for QA and for developers using the factom-cli

Maintenance\QA\Debugging pending in next release

  • Message filtering issue caused excess holding and fail to transition to following by minutes. – Fix the message filters which were stopping factomd frown downloading the 2nd pass when catching up with the blockchain.
  • Factomd slows down 2nd pass with full holding map – Speeds up 2nd pass download when syncing the blockchain from the network
  • Brain swap activation height is off by 1 – Activate the brainswapped identity at the expected height.
  • Stop continually evaluating entries from the last block – reduce database traffic when running and loading a database with lots of entries. Also simplify how entries are selected for downloading.
  • Ack message dbheight exploit – Investigating this issue allows us to discover ways to mitigate the risk and/or patch factomd to eliminate the threat.
  • Need user friendly error for method “identity-keys-at-height – Return a meaningful error message when there are problems with tracking an on-chain identity.
  • The api identity-keys-at-height returns server error for heights before chain creation – Return a meaningful error message when asking about an identity at a time before it was created.
  • Disallow re-activation of a retired Identity Key – Close attack vector in Identity management which allows a 3rd party to replay an old identity key update
  • Misc. Identity Fixes – update identity spec to be better handled by some 3rd party JSON libraries.  Also, add more restrictions to a key’s validity in identities.

In Progress

New capabilities

  • Standing Parties (Protocol Grant)
    • Ensures no other standing party can lay claim to the same standing
    • Cannot be intercepted by an attacker before secured in the Standing Chain
    • Can be easily tallied to calculate the standing behind a decision in the Factom protocol
  • FactomD Recognizes Voting (Protocol Grant)  
    • The tracking of identities which can be used to track the standing parties and delegate votes to provide support for ANOs, Grants, etc.
    • Provide APIs to expose information about staked FCT to allow the 3rd parties to do calculations based on this data.
    • Provide APIs to expose information about staked ECs to allow the 3rd parties to do calculations based on this data.
  • Enable ability to retrieve receipts at directory block level – New API that gives much clearer visibility to anchor transactions.
  • Factomd V2 API returns Eth txid – With ethereum anchoring enabled, we need a way for developers to ask factomd which anchors covered a given transaction.
  • Update Database Integrity Checker – Useful for testing, for development, and as a sanity check for database changes.
  • Downloads dbstates in batches when syncing from the network – Increases efficiency when syncing from the network
  • Check for repeat ABlock signatures – Allow the database integrity checker to detect an edge case found in testing where there are multiple signatures with the same pubkey.  Not expected in production.
  • Add more brainswap testing scenarios – Adding more extensive testing for other brain-swap scenarios will help catch more types of backward-incompatible changes from making it into the codebase.
  • Community Contribution – CrossBoot replay garbage collection never ends – let Garbage collector use less resources with cross boot replay filter
  • Inefficiency with falling time.now() multiple times – use less CPU when handling p2p peers
  • Community Contribution – Legibility Improvements Part 1 – Factomd code should be more legible when possible.

Maintenance\QA\Debugging

  • Add DBSigs for top DB State to Database – Keeps signatures from the highest block in the database so that we can provide the block to other nodes
  • The simulation Test other than TestSetupANetork are not run on CircleCI – Expand the number of tests run on Circle CI to help with automated QA
  • Raw Object not found on transaction acknowledgement – Fix intermittent timing errors when using the raw API in factomd
  • When loading from a save state if the current dblock had FCT/EC transactions the balances were not loaded prior to evaluating the transactions. – Fix timing issue when evaluating FCT and EC balances.
  • Factomd Authority JSON marshalling should be usable – correctly export server identity coinbase address and efficiency fields in JSON unmarshaling.
  • Clean up logging of messages. Force filename to lower case.
  • Make the version and git version set-able from goland – correctly export server identity coinbase address and efficiency fields in JSON unmarshaling.
  • Inefficient DBState loading while syncing from the network – Better handle downloading the first pass of getting the blockchain
  • Logging improvements from old revisions – Add millisecond to times for better analysis. generally better log behavior. No crash on nil state logging.
  • Wallet database reinitializes – automatically clear the cache of factom-walletd when detected a new local test-blockchain.
  • Make factomd stall if elections try to replace more than half the federated served in a single block – Fixes a potential vulnerability in leader set where a bad actor could cause a coup and take over the blockchain
  • Sort auth set only on block boundaries – Bug fix related to elections to allow them to resolve properly in some elections.
  • Local EOM or DBSig could be follower instead of leader executed  executed in rare cases and needed to be added to holding – keep around internal EOM messages even if in an odd edge condition.
  • Extra DBSTATE messages were generated when restoring from a savesate – Don’t create extraneous DBstates when loading from a savestate file.
  • Testing API to filter inbound and outbound messages – Add a message filter and an API to set it so that QA can simulate nodes disappearing from the network
  • Factomd not reading from CustomSeedsURL – In a custom isolated test network, allow followers to find peers with the CustomSeedsURL
  • Fast track chain commit/reveals – Prioritize Chain creation over Entry creation to help prevent backed up queues
  • Make holding management be dependency driven instead of scan driven – Replace holding queue with a holding map to increase performance as well as to prepare for sharding
  • Couple tweaks done to 706 that didn’t get integrated.

Future Plans

Protocol Grant

  • Enable the ability to count votes of standing parties
  • Ongoing Performance Improvements
  • Ongoing Maintenance corrections
  • Continue supporting the protocol and issues that arise

Oracle Grant

  • Improvements to Rate Change to make keeping rate accurate more user friendly
  • Rate change alerts

Anchor Grant

  • Ethereum Anchors
  • A monitoring tool to give better internal UI feedback on anchoring status

Awareness

POSTED: March 9, 2019 BY Kevin Casper IN Technical Updates
ABOUT THE AUTHOR

Factom Inc's community manager, carrying a load of marketing operations experience from the enterprise tech and video gaming industries.