Outputs and Outcomes
Our focus is to:
- Improve the performance and reliability of the Protocol
- Monitor the exchange rate and make adjustments as needed
- Maintain the Anchor Master Securing the Factom blockchain for the long term
Status and Achievements
May was a successful month for Factom development. There was a successful release of factomd to mainnet and a release to testnet. There were also some good security upgrades that went into effect.
The biggest thing to happen was activation of the 2019 2nd quarter Factom grants. These were successfully deployed across the network by the community starting May 22. These culminated in activation on May 25th.
Another big development was getting the latest bugfix release deployed to the Community Testnet. That started deployment on May 15. That release (Bond) will bring the codebase running on the network up to date with recent developments. The testnet experimentation did highlight some potential weak points that needed investigation. Even with the potential weaknesses that were pointed at in May, Bond was showing to be far more robust than previous versions.
We also saw an uptick in the involvement from non-Factom, Inc developers. Community developer standups continued in May. They were attended by 2 developers from BIF, including Sander and Laurens. Sander continued to provide some good development advances in May.
Matt York and Clay continued to make progress on the dependant holding upgrades, which are showing a lot of promise to improve the capacity of the network. These will help when there are large bursts of traffic on the network.
Another advancement that showed a lot of promise is the optimistic entry writing. This will eliminate a bottleneck in factomd that is exposed under high load. It allows factomd to save entries to the hard drive over the entire 10 minute block period instead of all at once after the block finalizes. Currently, when waiting for finalization, factomd blocks and waits on the hard drive, which can be a slow process when there are a lot of entries. This had a fair amount of local testing in May.
May 2019 also marked the 1 year anniversary of the network restart assist system. This anniversary marked an expiration date, which prompted a security upgrade that went into place in May. Unfortunately, on May 25, the network did experience a pause. The good news from this is that the security upgrades were successful.
There were two Sponsor meetings this month, May 7 and 21. The May 7 meeting was recorded and published here. The sponsors in attendance in May were Domonic Luxford, Nolan Bauer, Nikola Nikolov, and Valentin Ganev (both of Factomatic).
May was a good month in factom development and June is likely to be good as well.
New capabilities that were released
- Community Contribution – Activate grants for 2019-2
- Keep Leaders from getting out of sync on minutes – Allow the network to continue processing entries when Federated servers are booted at different times.
- Rework Entry Syncing with channels and go routines – Entry Syncing has some performance issues. This rework ensures we only handle entries once as we sync (maybe a few requests, but no trashing when an entry has been handled).
- Fix an issue with the write go routine for writing entries – Speed up saving the blockchain on slower machines
- Reveals must wait on their Commits – Handle messages in the correct order to prevent seizing up of a node
- Send Out all messages in the holding queue at a proper rate – Allow messages that are known about to propagate through the network under edge cases.
- Off by one error in DBsigs removed from holding queue – Allow factomd start up more easily under load
- Validate reveals before rebroadcast – Ensures that reveals are validated before they are sent to peers, improves inefficient code that doesn’t improve performance
- Invalid Transactions should not be sent to the network – Limit a node from rebroadcasting invalid transactions to the network.
- Remove simulator load creation from it’s own thread – Allows for a higher load in simulation to stress the system more.
- Refactor sim testing – This ticket rolls up past work into a the latest revision. This enables new types of tests to be written that include adding a node during simulation and also the ability to filter outs specific messages during testing . Additionally this will run simulation tests that previously did not execute on Circle.ci
- Enable ability to retrieve receipts at directory block level – New API that gives much clearer visibility to anchor transactions.
- Split receipts API into receipts and anchors – With ethereum anchoring enabled, we need a way for developers to ask factomd which anchors covered a given transaction.
- Improve simulation by allowing fnodes to be added during tests – Allow us to do new kinds of testing where network topology of sim nodes can change dynamically during testing.
- Optimistic Entry Writes – Distributes the database writes across block time to avoid pauses in processing on block boundaries.
- Downloads dbstates in batches when syncing from the network – Increases efficiency when syncing from the network
- Check for repeat ABlock signatures – Allow the database integrity checker to detect an edge case found in testing where there are multiple signatures with the same pubkey. Not expected in production.
- Add more simulation testing scenarios – Adding more extensive testing for other brain-swap scenarios will help catch more types of backward-incompatible changes from making it into the codebase.
- Community Contribution – add if booted from disk to diagnostics API – allow outside programs to know when the 1st pass has been fully processed from disk.
- Cleanup Failing sim tests – Clean up details of several unit tests and modify some of the tested functions to be more testable.
- Create New HoldingList datastructure – Existing holding queue structure is not as efficient as it could be. This new struct will enable more efficient processing of messages whenever their requisite payments or commit/reveal pair messages become available.
- Refactor Messages to use New Dependent holding – As a preparation for more aggressive refactoring – messages should be refactored to use new Dependent holding queue. Eventually this will replace the older method of holding entirely.
- Null pointer exception is possible checking payments for commits – Don’t panic when timing issues create a null process list
- Comments, Dead Code, Redundant Code – removal of dead code, unneeded comments, blank lines, a redundant call.
- Resuming from sleep did not catch up to mainnet for several hours – Make sure that a node that gets behind catches up with the blockchain when reconnecting to the Internet.
- Community Contribution – CrossBoot replay garbage collection never ends – let Garbage collector use less resources with cross boot replay filter
- Inefficiency with falling time.now() multiple times – use less CPU when handling p2p peers
- Community Contribution – Legibility Improvements Part 1 – Factomd code should be more legible when possible.
- Community Contribution – add config option to set factom-walletd config file path – Allow factom-walletd to use a config file loaded from a configurable path.
- Preemptively save entries to database as they are processed rather than when block is done – Increase performance under high entry load by spreading out expensive database operations over the entire block building period.
- Update Database Integrity Checker – Useful for testing, for development, and as a sanity check for database changes.
- Make holding management be dependency driven instead of scan driven – Replace holding queue with a holding map to increase performance as well as to prepare for sharding
- Refactor Directory Block building – The directory block has no information independent of the components collected while building the next block. So instead of trying to build it as we go, we should build it in dbstatemanager.fixuplinks where we finally have everything we need to create a directory block. This has a side effect of avoiding any caching of hashes that could damage or blow up our state.
- Dependent holding should be periodically cleaned – This is an additional enhancement to new style of message holding. adding a method to periodically expire messages will ensure that stale data gets removed so system will not get backlogged.
- Replay filter of entries is not populated on reboot – Allow a federated server to reject replayed messages which are also rejected by the rest of the network within the first hour of booting
- Factomd thrashes when things are in holding queue – Don’t needlessly burn CPU when there is an item in the holding queue
- Resolve dual ACK bug where under load a leader could produce two ACK for the same height – Don’t stall the network when high traffic causes extraneous messages that break consensus and stop forward progress
- Community Contribution – 2nd pass flagging issue on boot – Remove confusion when starting factomd if the 2nd pass has actually completed when looking at the control panel.
- Logging saves only partial hashes but in some cases the whole hash is required. Add a separate log of all unique full hashes – to help with debugging curious messages on the network, print out the full entry hash to aid tracking down the problem
- EOMs from the future can be dropped in review holding – Don’t delete messages that are likely to be useful prematurely
- Remove code that tosses incoming messages based on holding – Retain messages instead of deleting them shortly before needing them.
- FetchPaidFor can have a nil pointer exception if it encounters an unallocated process list – Don’t panic during edge cases when starting factomd
- Make message sort and process/update state be in separate threads – Reduced deadlock potential (stall) and smooths out message processing.
- Add DBSigs for top DB State to Database – Keeps signatures from the highest block in the database so that we can provide the block to other nodes
- Slow sync with minutes for 10 minutes on boot – Quickly catch up with the federated servers and don’t peg the CPU while doing it.
- The simulation Test other than TestSetupANetork are not run on CircleCI – Expand the number of tests run on Circle CI to help with automated QA
- Raw Object not found on transaction acknowledgement – Fix intermittent timing errors when using the raw API in factomd
- When loading from a save state if the current dblock had FCT/EC transactions the balances were not loaded prior to evaluating the transactions. – Fix timing issue when evaluating FCT and EC balances.
- Factomd Authority JSON marshalling should be usable – correctly export server identity coinbase address and efficiency fields in JSON unmarshaling.
- Clean up logging of messages. Force filename to lowercase – Improves logging so we can better diagnose problems. Increases resolution of timing and standardize on output files.
- Make the version and git version set-able from goland – allow goland debugger to show git commit hash and golang version in factomd and the control panel.
- Inefficient DBState loading while syncing from the network – Better handle downloading the first pass of getting the blockchain
- Logging improvements from old revisions – Add millisecond to times for better analysis. generally better log behavior. No crash on nil state logging.
- Wallet database reinitializes – automatically clear the cache of factom-walletd when detected a new local test-blockchain.
- Sort auth set only on block boundaries – Bug fix related to elections to allow them to resolve properly in some elections.
- Local EOM or DBSig could be follower instead of leader executed executed in rare cases and needed to be added to holding – keep around internal EOM messages even if in an odd edge condition.
- Extra DBSTATE messages were generated when restoring from a savestate – Don’t create extraneous DBstates when loading from a savestate file.
- Testing API to filter inbound and outbound messages – Add a message filter and an API to set it so that QA can simulate nodes disappearing from the network.
- Fast track chain commit/reveals – Prioritize Chain creation over Entry creation to help prevent backed up queues
- Refine some of the unit test code – Add unit tests for local commits and local wallet simulations
- Fix a pokemon instance involving MessageBase – Catch a new style of Pokemon bug found on the testnet with MessageBase
- Ack holding – Fixes bug that could cause useless elections immediately after boot.
- Investigate 6.2.2 issues on mainnet – Need to unblock release of 6.2.2 – either we need to fully understand the misbehavior or diagnose and correct a bug when running alongside 6.2.0
- Missing Message Requests are tossed if the ask has an all 0 ID – Don’t ignore peers who added the identity of all zeros to their config file.
- Log peer behavior. Fix double p2p close – We have seen odd behavior on restoring from sleep. To better understand it we are adding logging for peer connection behavior. Also, stop disconnecting from a peer twice in a row.
- Factomd not reading from CustomSeedsURL – In a custom isolated test network, allow followers to find peers with the CustomSeedsURL
- Gossip Improvements – Reduce the redundant network traffic caused by Gossip protocol
- Ongoing Performance Improvements
- Ongoing Maintenance corrections
- Continue supporting the protocol and issues that arise
- Ongoing Maintenance corrections
- Ongoing Maintenance corrections