Status and Achievements
July was a productive month for factomd development. The Parchment version was finalized and promoted to the Master branch. It was the accumulation of many bugfixes and scaling improvements. It showed significant capacity on Testnet and showed a lot of promise. Parchment was still undergoing testing at the end of July.
There were 3 sponsor meetings in July 2019. Since every other guide meeting is recorded and published, this means there are two that are two recordings that were posted. The recording was from the July 3rd and July 31st meetings. The sponsors in attendance in July were Nolan Bauer, Valentin Ganev, and Nikola Nikolov (both of Factomatic).
July 3rd Sponsor Meeting: https://youtu.be/v5gyEmKBQRs
July 31st Sponsor Meeting: https://youtu.be/AKh-XWTY_9k
Ethereum Anchoring was also publicized during July. This added feature of the Factom protocol will help protect the protocol in the future and will assist in integration with Ethereum smart contracts. Info on the Ethereum anchor is available here: https://www.factom.com/company/blog/anchors-away/
The main event in July in Factom core development was the release of v6.3.3 which went by the codename Parchment. Parchment was a bugfix release that was built on top of Bond (v6.3.2). It had several bugfixes that were discovered and repaired. Parchment 239 commits on top of Bond, with 147 files changed. https://github.com/FactomProject/factomd/compare/v6.3.2…v6.3.3-rc1
One of the big fixes that happened was that the state was being built incorrectly. Due to random differences in timing when factomd started up, different nodes could come to different conclusions about what balances different factoid addresses held. When the different Federated servers would see a transaction that they would disagree on, then only some of them would recognize it as valid. If one of the federated servers tried to include a transaction into the blockchain which the majority disagreed with, then that server would be faulted out. A survey of the nodes in mid July shortly after the bug was identified showed that a small minority of Authority nodes had this problem. The fix was included in the Parchment release, and was not released as an emergency fix.
Another update that was released with Parchment was Dependent Holding. This update gave factomd more intelligence as to the order of messages arriving into the node. This will allow the node to be able to handle more traffic, as it is able to more easily find the next item in line since they are pre-classified as they arrive. This feature also is a precursor to the refactor, separating out different concerns so that the code can be threaded more easily.
Another optimization that was released with Parchment was Optimistic Entry Writing. This feature also would increase capacity while the network is under load. During testing, a bottleneck was found where entries were written to the database at the end of the 10 minute block period. This had the effect of delaying a node at the end of the block, as it waits for the database to finish writing. It could cause a lag during high traffic periods. Saving the entries continuously over the 10 minute period eliminates this wait period and will allow higher traffic on the network.
Parchment also included updates from community developers Who Soup, Sander Postma, and Thomas Meier.
Another community developer effort which Factom, Inc is supporting through the development grant is the Livefeed API. The Sphereon team is working on that. Factom, Inc. is supporting that through design reviews, etc.
Also, a new developer has joined in the development process. Michael Lam had started joining the core dev standups in July. He also started with some pull requests in July.
Parchment was tested successfully and passed load testing on the Community Testnet. The ChockaBlock tool provided by the group Luciap was used to stress test the testnet. The Federated servers were able to keep up with the load provided by the tool.
After passing testing on the Testnet Parchment was released by promoting to the Master branch of the factomd repository, signifying it was the default version to run. The deployment to the Authority servers typically takes a bit longer after the code is released. The installation process had begun experimentally with a few servers at the end of July.
The follow on version of factomd which would contain bugfixes and improvements over Parchment is the Xuan release. Its development began in July. The code name Xuan was chosen by longtime community member Laup.
New capabilities and maintenance issues that were released
- First dbstate on boot is processed twice
- Stop double processing blocks.
- Off by One error in savestate can cause cause identities and coinbase errors
- Update database generation utility
- Depending holding items whose dependencies are met by the content of a DBState are not released
- don’t leak memory with Dependent Holding upgrades
- DBStateCatchup getNextConsecutiveMissing has off by 1 error
- Prevent asking for dbstates beyond the highest known block
- Community Contribution – Runtimelog doesn’t respect its own enabled setting
- Performance optimization when not debugging
- panic on FD-1053_bond_parchment_merge when follower is connected to mainnet
- Allow factomd to load from the database without a race condition
- Fix circle ci simtests for factomd
- More consistent testing results when using CircleCI
- Reposting eoms that never get executed builds up [EOM Poison]
- This is mainly exercised in unit tests with shorted block times than mainnet and would resolve some automated testing failures.
- Parchment + Bond Merge Fix Unit Tests + Minor Bugfixes
- allow further development using unit tests that had failed in the merge of Bond and Parchment
- DBStateCatchupList is not thread safe
- Don’t suffer a race condition when downloading the blockchain
- Improve scripts for dealing with logs from QA
- Allow better debugging when QA discovers problems
- Community Contribution – Add configuration ability to the peer connection limit
- Allow users to increase or decrease the number of peers they connect to.
- Null pointer exception is possible checking payments for commits
- Don’t panic when timing issues create a null process list
- Missing Message Requests are tossed if the ask has an all 0 ID.
- Don’t ignore peers who added the identity of all zeros to their config file.
- SimControl Dropping of messages drops on output before broadcast.
- allow testing to continue in a simulated lossy network environment
- Dependent holding should be periodically cleaned
- This is an additional enhancement to new style of message holding. adding a method to periodically expire messages will ensure that stale data gets removed so system will not get backlogged.
- Move Filter API to Debug
- Limit scope of testing tools on the API
- Ack holding
- Fixes bug that could cause useless elections immediately after boot.
- It looks like a node is asking for DBStates it clearly has.
- Improve download of the blockchain
- Community Contribution – Use different muxes for various web services
- Don’t overlap the various web services factomd provides
- fix a pokemon instance involving MessageBase
- Catch a new style of Pokemon bug found on the testnet with MessageBase
- Community Contribution – finish Election Sync fix
- more effectively clean up sync message handling
- Refactor sim testing
- This ticket rolls up past work into a the latest revision. This enables new types of tests to be written that include adding a node during simulation and also the ability to filter outs specific messages during testing . Additionally this will run simulation tests that previously did not execute on Circle.ci
- Add ability to set a path for the log file output by prepending it to the debugregex.
- Increase testing ability by specifying where logfiles can be collected
- Community Contribution – add if booted from disk to diagnostics API
- allow outside programs to know when the 1st pass has been fully processed from disk.
- Make Tests for FilterAPI
- Make sure that regression tests can be run without race conditions
- Community Contribution – close local TCP handler when connection drops
- Better handle error conditions with transient p2p connections
- Community Contribution – Brainswap an Audit Server
- Don’t panic when brain swapping in an Audit server.
- Add more simulation testing scenarios
- Adding more extensive testing for other brain-swap scenarios will help catch more types of backward-incompatible changes from making it into the codebase.
- Refine some of the unit test code
- Add unit tests for local commits and local wallet simulations
- Community Contribution – Legibility Improvements Part 1
- Factomd code should be more legible when possible
- make holding management be dependency driven instead of scan driven.
- Change holding queue to understand orders of operation to increase performance as well as to prepare for sharding
- inefficiency with falling time.now() multiple times
- use less CPU when handling p2p peers
- Community Contribution – CrossBoot replay garbage collection never ends
- let Garbage collector use less resources with cross boot replay filter
- Downloads dbstates in batches when syncing from the network
- Increases efficiency when syncing from the network
- Optimistic Entry Writes
- Distributes the database writes across block time to avoid pauses in processing on block boundaries.
- Inefficient DBState loading while syncing from the network
- Better handle downloading the first pass of getting the blockchain
- repair balance checking tool
- Corrects Issue with balance checking tool to allow use tool to check if database is corrupted
- Make the version and git version set-able from goland
- allow goland debugger to show git commit hash and golang version in factomd and the control panel
- Factomd Authority JSON marshalling should be usable
- correctly export server identity coinbase address and efficiency fields in JSON unmarshaling.
- Split receipts API into receipts and anchors
- With ethereum anchoring enabled, we need a way for developers to ask factomd which anchors covered a given transaction.
- Enable ability to retrieve receipts at the directory block level
- New API that gives much clearer visibility to anchor transactions.
- Ongoing Performance Improvements
- Ongoing Maintenance corrections
- Continue supporting the protocol and issues that arise
- Ongoing Maintenance corrections
- Ongoing Maintenance corrections
- Cambridge International Symposium on Economic Crime, Sept 1st – 8th, Cambridge England
- Smart City Expo Atlanta, Sept 11th-13th, Atlanta, GA
- Digital Mortgage Conference, Sept 23rd – 24th, Las Vegas, NV
- Annual Mortgage Bankers Conference, Oct 28 – 30th , Austin, Tx
- Blockchain Conference North America, Nov 13th -14th, Santa Clara, CA