@everyone from @Indiana_Gems. The blockchain is back online and working.

22 Aug 2023, 10:28

@everyone from @Indiana_Gems The blockchain is back online and working. That said, we want to provide you with a clear understanding of the recent disruption that led to our blockchain services being temporarily offline. We under promised in our previous announcement to ensure that we provided the worst case scenario. The issue arose due to a delay in account credits from One2Cloud, causing a discrepancy in our account balance. It's important to note that Alibaba has a strict 24-hour timeframe for addressing such credit shortfalls, after which their automated system suspends instances for security reasons. After discussing the matter with Alibaba, we were able to negotiate a three-day extension instead of the standard 24 hours due to our positive working relationship. During this period, One2Cloud rectified the credit situation, bringing us back to normal operation. To prevent similar occurrences, One2Cloud has now implemented enhanced credit threshold alerts and added more contact points for quicker notifications. In addition, the Vultr backup instance, which is used as a last-resort measure, faced challenges due to a lack of synchronisation with the blockchain. Estimating the time needed to synchronise accurately was difficult in this case which complicated matters further. To prevent future disruptions, we've taken the following actions: 1. Reactivated the Alibaba Active Genesis Block. 2. Set up daily synchronization for the Alibaba Backup Genesis Block with appropriate settings. 3. Migrated half the validators to Tencent Cloud to have multi vendor redundancy. 4. Implemented weekly synchronization for the Tencent Backup Genesis Block on a newly created instance, replacing the Vultr instance. We are also building a strong working relationship with Tencent Cloud so we are quite happy with this change. 5. Retired the Vultr Backup Genesis Block and migrated to a more robust Tencent Cloud vendor. 6. Planned the deployment of a Selfhosting Validators (Community Owned) scheduled for the future. Framework for this has already been completed as this was already part of the plan. 7. Introduced Dev house On-Premises Backup Genesis Block, aligning with specified configurations. Why is the genesis server important "The genesis server is the first peer (connection) in any blockchain-based protocol. It is the basis on which additional validators are added to form a blockchain network." By moving some of the validator clusters from Alibaba to Tencent Cloud, we have multi-vendor redundancy to make sure that 2/3 of our validator network don't go offline meaning the blockchain will remain online. Also by adding in the framework for self-hosting validators this will help safeguard the blockchain going down. Rest assured, we remain committed to strengthening our operations for greater resilience and efficiency. Warm regards, The Team