Don't forget about rule #1

-


So I screwed up...



Yes, as hard as that is for me to say. I royally screwed up (with some help from a colleague).


When setting up a new live deployment of your product. ALWAYS SET UP BACKUPS FOR YOUR DB's. Yes, it seems stupid that I should even say this, but for the first time during my 8 year career, I forgot to set up backups. The product went live about a week before and a colleague accidentally restored our blank database backup over the production database... Lots of data loss!










  • NOW WHAT?

  • WHERE DO I START

  • HOW DO I TELL THE CLIENT

  • CAN WE RECOVER?



I decided to Google How to roll back a database restore. You have to try right. No luck. I got served up countless posts about people in the same situation, all being told: "Restore your latest full backup and then restore the balance from your transaction log backup." Great... And then a few unlucky souls that ended up in the same situation with no options. Oops.


So after this, I had to call the client and explain what happened. "Can't you just restore the backups?" I got asked. Uhm... No, we forgot to set up backups.


In this instance, luckily the product only went live a couple of hours before, so there weren't that many transaction we had to recapture (Our point of sale prints out physical slips, we manually recaptured transactions from slips...) The major headache was getting everything set up again, basic configurations, logos, company and customer information. This manual fixing took about 2 hours and took about 2 years off my life expectancy. This was followed by waiting to receive scans of the slips to recapture transactions manually.


On the up side, the client was really impressed by how quickly we got things up and running again. A small consolation for the stress and the neck spasm I developed during the 2 hour ordeal.




To prevent this from happening again, we summarily ran through all of our systems including demo systems and test systems and ensured the backups were in place and running. We identified another issue, we had our demo databases running along side our production databases, way too easy to get access to the production databases then. We need to fix that! We updated our documentation with the steps to set up a new installation to include in big bold letters Set up backups. Lastly we will discuss this during our next retrospective to find solutions and implement safeguards such as restricting access to production databases and making sure that we work with a peer when doing things like this to make sure we don't miss out any crucial steps. 



Perhaps a new found appreciation for peer coding from my side. Maybe Extreme programming (XP) is onto something with always working in pairs.


Location: Westville, 3629, South Africa

Comments

Post a Comment

Popular posts from this blog

Decoding JSON Strings in C# using System.Text.Json

-
Decoding JSON Strings in C# using System.Text.Json Decoding JSON Strings in C# using System.Text.Json In modern software development, exchanging data in the JSON (JavaScript Object Notation) format has become a standard practice due to its simplicity and ease of use. In C#, you can effectively handle JSON data using the built-in System.Text.Json namespace. This blog post will guide you through the process of decoding a JSON string step by step. Prerequisites Before we dive into the coding process, ensure you have a basic understanding of C# programming and Visual Studio (or any other preferred C# IDE). Step 1: Setting Up the Environment To get started, launch your C# development environment and create a new Console Application project. Step 2: Importing the Required Namespace In your C# code, include the necessary namespace for working with JSON data: using System.Text.Json; Step 3: Defining the JSON String Let's say we have the following JSON string that we want to decode: {
Read more

How to serialise a Exception using JsonSerializer.Serialize in C#

-
Image
Serialising Exception Data to JSON in C# Introduction: In software development, handling exceptions is a crucial part of writing robust applications. When exceptions occur, developers often need to log and report them for debugging and monitoring purposes. One way to achieve this is by serialising exception data to JSON, allowing you to easily store and analyse error information. In this blog post, we will explore how to serialise exception data to JSON using the JsonSerializer.Serialize method in C#. We'll use a custom class, ExceptionData , to encapsulate exception information. Creating the ExceptionData Class: To get started, we need to create a custom class called ExceptionData . This class will capture various properties of an exception, such as message, source, stack trace, type name, and optionally inner exceptions. Here's the ExceptionData class: public class ExceptionData { public ExceptionData(Exception exception, bool includeInnerException = tru
Read more

Rubber Duck Debugging

-
Image
  The Origin of Rubber Duck Debugging The concept of Rubber Duck Debugging is rooted in the idea that explaining a problem or code issue to someone else can lead to a deeper understanding and, often, a solution. The story goes that a programmer would carry around a rubber duck and, when faced with a problem, would explain the code or issue to the duck. In the process of articulating the problem, the programmer would often realize the solution without any input from the duck. This practice gained popularity as a form of solo brainstorming and debugging, and it has since become a humorous yet widely recognized method in the software development community. How Rubber Duck Debugging Works The process of Rubber Duck Debugging is simple but surprisingly effective: Grab a Rubber Duck: Obtain a rubber duck or any inanimate object. The choice of the object is not critical; your ability to articulate the problem to it matters. Place the Duck on Your Desk: Set the rubber du
Read more