Issue Summary:

Our users experienced an Nginx server outage on February 4th, 2023, between 9:00 AM and 11:00 AM GMT, which caused poor service and problems accessing our website. The problem impacted almost 80% of our users. It was discovered that an incorrectly set up Nginx configuration was the issue's primary root cause of this outage.

Timeline:

9:00 AM GMT - I discovered the problem after getting a monitoring alert about the Nginx server.

9:15 AM GMT - Further analysis revealed that the Nginx server was slow in responding to queries, which was slowing down the service.

9:30 AM GMT - I went ahead and looked at the network configuration at first since it was initially thought that the problem might be with network connectivity.

10:00 AM GMT - The issue was escalated to senior engineers and mentors for further assistance in ascertaining the core cause of the outage.

10:15 AM GMT - The server logs were subsequently pulled up as I kept looking for the core cause after ruling out network problems.

10:45 AM GMT - The logs pointed to a flawed Nginx configuration and I immediately began working to fix it.

11:00 AM GMT - The problem was fixed and the Nginx server was back up and running, allowing our users to resume normal service.

Root cause and resolution:

An incorrectly set up Nginx configuration was the primary contributor to the problem. Due to improper request routing to the relevant backend servers, the Nginx server slowed down and stopped responding to queries. By altering the Nginx settings and correctly routing requests to the backend servers, the problem was fixed.

Corrective and preventative measures:

The following actions will be taken in the future to avoid a recurrence of similar events:

• Check and verify Nginx configurations frequently to guarantee a correct setup

• Utilize monitoring for Nginx server setups to spot any errors immediately.

• Create a procedure for inspecting Nginx configurations before deploying them.

• Engineers should receive more instruction on Nginx setups and recommended practices.

Tasks to address the issue:

• Examine the Nginx settings on each server.

• Monitor Nginx configuration implementation

• Create a procedure for verifying Nginx configurations before deploying them.

• Set up Nginx setups training sessions for engineers and other technical personnel.

Finally, we would like to express our regret for any trouble this outage may have caused our users. To give our users the very best experience, we are dedicated to constantly upgrading our systems and procedures. On a lighter note, our Nginx server has finally received a much needed makeover!