Skip to content

Commit

Permalink
Add header image, alt text
Browse files Browse the repository at this point in the history
  • Loading branch information
pierrelefevre committed Mar 22, 2024
1 parent 20faf1d commit 50ac427
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions hugo/content/News/2024-03-22.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,12 @@ title: "Incident Report: Major March 2024 Outage"

In March 2024, kthcloud experienced a significant outage that disrupted our services for several weeks. This detailed report outlines the incident, including the timeline, root cause, and the steps we're taking to ensure this does not happen again.

<img src="../../images/blog/server-fire.webp" alt="servers on fire" /><br/>
<small>Obligatory AI generated image</small>

---


## Preface
To our valued users, we extend our deepest apologies for the inconvenience and challenges brought about by the recent outage. Your trust in our service is something we deeply value and we understand the impact this disruption has had on your work and projects. We are grateful for your patience, understanding and support during this time, and we appreciate the feedback we have received.

Expand All @@ -25,7 +29,7 @@ Friday night was fast approaching, and we decided to leave the database in its c

>*Narrator: little did they know, things were about to get a whole lot worse*
<img src="../../images/blog/discord-monday.png" /><br/>
<img src="../../images/blog/discord-monday.png" alt="discord chat log warning the server room is too hot" /><br/>
<small>Figure 1, kthcloud monday morning chat log</small><br/>

Monday March 4th, we arrived to find most servers in Flemingsberg had shut down.
Expand All @@ -37,7 +41,7 @@ We reported the issue immediately, and were provided a temporary solution in the
On March 15th, were able to identify the root cause, a corrupted encryption key, and fix it using a backup. We then began the process of restoring services, which took several days due to the complexity of the infrastructure.

## What was impacted
<img src="../../images/blog/ccc-infra.png" /><br/>
<img src="../../images/blog/ccc-infra.png" alt="diagram of kthcloud, showing everything is dependant on cloudstack" /><br/>
<small>Figure 2, kthcloud infrastructure diagram</small><br/>


Expand Down
Binary file added hugo/content/images/blog/server-fire.webp
Binary file not shown.

0 comments on commit 50ac427

Please sign in to comment.