Incidents reported on status page for Greev.eu https://status.greev.eu/ https://d1lppblt9t2x15.cloudfront.net/logos/9ad19587abf1914366d129e1cebb0159.png https://status.greev.eu/ en https://status.greev.eu/incident/328081 Sat, 17 Feb 2024 08:15:00 -0000 https://status.greev.eu/incident/328081#5cb19871b93e15d19770dcf383d03b00eee97136eb37b6f03cad6e1b8f778e7f Today, at around 04:30 UTC, the Server started to no longer allow players to join. This issue persisted until around 08:15 UTC. The issues started at around 23:25 UTC on the 16/02/2023 by one server creating hundreds of gigabytes of logs. This on its own wouldn't have been an issue, but as it started late at night and the alerts for high disk storage wouldn't fire until 03:40 UTC. A full disk can be a big issue for the server, especially if it’s using every last byte. Now, what exactly happened on Greev that caused a full disk to result in the whole network being unavailable? With the disk being full at around 04:00 UTC, it created some more errors in the console but didn't result in too many issues. The real problems started at 04:30 UTC, which is the time many of our non-dynamic servers, like KnockPVP or FastBridge, started to do their daily restarts. Together with their daily restarts, the servers also reset their worlds, some plugins, and, most importantly, Spigot tries to store the current settings again. With there being no storage left on the disk, they couldn't just save their settings, which resulted in corrupted files. In addition to that, plugins couldn't even connect to the database anymore; this was also an issue of the file system being full with no more bytes to spare. As connections on Linux are handled as "files" no new connections / "files" could be created, causing even more issues. These corrupted files, together with there being issues with loading plugins after they couldn't connect to the database anymore, resulted in the servers not starting up anymore, and therefore players were unable to play any more games. And with the lobby restarting at 04:30 as well, the whole network went dark. I got to know about this issue at around 07:00 UTC, just after I woke up. When I started to investigate what the issue was, I quickly noticed that just restarting the affected servers resulted in no fix. So I dug deeper and found out that no more space was available, quickly finding the log file with over 500 GB of logs. Subsequently to deleting the file, I quickly discovered that there were still issues on some servers. These servers were affected by corrupted files from the plugins and the servers. I had to load some files from backups, which I could do quite quickly. First, I thought it was just some plugin files with issues, but after finding out that some of the servers were still having issues after booting up, I had to do some more digging. On some of the servers, I didn't have a rank, and all my stats were gone. So I had to also check the integrity of the database; fortunately, no issues were found on the database side. The issue was a file that got reset by the server software after rebooting, probably with the intent of updating it with the latest updates, but with there being no more space, it couldn't do so, so it just rest the file. With all of that out of the way, some servers were still stuck in a reboot loop, even with all plugins and configurations being fixed. That last issue was because of the so-called "usercache" file the server creates. It should be a JSON file and cache some data about the players. But with the server unable being to create the whole file, it created an empty file without the correct JSON syntax. With such a broken usercache file, the server tries to load the data out of it but keeps crashing because it couldn't load the file. By just deleting the file, this was fixed. Now that everything has been restored to its original working state, it is expected that there will be no further complications. https://status.greev.eu/incident/328081 Sat, 17 Feb 2024 08:15:00 -0000 https://status.greev.eu/incident/328081#5cb19871b93e15d19770dcf383d03b00eee97136eb37b6f03cad6e1b8f778e7f Today, at around 04:30 UTC, the Server started to no longer allow players to join. This issue persisted until around 08:15 UTC. The issues started at around 23:25 UTC on the 16/02/2023 by one server creating hundreds of gigabytes of logs. This on its own wouldn't have been an issue, but as it started late at night and the alerts for high disk storage wouldn't fire until 03:40 UTC. A full disk can be a big issue for the server, especially if it’s using every last byte. Now, what exactly happened on Greev that caused a full disk to result in the whole network being unavailable? With the disk being full at around 04:00 UTC, it created some more errors in the console but didn't result in too many issues. The real problems started at 04:30 UTC, which is the time many of our non-dynamic servers, like KnockPVP or FastBridge, started to do their daily restarts. Together with their daily restarts, the servers also reset their worlds, some plugins, and, most importantly, Spigot tries to store the current settings again. With there being no storage left on the disk, they couldn't just save their settings, which resulted in corrupted files. In addition to that, plugins couldn't even connect to the database anymore; this was also an issue of the file system being full with no more bytes to spare. As connections on Linux are handled as "files" no new connections / "files" could be created, causing even more issues. These corrupted files, together with there being issues with loading plugins after they couldn't connect to the database anymore, resulted in the servers not starting up anymore, and therefore players were unable to play any more games. And with the lobby restarting at 04:30 as well, the whole network went dark. I got to know about this issue at around 07:00 UTC, just after I woke up. When I started to investigate what the issue was, I quickly noticed that just restarting the affected servers resulted in no fix. So I dug deeper and found out that no more space was available, quickly finding the log file with over 500 GB of logs. Subsequently to deleting the file, I quickly discovered that there were still issues on some servers. These servers were affected by corrupted files from the plugins and the servers. I had to load some files from backups, which I could do quite quickly. First, I thought it was just some plugin files with issues, but after finding out that some of the servers were still having issues after booting up, I had to do some more digging. On some of the servers, I didn't have a rank, and all my stats were gone. So I had to also check the integrity of the database; fortunately, no issues were found on the database side. The issue was a file that got reset by the server software after rebooting, probably with the intent of updating it with the latest updates, but with there being no more space, it couldn't do so, so it just rest the file. With all of that out of the way, some servers were still stuck in a reboot loop, even with all plugins and configurations being fixed. That last issue was because of the so-called "usercache" file the server creates. It should be a JSON file and cache some data about the players. But with the server unable being to create the whole file, it created an empty file without the correct JSON syntax. With such a broken usercache file, the server tries to load the data out of it but keeps crashing because it couldn't load the file. By just deleting the file, this was fixed. Now that everything has been restored to its original working state, it is expected that there will be no further complications. https://status.greev.eu/incident/328081 Sat, 17 Feb 2024 07:25:00 -0000 https://status.greev.eu/incident/328081#b26bc965cc57745b149999fa407c627ed3792b6f6d1e8ad30a030f7a2c62c002 The issue has been found it we are fixing it at the moment. https://status.greev.eu/incident/328081 Sat, 17 Feb 2024 07:25:00 -0000 https://status.greev.eu/incident/328081#b26bc965cc57745b149999fa407c627ed3792b6f6d1e8ad30a030f7a2c62c002 The issue has been found it we are fixing it at the moment. https://status.greev.eu/incident/328081 Sat, 17 Feb 2024 07:00:00 -0000 https://status.greev.eu/incident/328081#e2837ddf3aac881d9ffa2b754ef0891a0dd38c1f0a0a7dad347a1dcd62ff9778 We got notified by users which couldn't join the server anymore and started to investigate the issues immediately. https://status.greev.eu/incident/328081 Sat, 17 Feb 2024 07:00:00 -0000 https://status.greev.eu/incident/328081#e2837ddf3aac881d9ffa2b754ef0891a0dd38c1f0a0a7dad347a1dcd62ff9778 We got notified by users which couldn't join the server anymore and started to investigate the issues immediately. https://status.greev.eu/incident/328081 Sat, 17 Feb 2024 04:30:00 -0000 https://status.greev.eu/incident/328081#d36d6e983a8d09981e69a73091a2ee095d3b243f3487b31d9f367461d4a93a2b Users are no longer able to join the server. https://status.greev.eu/incident/328081 Sat, 17 Feb 2024 04:30:00 -0000 https://status.greev.eu/incident/328081#d36d6e983a8d09981e69a73091a2ee095d3b243f3487b31d9f367461d4a93a2b Users are no longer able to join the server. https://status.greev.eu/incident/260989 Sat, 23 Sep 2023 06:00:00 +0000 https://status.greev.eu/incident/260989#907d9f237c4baa90e1abedc6fd25d1b9928d29ff344ab92cba792fd31e346d62 Maintenance completed https://status.greev.eu/incident/260989 Sat, 23 Sep 2023 06:00:00 +0000 https://status.greev.eu/incident/260989#907d9f237c4baa90e1abedc6fd25d1b9928d29ff344ab92cba792fd31e346d62 Maintenance completed https://status.greev.eu/incident/260989 Sat, 23 Sep 2023 06:00:00 +0000 https://status.greev.eu/incident/260989#907d9f237c4baa90e1abedc6fd25d1b9928d29ff344ab92cba792fd31e346d62 Maintenance completed https://status.greev.eu/incident/260989 Sat, 23 Sep 2023 04:00:00 -0000 https://status.greev.eu/incident/260989#305fbf93ce33ad6bb2a8a8a5d072b79964161ae95a58240420398e990dccf921 We are planing to do a larger scale maintenance job on the network. Unfortunately, this requires us to temporarily shut down the server. We'll get it back online as fast as we can possibly can. https://status.greev.eu/incident/260989 Sat, 23 Sep 2023 04:00:00 -0000 https://status.greev.eu/incident/260989#305fbf93ce33ad6bb2a8a8a5d072b79964161ae95a58240420398e990dccf921 We are planing to do a larger scale maintenance job on the network. Unfortunately, this requires us to temporarily shut down the server. We'll get it back online as fast as we can possibly can. https://status.greev.eu/incident/260989 Sat, 23 Sep 2023 04:00:00 -0000 https://status.greev.eu/incident/260989#305fbf93ce33ad6bb2a8a8a5d072b79964161ae95a58240420398e990dccf921 We are planing to do a larger scale maintenance job on the network. Unfortunately, this requires us to temporarily shut down the server. We'll get it back online as fast as we can possibly can. https://status.greev.eu/incident/226375 Tue, 04 Jul 2023 04:00:00 +0000 https://status.greev.eu/incident/226375#11be330d5d803872eba1ecc0bfeb0bba0d5341967d5c6a421c5654fd14b11979 Maintenance completed https://status.greev.eu/incident/226375 Tue, 04 Jul 2023 04:00:00 +0000 https://status.greev.eu/incident/226375#11be330d5d803872eba1ecc0bfeb0bba0d5341967d5c6a421c5654fd14b11979 Maintenance completed https://status.greev.eu/incident/226375 Tue, 04 Jul 2023 04:00:00 +0000 https://status.greev.eu/incident/226375#11be330d5d803872eba1ecc0bfeb0bba0d5341967d5c6a421c5654fd14b11979 Maintenance completed https://status.greev.eu/incident/226375 Mon, 03 Jul 2023 19:00:00 -0000 https://status.greev.eu/incident/226375#0c649d6a7f1475bde351c8bd7f77df97cc77d1d169e1a48e98113b2044a7d4d4 Our hoster will perform some maintenance on the connection between Frankfurt and Falkenstein where our servers are located. This might result in a higher ping than usual for a few hours. Additional Information: https://status.hetzner.com/de/incident/fbe47188-3bb5-4e45-bc5c-f6dcd3de9b0b https://status.greev.eu/incident/226375 Mon, 03 Jul 2023 19:00:00 -0000 https://status.greev.eu/incident/226375#0c649d6a7f1475bde351c8bd7f77df97cc77d1d169e1a48e98113b2044a7d4d4 Our hoster will perform some maintenance on the connection between Frankfurt and Falkenstein where our servers are located. This might result in a higher ping than usual for a few hours. Additional Information: https://status.hetzner.com/de/incident/fbe47188-3bb5-4e45-bc5c-f6dcd3de9b0b https://status.greev.eu/incident/226375 Mon, 03 Jul 2023 19:00:00 -0000 https://status.greev.eu/incident/226375#0c649d6a7f1475bde351c8bd7f77df97cc77d1d169e1a48e98113b2044a7d4d4 Our hoster will perform some maintenance on the connection between Frankfurt and Falkenstein where our servers are located. This might result in a higher ping than usual for a few hours. Additional Information: https://status.hetzner.com/de/incident/fbe47188-3bb5-4e45-bc5c-f6dcd3de9b0b https://status.greev.eu/incident/215621 Sat, 03 Jun 2023 19:29:00 -0000 https://status.greev.eu/incident/215621#e19fe202a2036c5a90517a0793a3923c15c3e65e2b7e2eeaf8c6e84d3e794a1d We got the update that our traffic will from now on be routed though a backup system to improve the connection. https://status.greev.eu/incident/215621 Sat, 03 Jun 2023 19:16:00 -0000 https://status.greev.eu/incident/215621#ffe19290b63add01afa775364334b089d4dfcd4c6518ffb33b2e92ebfee33ef2 The issue reoccurred, waiting for feedback from the DDoS protection provider. https://status.greev.eu/incident/215621 Sat, 03 Jun 2023 19:08:00 -0000 https://status.greev.eu/incident/215621#2500aff23c82b046b312889b1882b274c7660b6fff0a0312a53c2abf97e5f1f4 Status update from the DDoS Protection provider: "Attack patched [...] Continuing to monitor". We are hoping it's resolved by now then and the Server will be fully available once again. https://status.greev.eu/incident/215621 Sat, 03 Jun 2023 18:47:00 -0000 https://status.greev.eu/incident/215621#1cf709836fc8fbb73cc322ce7e6523103bc82baa373e99c554bbbfc14e64d127 Currently, we are experiencing some networking issues from our DDoS Protection provider. We sadly are not able to do anything for the time being and have to wait for a fix from their side. https://status.greev.eu/incident/186022 Fri, 17 Mar 2023 13:48:00 -0000 https://status.greev.eu/incident/186022#8409c098994bce85e0f643a58a1b9c3b642669cfb5f9c3db45e5085f54a07564 The DDoS Protection has been upgraded and prevent future DDoS attacks. https://status.greev.eu/incident/186022 Fri, 17 Mar 2023 07:30:00 -0000 https://status.greev.eu/incident/186022#4a9bb80b0b076314f01d6fa87040ea43842a0ac80b3eb38eb01278d49fc87ebe Currently, we are suffering from ongoing DDoS attacks. We are trying our best to resolve these as fast as we can. https://status.greev.eu/incident/108470 Wed, 10 Aug 2022 11:00:00 -0000 https://status.greev.eu/incident/108470#dc1fc7e50ce8cf07ba209e799639289097ea99d7c3b4e4f4cf80e29e7287c17a All Websites are currently on Maintenance. We have to upgrade a few internal components, all will be back to normal as soon as possible. As workarounds for the stats page, you can use the Command "/stats" In game. And as a workaround for the redeem page you can use the "/reward" command in the hub. https://status.greev.eu/incident/96952 Wed, 22 Jun 2022 14:30:00 -0000 https://status.greev.eu/incident/96952#213e1d2419680d196bca29965efd0d00fe62e147f6c95a7d21e8cbaca418098e It seems like it's back online. Some Servers / Minigames might still need to recover tho. https://status.greev.eu/ https://status.greev.eu/incident/96952 Wed, 22 Jun 2022 13:56:00 -0000 https://status.greev.eu/incident/96952#7a31c38bd81e28ed416b6cde787e12d5090d39a55d2552a1605909f9398682bb The Minecraft Network is currently offline due to a power outage at the data center. https://status.greev.eu/incident/62593 Thu, 16 Dec 2021 17:38:00 -0000 https://status.greev.eu/incident/62593#fe90baa9d9b8c3fff752628b8950b195d8433fc63a7175dab96f871c40ae5016 The Connection seems to be gone by now. https://status.greev.eu/incident/62593 Thu, 16 Dec 2021 17:38:00 -0000 https://status.greev.eu/incident/62593#fe90baa9d9b8c3fff752628b8950b195d8433fc63a7175dab96f871c40ae5016 The Connection seems to be gone by now. https://status.greev.eu/incident/62593 Wed, 15 Dec 2021 20:00:00 -0000 https://status.greev.eu/incident/62593#420a8e6dc84092e2b15dc9b0dd4cb1db21cf89c955cdd80ad25992efefd8a9a3 The Network is currently not as responsive as usable due to Network Maintenances from one of our Upstream Providers. This will probably be resolved in less than 24 Hours. If you have problems connecting please try again since sometimes connections to the auth servers fail. https://status.greev.eu/incident/62593 Wed, 15 Dec 2021 20:00:00 -0000 https://status.greev.eu/incident/62593#420a8e6dc84092e2b15dc9b0dd4cb1db21cf89c955cdd80ad25992efefd8a9a3 The Network is currently not as responsive as usable due to Network Maintenances from one of our Upstream Providers. This will probably be resolved in less than 24 Hours. If you have problems connecting please try again since sometimes connections to the auth servers fail. https://status.greev.eu/incident/54154 Sun, 05 Sep 2021 20:00:20 -0000 https://status.greev.eu/incident/54154#1105d729a47d8248b1c1948b29e7611b7ea8791845415b2377ce010d2a8bbdbe Server Downtime As you all probably noticed, the server was down for around 4 hours. Here a brief summary of what happened, what happens and what the current status is. What happened? Someone, I don't know who — so far. Ran a DDoS Attack against one of our Servers. I didn't know the magnitude of the attack, but it was enough to knock the server out. I talked with the hoster of that server about the issue but since it was a Sunday they were not able to anything. Furthermore, I gave them the needed logs they assured me they will look into it but as of writing this text I haven't got a response. Since the server normally should have blocked that attack, I am eager to hear back. Anyway, as soon as I realized that they will not be able to do anything for me, I switched to our backup plan. What is the current Status? Since I obviously didn't prepare to move everything in a matter of one / two hours, let alone minutes, I only started the most important servers. Which of course means not all servers are currently running. Server which are currently offline because of that include: Fastbridge Short 1, Fastbridge Short Inclined, KnockbackPVP 2, Builder Server, Replay Server and Fastbridge Short VIP. All other services like the stats page etc. are not impacted. I will try my best to get all of these back online asap. In the meanwhile, players can enjoy the network as if nothing happened. What happens now I am still in talks with the hoster. As soon as this is done, I will start to roll everything back to the old state — including starting all servers back up. https://status.greev.eu/incident/54154 Sun, 05 Sep 2021 16:03:26 -0000 https://status.greev.eu/incident/54154#38f92fd69aac934881dac9c4bc2aa5317505abeb3b4f6988f5c18ba742b47aad The Server is currently unavailable because of a DDoS attack. https://status.greev.eu/incident/53506 Tue, 24 Aug 2021 15:10:00 -0000 https://status.greev.eu/incident/53506#1166ea4163dc742cbf63deadb48d1fc0c509f467e898b94a0bcdbaa46994e51d The Website is now back online https://status.greev.eu/incident/53506 Sat, 21 Aug 2021 14:59:20 -0000 https://status.greev.eu/incident/53506#c1cbb3953ca576e7da97d4d9e4431d2b15a5921582da10c98d43d608e6a9105e We are redirecting the website for now to the forum https://status.greev.eu/incident/53506 Fri, 20 Aug 2021 14:56:05 -0000 https://status.greev.eu/incident/53506#3f484baf5ab974d190670f50495aa98ea539157d9bdc9fd44078fd9fb735cafa We are currently changing the server and therefore the API as well as the Website are offline. API is back online, however the Website as well as the stats page are still offline.