• OwnCube has been hosting Open Source Clouds since 6 years

OwnCube Network status

This web page publishes announcements and current fault reports from our datacenters.

All systems online
19 January, 2019

All systems online

 

Outage:
Shared DE21 *HDD issue - Outage*

 



Permalink
Updates and reboot Shared CH32 * closed *
19 January, 2019

Dear clients,

Status: closed
Service: UP
Servers: Shared CH32

we will do important kernel and system updates on CH32 today that will cause a downtime of the service.
Because of the updates, its required to reboot the server when its done.

Please allow some time.

Thank you for your time.

We will update the status when we start the action.

UPDATE #1: we will start the updates and reboot now.

UPDATE #2: System updates done. We shutdown WEB, MYSQL now to do updates to php7.2, mariaDB,...

UPDATE #3: Updates and reboot done. System is back online

Thank you for your time and have a great day



Permalink
Single DE cloud updates * closed *
15 January, 2019

Deutsch

Sehr geehrte Kunden,

wir möchten sie informieren, dass wir am 16.01.2019 ab ca. 17:00 Uhr, Updates auf unseren Single DE Systemen ( DE Nextcloud & DE ownCloud ) vornehmen. Während dieser Zeit wird die Cloud in den Wartungsmodus gesetzt und es ist kein Zugriff möglich.
Aufgrund der großen Anzahl an Dateien und Usern, können die Updates einige Zeit in Anspruch nehmen. Unsere Technik ist 24h vor Ort und aktiviert die Clouds sobald die Updates erledigt sind.

Wir bedanken uns für Ihr Verständnis

UPDATE: Wir haben alle Arbeiten abgeschlossen

 

English

Dear customers,

We would like to inform you that on the 16.01.2019 from about 17:00 we will start updates on our single DE systems (DE Nextcloud & DE ownCloud) .
During this time, the cloud is put into maintenance mode and no access is possible.
Due to the large number of files and users, the updates may take some time. Our technician are on site 24 hours a day and we enable the clouds as soon as the updates are done.

Thank you for your understanding

UPDATE: Updates are done



Permalink
Shared DE21 *HDD issue - Outage*
08 January, 2019

Status: In progress
Service: Down
Servers: Shared DE21
Status: Closed
Service: UP
Servers: Node DE31 - VPS systems

 

Deutsch

Sind Sie vom Ausfall betroffen ?
Sie können Ihre Cloud domain/subdomain per ping ansprechen. Wenn Sie eine "Antwort" von 138.201.148.* erhalten, sind Sie betroffen und am DE21 gehostet.
Sie können den Ping auch online ausführen: https://ping.eu/ping/ und Ihre Server Ip zu ermitteln.

 

Letztes Update:

Alle VPS sind noch immer online und arbeiten. Raidsync der zweiten Harddisk ist abgeschlossen.
Alle Accounts ( 58% ), die keine Schreib/Lesefehler angezeigt haben, sind bereits auf einem neuen Server online.
Aufgrund der noch bestehenden i/o Probleme ist es uns aktuell nicht möglich, die verbleibenden Accounts zu transferieren.
Wir haben uns Unterstützung von unserem Techniker-Datenrettungsteam aus dem RZ geholt, und diese arbeiten ab heute an der Wiederherstellung, da es bessere und schnellere Wege gibt, dies im RZ selbst zu bearbeiten als von remote.
Es tut uns leid, dass der Ausfall für manche Kunden noch immer andauert sowie der Rescuetask noch einige Zeit in Anspruch nehmen kann.
Soweit möglich, erhält jeder Kunde nach Abschluss und Wiederherstellung einen Download für die alte Cloud+Daten.
Sollte Ihr Account aktuell noch betroffen sein, bitte kontaktieren Sie unseren Support, um eine Neuinstallation auf einem anderen Server zu erhalten. Ihre Subdomain/Domain bleibt natürlich bestehen und das Setup ist kostenlos.
Weiters erhalten alle betroffenen Kunden doppelten Speicherplatz und 2 Monate kostenlosen Service.
Alle OwnCube Kunden werden ab heute kostenlos auch in unser internes Datenbank-Backupsystem integriert, um bei einem derartigen Ausfall von Mysql direkt extern weiter hosten zu können.
Wir möchten uns nochmals für die Umstände und Unannehmlichkeiten entschuldigen.

Aufgrund dieses aktuell kritischen Zustandes können wir keine Angaben zur Dauer der Wiederherstellung geben.

 

Ticketstatus: ( Anfragen Neuerstellung Cloud )
Alle erledigt

 

 

English

 

How to check if you are affected by the outage:
PING your domain/cloud subdomain. If you get a response from 138.201.148.*, you are hosted on DE21.
You can also use public ping checks like: https://ping.eu/ping/ to check the IP of your cloud.

 

Latest update:

* All VPS are still online and working. Resync raid on second drive done. All accounts ( 58% ) which do not show any i/o errors have been transferred to a new server already and are online!
We are unable to transfer the accounts that still show i/o errors. We will request help from the data-rescue company in our DC to do detailled work on the drives which is not possible from remote and with "default tools".
We are sorry that some accounts cannot be transferred at the moment. Data rescue tasks can take very long and we reinstall/recreate clouds which are still not migrated asap to new nodes. Of course we will provide a backup of your cloud... when the rescue of the drives has been completed.
If your cloud is still down, please contact our support team to get a new cloud setup done on our new performance servers. Of course with the same domain/subdomain you have had, this is not going to change!
We will provide double disk space and 2 months of free service for all customers who were affected.
Additionally, we provide FREE service for customers who are still affected until we have a solution or task end by the rescue team.
After the rescue task is done, we will also provide the download of a FULL backup ( data,... ) of the accounts which are not yet working.
We are very sorry for the issues which were caused and are still being caused for some clients.


Because of this critical issue, we are unable to provide any ETA for this downtime.

 

Ticketstatus: ( to setup a new cloud on a other server )
All answered and done.

 

HISTORY

 

Dear clients,

we currently investigate a issue on our NODE DE31. This also affects one shared server DE21 at the moment.

We will update the status when there are news about this.
Thank you for your time and sorry for the issue.
Team OwnCube

 


UPDATE #1:
Server is back online and all VPS, shared serverDE21 too.

UPDATE #2:
Server will be shutdown ones more to verify everything fixed
We update the status asap

 

UPDATE #3:
We currently investigate a issue on this server that is causing reboots of the Node all the time without any action.
Our datacenter techs are currently do a hardware check to see if the electricity or any other hardware needs to be replaced.
Harddisk, data and vps are already checked and are "CLEAN" so safe and the issue needs to be somewhere else.
We update the status after the hardware check

 

UPDATE #4:
After a hardwaretest we have seen some reboots of the RAID hardware controller that could be the issue.
We will replace it in the next 30 minutes and boot up all servers to see if its fixed.

 

UPDATE #5:
The replacement of the Raid controller fixed the issue and all systems are back online
Thank you again for your time and understanding.

 

UPDATE #6:
Server went down again after a few minutes without any action.
We have contacted the DC to start a detail check for the hardware.

 

UPDATE #7:
Shared DE21 is back online and working!
VPS on the Node are in progress to boot one by one

 

UPDATE #8:
This evening ( in the next hours ) we will perform a last shutdown of the node to replace a "broken" fan and do a system/drive check again with more details to find. It is required to do this checks when the node is offline/rescue mode to avoid system issues.
We will update the status when we power off the node and when its back online. Thank you for your time

 

UPDATE #9:
Power OFF done
* Syscheck started --> SYSTCHECK finished --> Result: OK, no issues found
* HDD checks running ( 41% ) -> 57% -> 65% -> 74% -> 81% -> 92% -> 97% --> Check stopped with a force filesystem check that has started now. Because of the large array on that server, this can take some time. We monitor this from our end and emergency techs are ready in the DC to work on this when the FScheck is finished.

 

Update #10:
After the FSCK, we have seen some issues on one disk in the array.
We will replace the harddrive asap, replacement ordered and on its way to the DC.
SHARED DE21 is back online. VPS systems are back online too.

Short update: New drive is ready in the DC, we will replace the drive today.
We update the status when the replacement starts and has been done.

 

Update #11:
Server shutdown for disk change in the next 10 minutes

Disk change done, server in rescue mode to add the new drive and resync the raid. Please allow some time for this because of the large array.
The new raid controller should also speed up the Raid and so your performance.
We will update the status every few minutes if there are news.

 

Update #12:
Raid recovery/add is at 58% with max speed at the moment. We await the server back online in the next few hours, latest during the night.
All affected customers will get 1 month free service and/or addtional addons for free.

Status updates:
( 03:57 ):
Raid recovery at 87% already, should be finished in the next hours. We will monitor this and boot all systems when its finished.

( 04:13 ): Raid recovery at 89%

( 04:33 ): Raid recovery at 93%

( 05:21 ): Raid recovery at 98%

( 05:31 ): Raid recovery at 100%

Raid seems in correct state. We check the disks now with FSCK to see if there are any errors.

 

Update #13:

EN: The FSCK ( file system check ) shows some errors that needs to be fixed before we boot up the system.
We have started fsck repair now, this will take a while because of the 50TB+ partition and millions of files on this server.
Please note that we cannot speed this up because its a system task and requires to check the whole Filesystem and for this reason we also cannot boot up the systems at the moment.
We will update the status when its finished, a progress status update is not possible for fsck because there is no status or "%" displayed for such checks.
At the moment we can only offer you to setup a new cloud on a other server and resync your files from your local device. A reinstall can be done within a few minutes and you can request it via the support section or email. We are sorry for the issue and we offer double disk space for free for all reinstalls. At the moment the server is in "offline mode" during the check and so we cant access/move/migrate any clouds or accounts.

Please bear with us. As soon as more information is available, we shall inform you via this website.
We apologize for any inconvenience. Thank you for your understanding.

 

DE: Nach unserem FSCK check wurden uns einige fehlerhafte Sektoren gemeldet die wir vor dem Start des Systems beheben müssen um mehr/weitere Probleme mit den HDD's zu vermeiden.
Bitte beachten Sie, dass dieser check sehr lange dauern kann da wir ein 50TB+ Dateisystem haben sowie Millionen von Dateien auf diesem Server hosten.
Wir können diese "Reparatur" auch nicht beschleunigen, da es ein Systemtask ist und das ganze Dateisystem geprüft werden muss.
Wir informieren Sie wenn wir neue Informationen haben oder fsck abgeschlossen ist. Aufgrund dass wir keine Statusanzeige bei fsck haben ,können wir keine genaue Angabe zur Dauer oder Restzeit machen.
Im Moment können wir betroffenden Kunden anbieten, eine neue Cloud auf einem anderen Server zu installieren und Sie syncen Ihre lokalen Daten erneut in die Cloud. Eine Neuinstallation ist per Ticket/Email jederzeit und innerhalb von Minuten möglich. Aufgrund der Umstände bieten wir jedem Kunden bei einer Neuinstallation den doppelten Speicher kostenlos an. Aufgrund dass der Server aktuell im "rescue mode" ist wo der FSCK durchgeführt wird, können wir keine accounts/clouds auf andere Nodes verschieben oder migrieren.

Bitte haben Sie etwas Geduld und entschuldigen Sie die Unannehmlichkeiten. Sobald neue Informationen vorliegen, werden wir Sie auf dieser Website informieren.
Vielen Dank für Ihr Verständnis.

 

Update FSCK progress:
* ( 07:42 ): Fsck is fixing some inode blocks at the moment. We monitor this of course.
* ( 10:38 ): Fsck still running and does not show many errors/repairs until now.
* ( 11:57 ): Fsck still running
* ( 13:28 ): Fsck still running. We are unable to access, nor boot the server, or mount the exist drives at the moment to migrate or move a service/cloud. The reason is, because the fsck is still running and its very important to not touch the harddrive or any array before its finished. A cancelation of this progress can cause a data loss or problems with the file system that we want to avoid. All available techs are set to high prio and we will instantly start working again on the server when fsck is finished.
* ( 16:15 ): Fsck still running. No new errors showing on the screen. fsck running in the process list, not finished yet.
* ( 18:21 ): Fsck is still scanning inodes and blocks.
* ( 20:59 ): Fsck Pass 1: Checking inodes, blocks, and sizes
* ( 02:59 ): Fsck Pass 2: Checking directory structure
* ( 05:11 ): Fsck Pass 3: Checking directory connectivity.
* ( 05:31 ): Fsck Pass 4: Checking reference counts
* ( 09:11 ): Fsck Pass 5 started
* ( 12:33 ): Fsck still running
* ( 12:42 ): Fsck finished, but the server does not boot and show i/o errors. Our team will check the status in the rescue mode.
* ( 13:51 ): Because the server does not boot we tried to mount the data partition with a live iso and that was possible. We dont know if all data are fine, we need to verify this when its on a live system.
The "root" partition were our OS of the node is installed still show i/o errors and we replace the HDD now too for the OS.
* ( 14:28 ): We have mounted the working data partition for all VPS services on this node to a external NodeOS and the VPS are starting at the moment and you should see them online in the next minutes too. We will check if everything is fine for the VPS services. If there are no errors on the VPS service partition, we will move the VPS to a new Node to avoid more issues.
* ( 17:43 ): All VPS are still online and working. We are in progress to migrate them to brand new nodes to avoid any more issues.
We also were able to boot the SHARED DE21 and access the filesystem, We have checked EMAIL service and WEB service, both are starting fine. The current issues are read/write errors on some folders/files, that also affects the MYSQL Service and so the clouds and software that require Mysql does not work.
* ( 22:23 ): All VPS are still online and working. Mysql on DE21 is still not working, either not with recovery settings. Because the VPS already running on different nodes , we can replace a second drive on shared DE21 that seems to have errors in SMART.
Because we have filesystem access to DE21, we already started the transfer of the data/cloud folders from shared DE21 to a new server. Then we can start to import accounts, so websites, email, cpanel will start working. If we get mysql working again, we can migrate it to the new server too. In the worst case, we need to get help from a datacenter data rescue team to assist in this.
* ( 06:51 ): All VPS are still online and working. 2nd drive replaced on de21 and resync at 67%. Transfer of accounts that does not show any i/o errors are already transferring to a new server. 10% of the accounts are back online, email/web is working. Mysql still offline on DE21 and we cant migrate it at the moment.
* All VPS are still online and working. 2nd drive replaced on de21 and resync at 95%. Transfer of accounts that does not show any i/o errors are already transferring to a new server. 18% of the accounts are back online, email/web is working. Mysql still offline on DE21 and we cant migrate it at the moment.

 

Server uptime info since outage:
Down: 08.01 12:00 -> 18:00
UP: 08.01 18:00 -> 09.01 01:00
Down: 09.01 01:00 -> 09.01 06:00
UP: 09.01 06:00 -> 09.01 16:15
Down: 09.01 16:15 -> IN PROGRESS

 

 



Permalink
US network issue ( shared US30 ) * closed *
07 December, 2018

Dear customers.

there is a network issue in the US DC at the moment. We are in contact with the DC to get this fixed. Because it affects the whole DC, we need to wait for a update from them.

We will update the status when we have news

thank you for your time

 

UPDATE:
All services back online, only Shared US30 is having some WEBserver issues that we check at the moment.

 

UPDATE#2:
US30 shared server is back online too



Permalink
KVM Node CP17 * closed *
07 November, 2018

Dear clients,

tonight we update the server software, kernel... on the Node CP17. This require a reboot of the node.

Addtional we will replace the RAM on the server to brand new ECC DDR4. This should only take a few minutes after the shutdown.

We will keep this info up2date during the downtime and when its completed.

Thank you for your time.

 

UPDATE1: Software updates done, server will be shutdown now for RAM change.

 

UPDATE2: Node is back online and all VPS bootet and ping. If you still have any issue please contact us via the support panel.

Thank you for your time



Permalink

Our business partners - OwnCube: trusted by companies, clubs and schools worldwide



Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
Goodbye Dropbox Google Drive and Co. Get your own Cloud - Ihre eigene Cloud
FREE Nextcloud Single Account: More Informations
OwnCube is EU-GDPR ready and provides DPA online