美国服务器提供商IXWebHosting CP9 Services 宕机

准备好了

（原文转载地址： http://blog.shuxiong.net/archives/1686）

   美国服务器提供商IXWebHosting CP9 Services 于北京时间2009-12-24 9:44:43 宕机。此次宕机影响非常严重，IXWebHosting 出动所有高级工程师紧急抢修，据说是HP的硬件出了问题导致宕机。据说官方博客统计，CP9宕机最少让web Servers、MySQL Servers、mail Servers用户等合计约4000多用户受影响。其中涉及大约20000个网站不能访问。

今天10点起床就发现挂在IXWwbhosting的10个网站全部不能访问，当时还以为给墙了。立刻和IX的工程人员取得联系才得知CP9宕机了，心才平静了下来。毕竟这都是自己的心血做出来的。

等了漫长的几个小时，CP9 Services 终于在北京时间2009-12-24 15:44:43 抢修完工。历时6消失的抢修终于尘埃落定，网站终于可以正常访问了。

以下是和IX工程员的对话：

Oleg Barannik: Welcome, my name is Oleg, please let me know how can I help you today?

you: How it? The server is not available an hour,IP:96.0.216.168

Oleg Barannik: Could you please provide the domain name?

you: SHUXIONG.NET

you: ZHAOLUNWEN.NET

Oleg Barannik: Please hold on, let me check

Oleg Barannik: I am very sorry for this recent inconvenience, please accept my sincere apologies. Our system administrators have been notified and they are doing their best to fix the problem as soon as possible. It may take some more additional time to get it fixed.

Oleg Barannik: Thank you for bearing with us. We hope for your understanding and cooperation.

Oleg Barannik: You can check the status of this issue on our status blog:

Oleg Barannik: http://status.ixwebhosting.com/2009/12/23/cp9-services-down/

you: oh,i see ,thanks

Oleg Barannik: You are welcome

Oleg Barannik: Please feel free to contact us if you need further assistance. We are available 24/7.

IXWebhosting 官方关于CP9宕机的公告：

CP9 Services down

Hello,

We are currently having issues with all cp9 services. This includes all web, mail and mysql servers. You will experience very slow or no service at all. We have no ETA at this time but we’ll update the blog every 30 minutes to keep you updated.

Regards,

David Stuard

System Administrator

UPDATE @ 9:32

The servers and services are still down. We have called in extra help and are trying to get the services back to normal as quick as possible. We will keep you posted on the progress.

UPDATE @ 9:56

We are still experiencing an outage with cp9. We are currently having issues with our storage for the cp9 servers. We are working on the issue and right know we don’t have an eta. Will update you when we have more information. – Lucas

UPDATE @10:19pm

The problems we are experiencing are really serious. We have a 6hour-guaranteed-repair contract with HP which has been activated about 1 hour ago. We will keep you updated – Tiberiu Ungureanu

UPDATE @11:11pm

We are still working on the problem. HP has dispatched several field engineers, and we are trying to get things up as fast as possible. We take all the precautions necessary to avoid any kind of data loss. – Tiberiu Ungureanu

UPDATE @12:03am

The HP engineers are on site (they have been for a while) and they are working on the SAN. They are also in conference with highest level engineers (we woke up several people). We still don’t have an ETA, but we hope this won’t last much longer — Tiberiu Ungureanu

UPDATE @12.29am

The SAN is coming back up. The preliminary tests show 100% data integrity. We will update you shortly

UPDATE @12.46am

The SAN is back up. We are now bringing up the servers. To avoid having sites giving errors (like ‘cannot connect to database’) we will try to bring the servers up in the order Databases – Web servers – Mail servers. We will keep you updated. Current estimated uptime for everything is 3.45am

UPDATE @1:40am

We are still bringing up the servers right now. The new eta for all cp9 machines and services to be back up is 2:30 am. – Lucas

UPDATE @2:41am

All machines and services are now back online. This issue was caused by a faulty array controller in our EVA 8400 that is used for our storage on cp9. – Lucas

cpuer

IXwebhosting的宕机率确实有点高，