Today we moved a reasonable portion of the search cache infrastructure. This means that more queries are doing disk I/O than normal which results in slower response times until the data is cached. Once the new cache infrastructure is warmed up, search should start getting happy again.
Until then, you may experience slow results and intermittent errors.
Wednesday we moved over 100 machines including major portions of our ping handling, crawling and search infrastructure. Most everything went pretty smoothly though there were some network configs that kept pings offline longer than we wanted.
Our dev team has whipped up some changes to bypass slower internal network routes and the pings are flowing once again.
We are moving a large portion of our search infrastructure today. We are aiming for maximum uptime, but when moving 100+ machines, there are always casualties. We are also moving the main systems we use for this blog and our routine update system so it may be a bit difficult to keep you up-to-date on changes as they happen.
Expect delays in pings to continue into late Wed. Some searches may also run slowly, so please be patient.
We are experiencing high network latencies between our old and new colos which is impacting the availability of http://technorati.com/ping/. We are reconfiguring the network load balancing to help alleviate the bottleneck but don't expect to have this worked out fully until late Wed. afternoon.
Currently, all systems are working as expected though there may be some sporadic backlog as we continue to move infrastructure between colos.
The crawlers and blog claiming are off until Noon May 4 while we perform some post-move maintenance.
We have completed the move of our ping and claiming related infrastructure. We have processed the majority of the ping backlog and have just re-enabled blog claiming.
We'll keep you updated as the move progresses.
We completed the move of the necessary ping and crawling infrastructure Fri. afternoon and were able to resume ping processing and crawling at that time. We have had to leave blog claiming disabled as it shares resources and the processing queues are backed up from the move.
We will re-enable blog claiming on Sat. afternoon once the ping and crawl queue backlog has returned to a more normal level.
We are in the midst of moving parts of our ping handling infrastructure. As half of it is in the old colo and the other half in the new colo, we are experiencing some network latency between the two facilities. This has resulted in some ping processing backlog and slower time to index.
We will be moving enough capacity to help us work through the backlog on Friday, but ask that you bear with us in the mean time.
All other systems are working as expected.
Maintenance complete. All systems should be catching up now. Thanks for your patience!
Is this hot? Share it with your friends so they can read it and vote on it! Use email addresses or Technorati member names.
