Why Amazon is rebooting 10% of its cloud servers

Amazon says a patch is needed to fix Xen hypervisor issue

Amazon Web Services issued a blog post on Thursday providing some more details of why the company needs to reboot up to 10% of its cloud servers in the coming days, and it doesn’t have anything to do with the so-called Shellshock vulnerability.

Amazon says that about Elastic Compute Cloud (EC2) servers from across the globe will be impacted by what it calls a “timely security and operational update” related to its open source Xen hypervisor. The blog post explains:

“As we explained in emails to the small percentage of our customers who are affected and on our forums, the instances that need the update require a system restart of the underlying hardware and will be unavailable for a few minutes while the patches are being applied and the host is being rebooted.”

The full blog post can be read here.
It appears to be just a coincidence that an update to the open source Xen Hypervisor is happening at the same time that security experts have identified a major vulnerability in Linux code known as the Bash Bug, which some are dubbing Shellshock. AWS officials say the two events are unrelated.

Amazon likely deals with many vulnerabilities each day and week, but Jesse Proudman, founder and CTO of cloud provider Blue Box says this Xen bug is different because it effects the hypervisor that creates virtual machines. The only way to appropriately patch the system is to reboot it.

AWS goes on:
“While most software updates are applied without a reboot, certain limited types of updates require a restart. Instances requiring a reboot will be staggered so that no two regions or availability zones are impacted at the same time and they will restart with all saved data and all automated configuration intact. Most customers should experience no significant issues with the reboots. We understand that for a small subset of customers the reboot will be more inconvenient; we wouldn’t inconvenience our customers if it wasn’t important and time-critical to apply this update.”

Amazon says that the updates must be done before October 1, when details of the Xen flaw are made public as part of the Xen update XSA-108 release. Expect at that time AWS and the Xen community will have more details as to the specific security flaw that is being patched.

Proudman suspects the issue is likely related to flaw CVE-2014-7155 In the Xen code, which was first announced on Wednesday. It was found that the bug can be exploited by a hacker to escalate its privileges, allowing the hacker to potentially glean access to other virtual machines. In contrast, an issue like Shellshock is something that can be patched in the Linux code and does not require a reboot of the machine.

Proudman says the CVE 7155 has been in the Xen code since the 3.2 release, which was in 2008. Still, he says that customers should not be too worried about the situation since Amazon will be updating all of its impacted machines before more details about the security vulnerability are publicly released on October 1. Proudman says AWS is absolutely doing the right thing by updating its systems and rebooting customer machines, even if that may cause some stress in the coming days.

The big takeaway for customers is that a subset of AWS instances will be rebooted at some point in the next five days. Cloud consultancy RightScale expects the reboots to begin at 10 PM ET on Thursday and run through Sept. 30 at 7:59 PM ET. Customers don’t necessarily have to do anything, but they should be prepared for their EC2 instances to go down for a few minutes if they’ve been notified by AWS. RightScale advises AWS users to test their system for a reboot. “It’s going to test the operational prowess of a lot of systems,” Proudman says.

---

Best Microsoft MCTS Certification, Microsoft MCITP Training at certkingdom.com

'Bigger than Heartbleed' Shellshock flaw leaves OS X, Linux, more open to attack

Well, this isn't good. Akamai security researcher Stephane Chazelas has discovered a devastating flaw in the Unix Bash shell, leaving Linux machines, OS X machines, routers, older IoT devices, and more vulnerable to attack. "Shellshock," as it's been dubbed, allows attackers to run code on your machine after exploiting the flaw, but the true danger here lies in just how old Shell Shock is—this vulnerability has apparently been lurking in the Bash shell for years.

Why this matters: A large swath of the web-connected devices, web servers, and web-powered services run on Linux distributions equipped with the Bash shell, and Mac OS X Mavericks is also affected. The fact that Shellshock's roots are so deep likely means that the vulnerability will still be found in unpatched systems for the foreseeable future—though the odds of it directly impacting you appear somewhat slim if you use standard security precautions.
MORE ON NETWORK WORLD: Free security tools you should try

Heartbleed redux

The news comes as the security community is just shaking off the effects of Heartbleed, a critical vulnerability in the widely used OpenSSL security protocol. "Today's bash bug is as big a deal as Heartbleed," says Errata Security's Robert Graham, a respected researcher.

Hold your horses, Robert. Before we dive into dire warnings, let's focus on the positive side of this story. Numerous Linux variants have already pushed out patches that plug Shellshock, including Red Hat, Fedora, CentOS, Ubuntu, and Debian, and big Internet services like Akamai are already on the case.

But Graham says Shellshock's danger will nevertheless linger for years, partly because "an enormous percentage of software interacts with the shell in some fashion"—essentially making it impossible to know exactly how much software is vulnerable—and partly because of the vulnerability's age.

"Unlike Heartbleed, which only affected a specific version of OpenSSL, this bash bug has been around for a long, long time. That means there are lots of old devices on the network vulnerable to this bug. The number of systems needing to be patched, but which won't be, is much larger than Heartbleed."

Now consider that more than two months after Heartbleed was disclosed, hundreds of thousands of systems remained vulnerable to the exploit.
Maybe not Heartbleed redux?

But don't panic! (Or at least not yet.) While Heartbleed had the potential to be widely exploited, Jen Ellis of security firm Rapid7 says the Shellshock bug's outlook isn't quite as grim, even if it is rampant.

"The vulnerability looks pretty awful at first glance, but most systems with Bash installed will NOT be remotely exploitable as a result of this issue," Ellis writes. "In order to exploit this flaw, an attacker would need the ability to send a malicious environment variable to a program interacting with the network and this program would have to be implemented in Bash, or spawn a sub-command using Bash."

As a result, Ellis and Rapid7 urge keeping a level head about the bug.
"We’re not keen to jump on the 'Heartbleed 2.0' bandwagon. The conclusion we reached is that some factors are worse, but the overall picture is less dire… there are a number of factors that need to be in play for a target to be susceptible to attack. Every affected application may be exploitable through a slightly different vector or have different requirements to reach the vulnerable code. This may significantly limit how widespread attacks will be in the wild. Heartbleed was much easier to conclusively test and the impact way more widespread."

While older Internet-connected devices (like, say, security cameras) seem to be likely victims of Shellshock, respected security researchers Michal Zalewski and Paul McMillan note that many embedded devices don't actually use the Bash shell at all.

Beyond Linux-based systems, Graham and Ars Technica report that Mac OS X Mavericks contains a vulnerable version of Bash.

To test if your version of Bash is vulnerable to this issue, Red Hat says to run this command:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If the system responds with the following, then you're running a vulnerable version of Bash and you should apply any available updates immediately:

vulnerable
this is a test

"The patch used to fix this issue ensures that no code is allowed after the end of a Bash function," Red Hat reports. So rather than spitting out "Vulnerable," a protected version of Bash will spit out the following when you run the aforementioned command:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test" bash: warning: x: ignoring function definition attempt bash: error importing function definition for `x' this is a test

What does this mean?

When it boils down to brass tacks, most major websites and modern gadgets you own likely won't be affected by this Bash vulnerability, and Apple will no doubt patch the OS X implementation quickly. (Here's a highly technical DIY fix for now.)

It's impossible to know just how far this flaw reaches, and it's likely to linger on in neglected websites, older routers, and some legacy Internet of Things devices—many of which are impossible to patch—providing an opening for determined hackers to sneak into those systems.

So what should you do? Here's some actionable advice from security researcher Troy Hunt's tremendous in-depth primer on Shellshock:

"In short, the advice to consumers is this: watch for security updates, particularly on OS X. Also keep an eye on any advice you may get from your ISP or other providers of devices you have that run embedded software. Do be cautious of emails requesting information or instructing you to run software – events like this are often followed by phishing attacks that capitalize on consumers’ fears."

PCWorld's guide to protecting your PC against devious security traps can help you I.D. bad actors, while Ian Paul has three tips for spotting malicious emails over at his Hassle-Free PC column.

---

Best Microsoft MCTS Certification, Microsoft MCITP Training at certkingdom.com

Happy Labor Day -- Will a bot take your help desk job?

Automation technology is getting better as help desk requests continue to rise

Competing forces are affecting people who work on help or service desks. One is improving automation tools, which advocates say can replace level 1 and 2 support staff. At the same time, the number of help desk tickets is rising each year, which puts more demand on the service desk.

These cross-currents in the industry make it hard to predict the fate of some IT jobs. A Pew survey, released in August, of nearly 1,900 experts found a clear split on what the future may bring: 52% said tech advances will not displace more jobs than they create by 2025, but 48% said they will.

Either way, a push toward automaton is certain. In the help desk industry, the goal is to keep as many calls for help at either Level 0, which is self-help, or Level 1, as possible. It's called "shift-left" in the industry.

"It costs way more to have a Level 3 or Level 2 person to resolve an issue, and it also takes a lot more time,' said Roy Atkinson, an analyst at HDI, formerly known as the Help Desk Institute. To keep costs down, help desks are increasingly turning to automation and improvements in technologies such as national language processing, he said.

A Level 1 worker will take an initial call, suggest a couple of fixes, and then -- lacking the skill or authority to do much more -- escalate the issue. The Level 2 worker can do field repair work and may have specific application knowledge. A Level 3 escalation might involve working directly with application developers, while Level 4 means taking the problem outside to a vendor.

Among the companies developing automation tools is New York-based IPsoft, a 15-year old firm with more than 2,000 employees. It develops software robotic technology and couples it with management services.

A majority of IT infrastructure will eventually be "managed by expert systems, not by human beings," said Frank Lansink, the firm's CEO for the European Union. IPsoft says its technology can now eliminate 60% of infrastructure labor tasks.

IPsoft's autonomic tools might discover, for instance, a network switch that isn't functioning, or a wireless access point that is down. The system creates tickets and then deploys an expert system, a software robot with the programming to make the repair. If it can't be done, a human intervenes.

Many service desk jobs have been moved offshored over the last decade, displacing workers. That trend is ongoing. One of the ideas underlying IPsoft's business models is a belief that offshore, as well as onshore, labor costs can be further reduced through automation.

Offshore firms are clearly interested. IPsoft's platform was adopted last year by Infosys and, more recently, by Accenture.

One IT manager using IPsoft's automation technology and services to support his firm's infrastructure -- including its network, servers and laptops -- is Marcel Chiriac, the CIO of Rompetrol Group, a Romania-based oil industry firm with 7,000 employees serving Europe and Asia.

"Without the automation, we would have to pay a lot more" for IT support, said Chiriac.

The cost savings arise from automatic repairs and routine maintenance that might otherwise be neglected, said Chiriac.

If he weren't using autonomic tools, Chiriac said he would have to hire more people for a similar level of service. But he can't easily estimate the impact on staff because of the firm's IT history. (Rompetrol Group outsourced its 140 IT staff, ended that relationship, then rebuilt an internal IT staff with about two dozen fewer workers; it also uses outsourcing as a supplement.)

Nonetheless, Chiriac doesn't believe that infrastructure automation will necessarily eliminate IT jobs, though it may shift them to other IT areas. "In IT, we're not going to run out of work for the next two generations," said Chiriac.

The work that help or service desks are asked to take on is increasing. Two-thirds of 1,200 organizations surveyed by HDI reported that the number of tickets, either to fix something broken or to outfit a new hire or change permissions, for instance, os increasing annually by more than 60%.

The top five reasons for this increase, according to HDI's survey, is an increase in the number of customers at surveyed firms, a rising number of applications, changes in infrastructure, increases in the scope of services, and the need to support different types of equipment and more devices. That latter could reflect BYOD use.

At the same time, support is being transformed in new ways. Service desks may, for instance, now act as a liaison for all service providers, including cloud and mobile carriers, said Atkinson.

"I think a lot of people have been predicting the death of support for a number of years, and it hasn't happened," said Atkinson.

---

Best Microsoft MCTS Certification, Microsoft MCITP Training at certkingdom.com