When good browsers go bad -- and they all do

[dhilipkumar]

When good browsers go bad -- and they all do
Better browsers. Better standards. Better tools. So why are Web pages still breaking?

Jeffrey Zeldman must have thought he'd never live to see the day. Ten years after he co-founded the Web Standards Project, all of the major browser vendors have shown renewed commitment to supporting World Wide Web Consortium (W3C) standards in the last few years -- and they're following through.

The specifications in the latest standards initiatives are tighter than ever. Web authoring tools are generating more compliant code. Two of the three rendering engines that underpin the major browsers in use today are open source. And, for the first time ever, the latest version of Microsoft's browser, Internet Explorer 8, will operate strictly in standards-compliant mode by default. In other words, it will support the W3C standards before it provides backward compatibility with nonstandard methods supported by Microsoft's previous browsers.

Those who lived through the browser wars of the '90s might think that hell has frozen over, were it not for one, small problem: Users still experience plenty of problems on the Web. A recent report issued by browser vendor Opera Software ASA showed that the average page on the Web strays from WC3 page markup specifications 47 times. Overall, just 4.13% of the 3.5 million Web pages Opera searched were fully standards-compliant when tested against the W3C validation suite.

Why? Because there are still older Web sites that haven't been updated, and many developers still don't design their pages to meet modern standards and best practices. Different browser vendors still interpret some standards differently in some cases and don't fully implement all of the W3C-standard features in others. Developers for some Web sites still lock out users who have the "wrong" browser by using whitelists or blacklists that block unapproved browsers or browser versions.

As for the standards themselves, some existing specifications still suffer from ambiguities or leave some desired features undefined. And improvements in the thoroughness of new standards, such as CSS 2.1, also have a down side: They have slowed the standardization process -- a fact that has frustrated some Web developers and could tempt browser vendors to once again innovate outside of the standards framework.


Turning the Microsoft ship
Microsoft's Internet Explorer doesn't dominate the browser market in the way it once did, but it still has the largest market share by far. IE currently accounts for about 70% of the browsers in use, down from a high of about 95% in the pre-Firefox days. Of that 70%, about half are running IE7 while about one quarter are still using IE6 or earlier versions.
Idiosyncrasies in the various versions of IE -- especially IE6 -- have created problems for other browser vendors and Web developers. Even Microsoft has struggled to maintain backward compatibility with bugs, nonstandard methods and other issues it has created for itself. As a result, Microsoft has had to drag along an installed base of corporate and public Web sites that were built to the quirks of its previous browsers, trapping itself into endless iterations of standards noncompliance to provide backward compatibility.


"It's difficult for them to keep their own bugs alive over the years because they know that they can break thousands of pages," says Philippe Le Hegaret, interaction domain lead at the W3C.

For example, in an inconsistency in IE that Chris Wilson, platform architect for Internet Explorer, calls the "CSS overflow problem," Microsoft's IE7 browser displayed overlapping text when viewing Web pages designed to IE's nonstandard approach used in IE6. The Web sites were sending IE 7 a special case style sheet for IE6 while serving up standards-compliant renditions of the pages to Firefox and every other standards-compliant browser. The sites worked with every browser except for IE7

[dhilipkumar]

When good browsers go bad -- and they all do

Because of IE's overwhelming market share, many developers have historically coded their sites to the nonstandard methodologies and work-arounds required to support IE6 and earlier browsers. Those sites often don't render properly when viewed from competing browsers, even if those browsers follow W3C standards. To address that, many developers keep two sets of cascading style sheets -- one for IE and one for everything else.

But times have changed. It used to be that everyone developed for IE first. Now it is the special case. Many developers code to a standards-compliant browser such as Firefox first and then present a different style sheet to IE users.

That's what Eric Meyer does. Meyer, a prominent Web developer, author and principal at Complex Spiral Consulting, says he might develop in Firefox first, using an extension like Firebug, then test it in Safari, Opera and other standards-compliant browsers. "In the end, I'll fire it up in IE, find out what went wrong and work the code around that," he says.


Looking toward IE8
As it develops its next version, Internet Explorer 8, Microsoft has gotten serious about supporting standards first. "In the past we've focused more on [backward] compatibility than interoperability. Now we're trying to be as interoperable as possible," says Wilson.

Microsoft's IE group is also playing a larger role at the W3C. "Microsoft has expressed that they want to support standards 100% on several occasions, and they are participating in our groups," says Le Hegaret.

Despite the progress, Microsoft still lags behind other browser vendors in incorporating the latest standards, such as Scalable Vector Graphics (SVG), which is implemented in Firefox, Safari and Opera.

But with many parts of Internet Explorer in need of updating, Microsoft had to choose where to focus its attention when it developed IE8. "We tried to move everything forward across the board in IE8, but we picked one area we wanted to get right," and that was support for CSS 2.1, says Wilson.

Wilson's goal is to have a complete implementation of CSS 2.1 when IE8 finally ships. But the CSS 2.1 specification is still in the Candidate Recommendation stage. Wilson says IE's release will go forward next year whether CSS 2.1 is final or not.He's confident, he adds, that there will be "minimal interoperability problems." If definitions do change in the final Recommendation, future versions of IE will change to support that, he says.

As with any other fully standards-compliant browser, IE8 will work on all Web sites that follow the standards -- and break when visiting sites that were designed to exploit the proprietary features and other idiosyncrasies of Internet Explorer 6 and 7. To fix that, Web site designers will either need to make their Web sites conform to standards or add a tag in the Web page header that tells IE8 to switch to IE7 behavior.

[dhilipkumar]

When good browsers go bad -- and they all do

Compatibility drag
However, not all the problems in IE7 are being fixed in IE8. One example of a problem that is being carried over is the nonstandard way in which IE defines an event handler in JavaScript. In event handling, a programmer writes a script that executes in response to an event, such as a user clicking on a link on a Web page.

"Microsoft deliberately ignored the standard," says freelance Web developer and author Peter-Paul Koch, who is known for his publication of browser compatibility tables at Quirksmode.org. "They said, 'OK, we have the standard, but we are not going to implement it,'" and now that same event-handling module is moving into IE8. In response, Microsoft's Wilson says, "Event handling is something we still intend to bring into compliance with the DOM standard. However, we simply couldn't get it into IE8." And he notes that there are "well-known work-arounds" that developers can use.

Fortunately, the event-handling issue doesn't come up often, says Koch. He says he's glad to see Microsoft making progress with IE8's support for CSS 2.1, and he thinks it is unreasonable to expect Microsoft to fix everything overnight.

Even so, some observers believe that Microsoft is a few steps behind competing browsers, such as Firefox, Opera and Safari. The makers of those browsers are already implementing some of the more mature modules in the emerging specifications -- including the Canvas video element in HTML 5 and Scalable Vector Graphics. Some competing browser vendors "are already implementing CSS 3 because some modules are in the candidate recommendation phase," says W3C's Le Hegaret.

Field of challengers
The four biggest competitors to IE -- Firefox, Safari, Opera and Chrome -- have their share of problems as well, but they pale in comparison to the problems surrounding IE -- particularly earlier versions such as IE6. "There are slight differences between Firefox and Safari and Chrome but they are very minor," says Koch.

But there are problems. Firefox is the No. 2 browser on the market, with a share of about 20%. While Firefox follows standards, it is not perfect. For example, Web sites may write to Firefox-specific properties, such as the Gecko DOM Range.comparePoint method and other similar events, says Bruce Lawson, Web evangelist at Opera.

In fact, differences in how standards have been interpreted and bugs in the implementation have created issues for Opera, he says. Lawson calls those problems "Mozilla-isms."Opera had to resort to adding JavaScript-based hacks to work around the compatibility layer that some Web sites had created for Firefox, says Lawson. For example, Opera changed how it handled the Date.getYear() method so that it returned a full year when the year was > 1999 or < 1900.

It did this for cross-browser compatibility, "despite breaking the JavaScript spec," Lawson explained. Opera changed it, breaking the spec in version 9.5, because, Lawson says, "Mozilla and Apple got it wrong and it was breaking many pages -- mostly airline calendar widgets."In some cases, we supported the IE technology that the compatibility [layer] worked around but didn't support the Mozilla properties, so we had to force the site to give us the original IE code for that functionality. This is clearly not the best way to develop cross-browser Web sites."

No browser today is 100% standards-compliant, says Meyer. "I don't think they ever can be. But they're certainly much, much closer than they used to be."

[dhilipkumar]

When good browsers go bad -- and they all do

Tooling up for standards
Offering developers good tools that produce clean, standards-compliant code is critical to having an interoperable, standards-compliant Web. "Tools like Dreamweaver have made tremendous strides forward in standards support," says Derek Featherstone, group lead for the Web Standards Project. This advocacy group is best known for the Acid3 Browser Test, which checks browsers for standards compliance.
Meyers agrees that the improvement has been dramatic. "You really have to work to make them not produce standards-oriented markup and CSS now," he says.

But Web authoring tools are just one piece of the puzzle. Other sources of Web output -- such as Microsoft Word, with its ability to save a document as a Web page, and the default page templates used by many content management systems -- aren't always up to snuff.

"You have many automated tools -- wikis, content management systems -- that produce millions of pages every day. We need to make sure all of those tools provide valid markup," says Le Hegaret. While some tools do a good job -- and some even link to the W3C's HTML validator -- overall "we still have less support in authoring tools than we'd like," he says.


Educating the masses
You can have complete, unambiguous standards in place, fully compliant browsers and state-of-the-art authoring tools that generate compliant code, but nothing will change until Web site developers change their behavior. "There are lots of people who just slap code together until it works," says Meyer. But how do you get the world's Web developers -- and the more than 20 billion Web pages they've created -- up to speed?

Although the user is running Firefox 3, this site doesn't think so.
Click to view larger image.
Getting developers on board has always been a challenge. When Cascading Style Sheets were largely ignored by most developers in 2001, developer apathy almost killed the effort. Zeldman and his peers at the Web Standards Project evangelized to the broader community, converting high-profile developers at some of the largest and most successful Web sites to the new standards religion. "Today, as a result of that grass-roots advocacy, all browsers support standards, and most clued-in Web designers use standards as a matter of course," he says.

But many still aren't clued in, and education is now the Web Standards Project's biggest focus, says Featherstone.

Zeldman says developers who aren't following standards today fall into three groups. "Some are unwilling, some don't know any better, and some are willing and know better but are prevented from implementing best practices because of wrong-headed directives by out-of-touch marketing departments -- and sometimes, even IT departments."

Koch thinks that too much of the Web development community today consists of amateurs. "There are a lot of Web developers who haven't the faintest idea of what they're doing," he says. But he expects that to change as standards take hold. "Over the next few years, the people who aren't professionals will be out of a job."But Le Hegaret thinks amateur designers are part of the fabric of the Web. He doesn't think they need to understand the details of the specifications to publish on the Web. "People shouldn't have to know CSS to produce nice Web pages. They should just have their tools produce CSS for them," he says.

[dhilipkumar]

Corporate holdouts
The last to get standards religion may be internal-facing corporate Web sites, many of which were built specifically to work with Internet Explorer. "You find more proprietary approaches in the corporate environment because they control which browser they're using," says Le Hegaret. For example, ActiveX controls, which work only on Windows and are supported only in Internet Explorer, are more common on internal-facing Web sites.
Because IT can control which browser is used, organizations can protect themselves in the short term from the move to a standards-based Web. However, as market share for earlier versions of Internet Explorer drops over time, more and more Web developers will stop building customized sets of style sheets for Internet Explorer. It's simply too expensive to keep supporting a one-off case.

"If your Web site has to work with IE6, you have to add 20% to 30% in extra development time," says Koch. He predicts that developers will start dropping support for IE6's quirks within the next 12 to 18 months.

By making IE8 standards compliant by default, Microsoft is putting its corporate customers on notice that it's serious about standards. Koch was pleasantly surprised by Microsoft's change of heart. The broader community of Web developers is "suddenly more important than corporate developers. That's something I never expected to see."

Several other factors keep developers stuck in the past when it comes to coding Web sites. Many people are learning using outdated tutorials or are working with tools that aren't up to date and generate noncompliant Web pages. And the ease with which designers can borrow from one another on the Web helps propagate problematic practices.

"We're feeling the effects now of decisions that were made seven years ago. That's what frustrating to me," Featherstone says. And the challenge doesn't just involve overcoming technical problems such as the need to rework Web sites built exclusively for IE6. The attitude toward Web authoring was different then, he says: "It was, 'Let's get it out there quick and get it live,' not 'What's the best way to do this?'" In some quarters that attitude still persists.


The first release of Google's Chrome had a problem loading Gmail.
Click to view larger image.
But Web developers who want to learn can find plenty of up-to-date educational resources online says Zeldman. These include information offered by the Web Standards Project; the W3C's W3C tutorials and validation suite for Web pages, and Opera Software's Web Standards Curriculum. Resources are also available at sites such as Zeldman's A List Apart, Digital Web Magazine, and many others.

Zeldman says if you're not on the standards bandwagon, it's time to get on board. "Your Web sites will load faster, on more browsers and devices, reaching more people at lower cost," he says. "You'll spend more time iterating new features and coming up with new designs and net content ideas, and less time just getting the damn thing to work."

[dhilipkumar]

Innovations
As in the bad old days of browser wars, browser vendors still develop and include new features in their browsers that aren't yet standards -- and it may not be clear to developers that what they're embedding in their Web sites could change. In some cases, features are added to browsers and then submitted as possible standards later -- a process that could force developers who implement those technologies too early to rework their Web sites as the specification evolves.

For example, Apple Inc. released its Canvas design element in Safari for developers to play with, and then later offered the Canvas element to W3C as a possible CSS module. The idea has been accepted, although the first working draft has yet to be created.

For its part, IE8 will include a new feature called Access Control Specification, which handles cross-domain requests. "It's a new object, a new API that lets Web developers make a request across Web site domains," and it creates an object called a CrossDomainRequest (XDR) object, Wilson says. But it may be at odds with a similar effort at the W3C, the Access Control for Cross-Site Requests draft specification.

While Access Control is a standards draft in the Web Apps Working Group, says Wilson, "the working group declined our proposal of the separate object XDR, and wants to use Access Control" instead. While Microsoft has "concerns" about that approach, Wilson says that developers can allow the use of both objects by writing code that switches between objects as needs dictate. But in the long run, he says, "We hope to implement the working group's final spec."

Do developments like these foreshadow a return to the bad old days of browser wars? "If one of the teams starts ignoring that hard-won wisdom, we could be back where we were before," Meyer says. Adding lots of new capabilities outside of the standards-making process could easily "break forward movement" on standards.

Wilson doesn't see the XDR action as a move away from standards. "We were pragmatic, however, in recognizing we were better off building a parallel object for the time being," he says.But all in all, Meyer thinks that Microsoft and other vendors are more cautious than they once were about getting too far ahead of the curve. "To a large degree, the browser vendors have learned their lesson."

Broken browsers: A sign of life?
While most of the Web code out there still isn't fully standards compliant, most major Web sites are much better than they used to be. Even with the 95% of Web sites that aren't fully compliant with all W3C specifications, in most cases the violations are trivial to the end user experience. "They forget an Escape or an ampersand," says Le Hegaret.

The browser's user agent string can easily be changed to fool browser detection schemes. Click to view larger image.
"There are developers who are unwilling or unable to follow the specifications.... there are gaps in the specifications that allow vendor-specific technology to take off, and there are parts of the existing specs that have not been implemented by all major browsers," says Lawson. But, he says, "I think the situation is improving."Web standards are an ongoing process in which the goal posts keep moving all the time. They will never be perfect, nor will Web pages that follow them, even when those pages are created by standards-aware professionals with the best of intentions, says Meyer.

He points to one project where he was pulling images from Flickr into a Web site as an example. "The code [from Flickr] had unencoded ampersands in the links," he says. Meyer had a choice: "I could spend a half a day writing code to make sure I encoded every ampersand or I could do some real work." He chose the latter.For all of the recent gains, the open Web is still very much a work in progress, but that's a good thing, Meyers says. "The only time you don't have issues to be resolved is when a system is closed and dead."