Test the Web Forward

Article
06/29/2012

The quality and correctness of different browsers’ HTML5 engines continue to vary widely. We continue to contribute to the test suites under development at W3C to further the goal of web platform interoperability and same markup. In total, we have submitted 7573 tests that you can view at the IE Test Center as well. As different browsers improve their same-markup support, we can all realize the promise of HTML5.

The title of this post refers to the week-end event hosted by Adobe on June 15 and 16 at their San Francisco office. Dozens of volunteers joined W3C experts and members from Adobe, Google, Mozilla, Apple, HP, and Microsoft to learn about Web standards testing, how to write CSS and SVG tests, how to file good bugs, as well as the tools available for test suite management.

The meeting then turned into a test ‘hack-a-thon’ fueled by free drinks and food throughout the day. Volunteers spent most of their Saturday writing new test cases for the CSS OM, Transforms, Backgrounds & Borders, Exclusions, SVG, and other modules. Participants were then nominated for several prizes.

Testing Web Standards

Adobe’s Alan Stearns introduced the participants to the general principles of W3C testing and the role of testing in moving specifications forward. In particular, establishing browsers’ individual pass rate for a given specification is not a goal of W3C test suites. In order for a specification to become a W3C Recommendation the Working Group must prove it can be implemented. In practice this means:

Creating a test case for each requirement in the specification (these are known as normative statements)
Verifying that at least two separate implementations pass each test

Note the difference between ‘at least two browsers must pass the entire test suite’ and ‘at least two browsers must pass each test in the test suite’. Browser testers usually describe this phase as ‘testing the spec’.

But an important side-effect of this testing process is to establish a common interoperable baseline that all browsers can develop and test against. Test suites help find bugs across all browsers and can sometimes identify issues in the spec.

Writing CSS and SVG Tests

There are three different types of tests:

Stand-alone tests typically rely on visual verification: if a failure condition occurs, red content will show.
Reference tests compare a test against a visual reference that has no dependency on the feature being tested. Note that the test includes a link to the reference test against which is should be compared.
CSS Object Model tests depend on a JavaScript test harness; they verify that the object model reflects what static style sheets specify. For instance, this CSS media query test.

W3C’s Doug Schepers covered SVG testing while Adobe’s Rebecca Hauck and Jacob Goldstein provided a test writing tutorial. Peter Linss, CSS Working Group co-chair, offered a deep dive on the CSS testing framework including the test suite build system and management tools such as Shepherd.

Filing Good Bugs

Mozilla’s Elika Etemad then gave attendees advice on what makes a good browser bug report:

The issue is specific and reproducible
The build and platform are identified
You have looked for duplicates
It includes steps to reproduce the problem
The expected and actual results are described
If possible, the issue has been reduced i.e. all HTML, JavaScript and CSS that is not necessary to reproduce the problem has been eliminated from the problem page and the remainder attached to the bug.

Building a Test Suite

Building a test suite is a significant investment. One of the reasons it took a long time for CSS2.1 to reach the Recommendation stage was the size of the specification and the underlying number of requirements to test. The latest version of the test suite contains 9,422 tests.

Microsoft contributed over 7,000 of those tests, and we are continually contributing more tests for other standard specifications.

In IE10, we have added support for a long list of new standard features across CSS, HTML, SVG and the DOM. We have published some of our testcases for these new features on our IE Test Center. We will be submitting more, notably around those features recently unprefixed in the IE10 Release Preview.

How You Can Help

We are excited to be part of the community working towards a more interoperable web. If you want to help move the Web forward, you too can help to drive interoperability higher. You can learn how to contribute tests, or review existing tests. More information for those is available on the CSS WG wiki as well as on the event page.

We will keep you posted on future events.

—Sylvain Galineau, Program Manager, Internet Explorer and
—John Jansen, Test Lead, Internet Explorer

Comments

Anonymous
June 29, 2012
The Standards compliance is great. Please bring the following shortcut to Internet Explorer 10 and Windows Explorer in Windows 8 for the love of keyboard mouse users: "While holding the Ctrl button, if you press browser's back or forward button it should open the corresponding (previous or next) page in new tab. This behavior is present in Firefox, Chrome and Safari." Please fix this bug as well: www.freewebs.com/.../209-print-preview-input.html
Anonymous
June 29, 2012
The comment has been removed
Anonymous
June 29, 2012
Ingin membuka situs berkeluarga
Anonymous
June 29, 2012
The comment has been removed
Anonymous
June 29, 2012
A windows 7 release for IE-10 might speed up testing the latest features.
Anonymous
June 29, 2012
great results!
Anonymous
June 29, 2012
It looks like Internet Explorer improved a lot and sometimes I think I should start using it, but when I try I miss a lot of things, my bookmarks are not in sync between computers, must install many additional software to do that, and all the small little things chrome provides are just missing. the developers tools are also much behind Chrome, "inspect element" for a software developer, just does magic upgrade, Chrome is updates every 8 weeks or 6 forgot, new API is a there every month, things become a bit faster here and there every month, that is something IE is missing.
Anonymous
June 29, 2012
Lets be serious... no real testing of this new IE10 browser will occur until it is available to download for our EXISTING computers which all run Windows 7. If you want to release IE10 without full testing go for it, but if you were smart you would make an IE10 release that we can test on our local machines - because right now I can't test anything in IE10.
Anonymous
June 30, 2012
The comment has been removed
Anonymous
June 30, 2012
The comment has been removed
Anonymous
June 30, 2012
Randell, thanks. My thoughts are similar, just didn't take my time to write them.
Anonymous
June 30, 2012
The comment has been removed
Anonymous
June 30, 2012
@GT "the developers tools are also much behind Chrome, "inspect element" for a software developer, just does magic" You have that on IE, it's just a bit harder to access. Open developer tools, click on the arrow icon, click on the item to inspect. Just two steps if you already have developer tools open and pinned. It's also two steps on Chrome: First step - right click, second step - click inspect.
Anonymous
June 30, 2012
@Microsoft, please fix the blog comment system so that sign-ins and posting works on the first try and comments are not lost randomly.
Anonymous
June 30, 2012
The comment has been removed
Anonymous
June 30, 2012
I wonder if, due to the "upgrade ceiling" on Windows XP, Internet Explorer 8 will be considered the "new IE6" in a few years time. (The other candidate is the default Android 2.3 browser.)
Anonymous
July 01, 2012
@Vlad NC your assessment of the IE dev tools is incorrect. Ask any developer which toolset is best and IE will always be at the bottom of the heap. In chrome you are looking at the element you want to inspect you right click an choose inspect (2 clicks, plus a small scroll) In IE you need to use your other hand or release the mouse to press F12... Then you need to move your mouse all the way over to the find by click icon then you have to move back to where you came from and click the element you want (1 key press, 2 clicks, and two big scrolls) Not even remotely close to equal. Then of course you have the actual DOM tree and tools... Chrome's are LIVE IEs are not, the tree presentation in IE is unhelpful showing garbage nested nodes like "empty text element" IEs tools look and operate with the same ugliness of all 1996 apps with next to no usability enhancements. IE has come a long way with their browser but their dev tools are still atrocious an definately the worst of all tools out there. If I was forced to develop in IE all day long... I would quit on the spot - it is a horrible experience. Neil PS this comment form is still broken! Seriously how many Microsoft engineers does is take to comment out the broken Legacy ASP postback and add a real submit button?!?!
Anonymous
July 01, 2012
@Neil - Also, its speed. It is so horribly slow.

If you have twenty log messages at the start up of the page, it starts to slowly (very slowly) duplicate them on page reload until it finishes re-initializing and then gets back to normal (also slow, of course).
Use the uncompressed (or the prettify option) jQuery, debug code that calls a jQuery method and step into the jQuery code. See where that gets you. ;) So bad.

Anonymous
July 02, 2012
The comment has been removed
Anonymous
July 02, 2012
Please, seriously, please, release a new Preview/Developer Preview/Beta/Technology Preview (whatever you wan to call it) of IE 10 for Windows 7. Most people do not have the time or ability to work with Windows 8 yet. If you want lots of real world testing, this is the only way you will get it.
Anonymous
July 02, 2012
It seems to me like the Windows 7 version is being held back solely because one or two features rely on Windows 8 exclusive code, such as the spell check. I think I speak for everyone when I saw we would be fine with a preview version that did not have those features, though I guess its too late now because the Windows 7 release candidate or whatever it will be called is probably only a couple months away.
Anonymous
July 02, 2012
I don't think Windows XP will be relevant for web designers much longer, anyone willing to stick with a 10+ year old OS is probably using a non IE browser or is someone who probably wont be visiting your website.
Anonymous
July 02, 2012
@@Dave H I'd like to think so. Let's all look at our web logs in 2014 and see where we're at! Obviously the pressures on development differ between different types of website. An informational government site funded by taxpayers will have different criteria on browser support to a funky online gaming site. An ecommerce site will gauge additional development costs to lost custom, etc.
Anonymous
July 02, 2012
The comment has been removed
Anonymous
July 03, 2012
The comment has been removed
Anonymous
July 03, 2012
an article about Microsoft's Lost Decade www.vanityfair.com/.../microsoft-downfall-emails-steve-ballmer explains a lot about Microsoft's mismanagement and why the company has a very hard time both keeping employes happy as well as innovating and creating good products on time. it doesn't specifically mention the years of failed Internet Explorer browsers and the OS's known as Windows ME, and Windows Vista but it does elude to the expected downfall of Windows 8 - another case where employees that complained about this mixed OS (tablet/desktop) was a bad idea but where shot down by management that didn't want to take the time to fix it.
Anonymous
July 03, 2012
Please bring data-uri for HTML, CSS, Javascript and other plain text in Internet Explorer connect.microsoft.com/.../data-uri-seems-to-require-base64-encoding
Anonymous
July 03, 2012
I'll just put this here: WebGL.
Anonymous
July 04, 2012
The reason microsoft has a dead fish as the main desktop theme in Windows 8 is because IE is DOA No WebGl, Low HTML 5 scores, more web fragmentation by not updating all windows versions. True Story!
Anonymous
July 04, 2012
The comment has been removed
Anonymous
July 04, 2012
@ Microsofts Thursday July 5th Agenda The IE Blog does not need fixing! It includes an intelligence test that you have obviously failed!
Anonymous
July 04, 2012
I HOPE IE10 IS MUCH LIGHTER THEN IE9 OR ELSE I AM GOING TO USE FIREFOX OR GOOGLECHROOME. I WANT A LIGHT IE10 AND NOT SLOW AND HEAVY
Anonymous
July 05, 2012
That article is a very interesting read, but why didn't you talked about the HTML5 WebGL tests you've written? I'm nervous that HTML5 WebGL support in IE10 might not be as good as in all the other browsers. And that would be a farce for a browser that pretends to adopt web standards.
Anonymous
July 05, 2012
Uh IE9 is really fast...
Anonymous
July 05, 2012
The comment has been removed
Anonymous
July 06, 2012
@ie team why IE9 for one webpage downloading become freeze ? firefox do this work very easily in background ! freeze all tab pages only for one page download ! :(
Anonymous
July 06, 2012
The comment has been removed
Anonymous
July 07, 2012
The comment has been removed
Anonymous
July 08, 2012
That's Microsoft for you! They haven't changed one bit and won't till people like large stock holders tell them that there unhappy with the bull that there trying to pull on the public and they will not do that till sales drop! So stop BUYING! Because they will keep ignoring us till there forced by economics to change
Anonymous
July 08, 2012
The comment has been removed
Anonymous
July 09, 2012
The comment has been removed
Anonymous
July 09, 2012
The comment has been removed
Anonymous
July 09, 2012
You typically need to buy four Windows licenses, because you need to test in four versions on Internet Explorer.
Anonymous
July 10, 2012
@@Zoya, last time I checked Windows 8 RP is free to download and use for 14 months. IE10 GA will come to Windows 7. Moreover, Windows XP, Windows Vista and Windows 7 users will get Windows 8 upgrade for only $39,99 (windowsteamblog.com/.../upgrade-to-windows-8-pro-for-39-99.aspx).
Anonymous
July 10, 2012
@Zoya That is not the argument. Some people can´t and will use Windows 8
Anonymous
July 10, 2012
The comment has been removed
Anonymous
July 10, 2012
The comment has been removed
Anonymous
July 10, 2012
@Jens, what argument? This blog post is about IE10.. so we are NOT talking about people who will not be using IE10. IE10 will be available for Windows 7 and Windows 8. If you aren't planning to use Windows 7 or Windows 8 then you don't belong here. In conjunction to that, Windows XP and Vista users who are willing to upgrade will get Windows 8 for 40 bucks! @Who-Cares#, What????? "They haven't changed one bit?" What kind of benchmarks can you provide to support your argument? Are you asserting technically standpoint or is just a regular whining? Your comment is stinkier than your English. You know that right?
Anonymous
July 10, 2012
The comment has been removed
Anonymous
July 10, 2012
@Nolan: The only one that doesn't work on a test page I have here is prompt(). I'm told that's by design. I'm testing with Windows 8 Release Preview; perhaps the others didn't work in the Consumer Preview. Anyway, I'll inform MSDN about the need to correct the documentation. Thanks for calling it out.
Anonymous
July 10, 2012
The comment has been removed
Anonymous
July 10, 2012
The comment has been removed

Partager via