Journal tags: iphone

8

Web Audio API update on iOS

I documented a weird bug with web audio on iOS a while back:

On some pages of The Session, as well as the audio player for tunes (using the Web Audio API) there are also embedded YouTube videos (using the video element). Press play on the audio player; no sound. Press play on the YouTube video; you get sound. Now go back to the audio player and suddenly you do get sound!

It’s almost like playing a video or audio element “kicks” the browser into realising it should be playing the sound from the Web Audio API too.

This was happening on iOS devices set to mute, but I was also getting reports of it happening on devices with the sound on. But it’s that annoyingly intermittent kind of bug that’s really hard to reproduce consistently. Sometimes the sound doesn’t play. Sometimes it does.

I found a workaround but it was really hacky. By playing a one-second long silent mp3 file using audio, you could “kick” the sound into behaving. Then you can use the Web Audio API and it would play consistently.

Well, that’s all changed with the latest release of Mobile Safari. Now what happens is that the Web Audio stuff plays …for one second. And then stops.

I removed the hacky workaround and the Web Audio API started behaving itself again …but your device can’t be set to silent.

The good news is that the Web Audio behaviour seems to be consistent now. It only plays if the device isn’t muted. This restriction doesn’t apply to video and audio elements; they will still play even if your device is set to silent.

This descrepancy between the two different ways of playing audio is kind of odd, but at least now the Web Audio behaviour is predictable.

You can hear the Web Audio API in action by going to any tune on The Session and pressing the “play audio” button.

Web Audio API weirdness on iOS

I told you about how I’m using the Web Audio API on The Session to generate synthesised audio of each tune setting. I also said:

Except for some weirdness on iOS that I had to fix.

Here’s that weirdness…

Let me start by saying that this isn’t anything to do with requiring a user interaction (the Web Audio API insists on some kind of user interaction to prevent developers from having auto-playing sound on websites). All of my code related to the Web Audio API is inside a click event handler. This is a different kind of weirdness.

First of all, I noticed that if you pressed play on the audio player when your iOS device is on mute, then you don’t hear any audio. Seems logical, right? Except if using the same device, still set to mute, you press play on a video or audio element, the sound plays just fine. You can confirm this by going to Huffduffer and pressing play on any of the audio elements there, even when your iOS device is set on mute.

So it seems that iOS has different criteria for the Web Audio API than it does for audio or video. Except it isn’t quite that straightforward.

On some pages of The Session, as well as the audio player for tunes (using the Web Audio API) there are also embedded YouTube videos (using the video element). Press play on the audio player; no sound. Press play on the YouTube video; you get sound. Now go back to the audio player and suddenly you do get sound!

It’s almost like playing a video or audio element “kicks” the browser into realising it should be playing the sound from the Web Audio API too.

This was happening on iOS devices set to mute, but I was also getting reports of it happening on devices with the sound on. But it’s that annoyingly intermittent kind of bug that’s really hard to reproduce consistently. Sometimes the sound doesn’t play. Sometimes it does.

Following my theory that the browser needs a “kick” to get into the right frame of mind for the Web Audio API, I resorted to a messy little hack.

In the event handler for the audio player, I generate the “kick” by playing a second of silence using the JavaScript equivalent of the audio element:

var audio = new Audio('1-second-of-silence.mp3');
audio.play();

I’m not proud of that. It’s so hacky that I’ve even wrapped the code in some user-agent sniffing on the server, and I never do user-agent sniffing!

Still, if you ever find yourself getting weird but inconsistent behaviour on iOS using the Web Audio API, this nasty little hack could help.

Update: Time to remove this workaround. Mobile Safari has been updated.

The magical and the mundane

The iPhone—and by extension, the smartphone—is a decade old. Ian Bogost has written an interesting piece in The Atlantic charting our changing relationship with the technology.

First, it was like a toy dog:

A device that could be cared for, and conspicuously so.

Then, it was like a cigarette:

A nervous tic, facilitated by a handheld apparatus that releases relief when operated.

Later, it was like a rosary:

Its toy-dog quirks having been tamed, its compulsive nature having been accepted, the iPhone became the magic wand by which all worldly actions could be performed, all possible information acquired.

Finally, it simply becomes …a rectangle.

Abstract, as a shape. Flat, as a surface. But suggestive of so much. A table for community. A door for entry, or for exit. A window for looking out of, or a picture for looking into. A movie screen for distraction, or a cradle for comfort, or a bed for seduction.

Design dissolves in behaviour. This is something that Ben wrote about recently in his excellent Slapdashery series: “Everything’s amazing and nobody’s happy.”

Technology tweaks our desire for novelty; but as soon as we get it we’re usually bored. There are no technologies that I can think of that haven’t become mundane.

This is something I touched on in my talk last year at An Event Apart. There’s a thread throughout the talk about Arthur C. Clarke, and of course I quote his third law:

Any sufficiently advanced technology is indistinguishable from magic.

I propose an addendum to that:

Any sufficiently advanced technology is indistinguishable from magic at first.

The magical quickly becomes the mundane. That’s exactly the point that Louis CK is making in the piece that Ben references.

Seven years ago Frank wrote his wonderful essay There Is A Horse In The Apple Store:

I have a term called a “tiny pony.” It is a thing that is exceptional that no one, for whatever reason, notices. Or, conversely, it is an exceptional thing that everyone notices, but quickly grows acclimated to despite the brilliance of it all.

We are surrounded by magical tiny ponies. I mean, just think: right now you are reading some words at a URL on the World Wide Web. Even more magically, I just published some words at my own URL on the World Wide Web. That still blows my mind! I hope I never lose that feeling.

Someday

In the latest issue of Justin’s excellent Responsive Web Design weekly newsletter, he includes a segment called “The Snippet Show”:

This is what tells all our browsers on all our devices to set the viewport to be the same width of the current device, and to also set the initial scale to 1 (not scaled at all). This essentially allows us to have responsive design consistently.

<meta name="viewport" content="width=device-width, initial-scale=1">

The viewport value for the meta element was invented by Apple when the iPhone was released. Back then, it was a safe bet that most websites were wider than the iPhone’s 320 pixel wide display—most of them were 960 pixels wide …because reasons. So mobile Safari would automatically shrink those sites down to fit within the display. If you wanted to over-ride that behaviour, you had to use the meta viewport gubbins that they made up.

That was nine years ago. These days, if you’re building a responsive website, you still need to include that meta element.

That seems like a shame to me. I’m not suggesting that the default behaviour should switch to assuming a fluid layout, but maybe the browser could just figure it out. After all, the CSS will already be parsed by the time the HTML is rendering. Perhaps a quick test for the presence of a crawlbar could be used to trigger the shrinking behaviour. No crawlbar, no shrinking.

Maybe someday the assumption behind the current behaviour could be flipped—assume a website is responsive unless the author explicitly requests the shrinking behaviour. I’d like to think that could happen soon, but I suspect that a depressingly large number of sites are still fixed-width (I don’t even want to know—don’t tell me).

There are other browser default behaviours that might someday change. Right now, if I type example.com into a browser, it will first attempt to contact http://example.com rather than https://example.com. That means the example.com server has to do a redirect, costing the user valuable time.

You can mitigate this by putting your site on the HSTS preload list but wouldn’t it be nice if browsers first checked for HTTPS instead of HTTP? I don’t think that will happen anytime soon, but someday …someday.

Orientation and scale

Paul Irish, Divya Manian and Shi Chuan launched Mobile Boilerplate recently—a mobile companion site to HTML5 Boilerplate.

There’s some good stuff in there but I was a little surprised to see that the meta viewport element included values for minimum-scale=1.0, maximum-scale=1.0, user-scalable=no:

Setting user-scalable=no is pretty much the same as setting minimum-scale=1.0, maximum-scale=1.0. In any case, I’m not keen on it. Like Roger, I don’t think we should take away the user’s right to pinch and zoom to make content larger. That’s why my usual viewport declaration is:

Yes, I know that most native apps don’t allow you to zoom but I see no reason to replicate that failing on the web.

But there’s a problem. Allowing users to scale content for comfort would be fine if it weren’t for a bug in Mobile Safari:

When the meta viewport tag is set to content="width=device-width,initial-scale=1", or any value that allows user-scaling, changing the device to landscape orientation causes the page to scale larger than 1.0. As a result, a portion of the page is cropped off the right, and the user must double-tap (sometimes more than once) to get the page to zoom properly into view.

This is really annoying so Shi Chuan set about fixing the problem.

His initial solution was to keep minimum-scale=1.0, maximum-scale=1.0 in the meta viewport element but then to change it using JavaScript once the user initiates a gesture (the gesturestart event is triggered as soon as two fingers are on the screen). At the point, the content attribute of the meta viewport element gets updated to read minimum-scale=0.25, maximum-scale=1.6, the default values:

var metas = document.getElementsByTagName('meta');
var i;
if (navigator.userAgent.match(/iPhone/i)) {
  document.addEventListener("gesturestart", gestureStart, false);
  function gestureStart() {
    for (i=0; i<metas.length; i++) {
      if (metas[i].name == "viewport") {
        metas[i].content = "width=device-width, minimum-scale=0.25, maximum-scale=1.6";
      }
    }
  }
}

That works nicely but I wasn’t too keen on the dependency between the markup and the script. If, for whatever reason, the script doesn’t get executed, users are stuck with an unzoomable page.

I suggested that the script should also set the initial value to minimum-scale=1.0, maximum-scale=1.0:

var metas = document.getElementsByTagName('meta');
var i;
if (navigator.userAgent.match(/iPhone/i)) {
  for (i=0; i<metas.length; i++) {
    if (metas[i].name == "viewport") {
      metas[i].content = "width=device-width, minimum-scale=1.0, maximum-scale=1.0";
    }
  }
  document.addEventListener("gesturestart", gestureStart, false);
}
function gestureStart() {
  for (i=0; i<metas.length; i++) {
    if (metas[i].name == "viewport") {
      metas[i].content = "width=device-width, minimum-scale=0.25, maximum-scale=1.6";
    }
  }
}

Now the markup still contains the robust accessible default:

…while the script takes care of initially setting the scale values and also updating them when a gesture is detected. Here’s what’s happening:

  1. By default, the page is scaleable because the initial meta viewport declaration doesn’t set a minimum-scale or maximum-scale.
  2. Once the script loads, the page is no longer scalable because both minimum-scale and maximum-scale have been set to 1.0. If the device is switched from portrait to landscape, the resizing bug won’t be triggered because scaling is disabled.
  3. When the gesturestart event is detected—indicating that the user might be trying to scale the page—the minimum-scale and maximum-scale values are updated to allow scaling. At this point, if the device is switched from portrait to landscape, the resizing bug will occur because the page is now scaleable.

Jason Weaver points out that you should probably detect for iPad too. That’s a pretty straightforward update:

if (navigator.userAgent.match(/iPhone/i) || navigator.userAgent.match(/iPad/i))

Mathias Bynens updated the code to use querySelectorAll which is supported in Mobile Safari. Here’s the code I’m currently using:

if (navigator.userAgent.match(/iPhone/i) || navigator.userAgent.match(/iPad/i)) {
  var viewportmeta = document.querySelector('meta[name="viewport"]');
  if (viewportmeta) {
    viewportmeta.content = 'width=device-width, minimum-scale=1.0, maximum-scale=1.0';
    document.body.addEventListener('gesturestart', function() {
      viewportmeta.content = 'width=device-width, minimum-scale=0.25, maximum-scale=1.6';
    }, false);
  }
}

You can try it out on Huffduffer, Salter Cane, Principia Gastronomica and right here on Adactio.

Right now there’s still a little sluggishness between the initial pinch-zoom gesture and the scaling; the scale values (0.25 - 1.6) don’t seem to take effect immediately. A second pinch-zoom gesture is often required. If you have any ideas for improving the event capturing and propagation, dive in there.

Update: the bug has been fixed in iOS 6.

Accessimobility

When the original came out, it was pretty impressive. Every subsequent iteration has featured improvements of varying levels of impressiveness. The latest version, though, has bowled me over.

I’m not talking about faster speeds, bigger storage, or any new fangled gizmos or geegaws. I’m talking about VoiceOver in the iPhone.

Watch the video to get the low-down. Then read this first-hand account of using an accessible touch-screen device.

That’s quite a design challenge: an accessible touch-screen device! I doff my hat in the general direction of the Apple engineers who rose to this challenge.

Speaking of exciting developments in the world of accessibility…

The second Accessibility 2.0 conference will be taking place in London on the 22nd of this month. It was a cracking event last year and, judging by the line-up, this year is going to be a winner too. Grab a ticket now.

iPhone, uPhone, we all scream for iPhone

Like everyone else in the geekosphere, I was vicariously attending MacWorld through all those Mac rumour sites as well as Twitter. Unsurprisingly, the iPhone has a lot of people excited.

Jason Kottke got so excited, he immediately whipped up a cardboard scale model of the iPhone. Jeff Croft is really excited that the iPhone is running Leopard.

A lot of people are talking about how to get their heads around this thing. Is it a phone or is it a PDA? John Allsopp wouldn’t mind losing the phone functionality altogether.

Mike Davidson thinks that the iPhone is worthy of the moniker Steve’s Amazing New Device. He must be pretty pleased that his prediction that Apple would no longer be just a computer company became reality with the official change of the company’s name. Khoi Vinh, on the other hand, will be disappointed to hear that the iPhone does indeed use iTunes to do its syncing.

I spent part of the keynote chanting Get to the web browsing! Get to the web browsing! Then Steve Jobs got to the web browsing… with expando-Safari.

Dave Hyatt may be slightly biased but he thinks that this may spell the beginning of the end for a separate mobile web. Dan Cederholm is pretty impressed too. Cameron Moll, on the other hand, believes that the iPhone won’t revolutionise the mobile web landscape for most people. Brian Fling disagrees. He thinks the impact of the iPhone will be huge.

Bursting Apple’s reality distortion field with Photoshop, Jon Hicks demonstrates the problem with the iPhone’s screen. Roger Johansson also throws a cold bucket of reality on proceedings when he asks where the tactile feedback is supposed to come from when there’s no keyboard. That’s a valid concern according to David Pogue’s hands-on experience.

The biggest downer probably won’t be anything to do with the device itself but the lock-in with some crappy provider, as Dave Shea explains. That’s still not enough to dissuade Jason Santa Maria from wanting an Apple mobile device.

I met up with Brighton’s own mobile guru, Tom Hume, for lunch today. He’s taking a pragmatic and somewhat pessimistic approach with his thoughts on the iPhone.

Only time will tell how Apple’s baby will fare once its released into the wild. For some historical perspective, I invite you to cast your mind back to a Slashdot article from 2001 announcing the iPod.

New Year’s Resolution

In a comment on Roger’s post about fixed and liquid layouts, Cameron wrote:

This issue seems to generate a heated debate every time it’s mentioned. I imagine one could pen an article with the headline “Fluid or fixed?” and nothing else, and yet dozens of comments would inevitably appear.

But rather than use that title, I couldn’t resist borrowing a pun from Andy, prompted by a post from Scrivs called What Resolution Will You Design for in 2007? (a classic example of the fallacy of many questions).

Now, firstly, we need to draw a distinction between monitor size and browser size. In other words, the difference between screen resolution and the viewport size:

There’s a real danger in thinking that “the numbers speak for themselves.” Numbers don’t speak for themselves; numbers need to be interpreted.

The numbers clearly show that monitor sizes and resolutions are getting bigger. The most common interpretation of that is more and more people have bigger displays. But an equally valid interpretation of the numbers is the range of displays is bigger than ever. It’s a subtle but important distinction. One interpretation focuses solely on the size of the highest numbers; the other interpretation focuses on the range of all the numbers.

The way I see it, the range is growing at both ends of the spectrum. Yes, desktop monitors are getting wider (though that doesn’t mean that viewports get any wider above a certain size) but handheld and gaming devices are likely to remain at the lower end of the scale. The Wii, for example, has a resolution of 640 x 480.

Mind you, the iPhone turns the whole question on its head with its scalable browsing. At MacWorld, Steve Jobs demonstrated this by visiting the New York Times, an unashamedly wide fixed-width website. On the Apple site, Wikipedia—a liquid layout— is shown fitting nicely on the display. The iPhone deals with both. Still, rather than letting my liquid layouts scale down to the iPhone’s width, I should probably start putting a min-width value on the body element.

Speaking of which…

A common argument against using liquid layouts is the issue of line lengths. On the face of it, this seems like a valid argument. Readability is supremely important and nobody likes over-long line lengths. But it’s not quite as simple as that when it comes to readability on screen compared to print, as Richard noted:

Surprisingly, I find short line lengths tiresome on screen; I don’t really subscribe to the empirical prescription of 7–10 words per line for comfortable reading. Most novels have 10–15 words per line and I think the upper region of that range is more appropriate for screen.

In any case, the idea that liquid layouts automatically means long line lengths on large screens is, I feel, a misconception. The problem is that a lot of the examples of liquid layouts aren’t very good and line lengths do expand without limit. But it doesn’t have to be that way.

In my opinion, the most important addition to Internet Explorer 7 is the max-width property. It means that we can now really start to look at creating fluid layouts within defined parameters, as demonstrated by Cameron in Andy’s book. In fact, I think we’re just scratching the surface of what’s possible in creating seamless adaptive layouts (and, more importantly, seamless adaptive page elements) using the dual power of max-width and min-width.

That still leaves Internet Explorer 6 and below. Should they get unbounded fluid layouts or should they get a fixed width fallback? The second is certainly an option using conditional comments, which is the Microsoft-approved way of dealing with rendering inconsistencies. I think that the lack of support for max-width certainly falls into that category. Call it transcending CSS if you will; I call it routing around damage on the designer’s network.

I want to hear what you have to say… if you’ve got something new to say. Let’s not just rehash the same old arguments that would inevitably appear had I simply asked “Fluid or fixed?”