Voice Control hands-on: Deep OS navigation for iPads, iPhones, and Macs

By | June 11, 2019

When Apple debuted Siri with 2011’s iPhone 4S, it effectively stopped work on a useful but more limited feature that had arrived with the iPhone 3GS only two years earlier: Voice Control, a speech recognition system that enabled users to make phone calls and play songs by just saying them to early iPhones — even without internet connectivity. Apple barely said anything about Voice Control after Siri debuted, and notably didn’t enable the feature to work whenever Siri couldn’t make an internet connection, but it kept it buried in iOS for users who needed it.

Now Voice Control is making a huge comeback this year across iOS, iPadOS, and macOS, where it’s effectively merging and expanding upon features that were previously floating in one operating system or the other. As of iOS/iPadOS 13 and macOS Catalina, Voice Control will let users do four key things without needing to rely on an internet connection or Siri.

  • Navigate through system spaces and apps, including going backwards to prior screens.
  • Interact with screen items using names and/or numbers laid out on a grid
  • Dictate, select, and edit text, including moving the insertion cursor, and
  • Interact with the device’s OS-level features, including volume adjustment, lock screen controls, screenshots, and making emergency SOS calls.

But, it should be noted, these features are all turned off and buried by default — hiding within the device’s Accessibility menus. There, you’ll have to look within the Mac’s Motor (Dictation) section, or an iPad/iPhone’s Physical and Motor (Voice Control) category. Once you open the header, you’ll see that there’s a lotthat the feature can control.

Each of these headers has at least a full page of different commands that Apple’s devices can now listen for and understand without the need to say “Hey Siri” ahead of time. Some of the commands have multiple natural phrases that will work (“Open” or “Show”), so you don’t have to learn just one word to evoke them.

Moreover, each command can be set to enabled or disabled, as well as to work either immediately when recognized, or after confirmation. If confirmation is required, you’ll see an on-screen dialog box asking you whether to “Execute” the command or not, an annoyance that’s turned off by default but might be useful for certain accidentally triggerable phrases.

There’s also a custom command creation tool that lets you type an evoking phrase, set an action based on (a) text insertion, (b) data pasting, (c) running a shortcut, (d) running a custom gesture, or (e) playing back recorded commands — including a sub-tool that lets you use voice controls to begin and stop recording those commands. You can choose virtually any application as your specific target for the custom command, as well as making the command universal across all apps.

By comparison with some of Apple’s more threadbare new feature introductions, where all the implementing functionality is hidden away and the feature “just works” as well as it can initially work with only a handful of settings, Voice Control feels like it’s made to be tweaked by users. There’s a Vocabulary feature so you can teach it new words that will be handy with apps or features, a Hints system that pops up a little gray box on the screen to guide you with phrases you might find useful, and a lot of customization for how assistive elements appear and disappear.

In the images above and below, you can see how Voice Control lets you assign numbers either to every item on screen or to a grid representing areas of the screen, different systems that could be useful for different users or apps. The feature is called Overlay, and can also be used to assign a more visually intrusive (and likely to be duplicative) word description for every item. Each can be set to dim to your desired level of opacity after a preferred number of seconds and even fractions of a second.

The scope of built-in commands is impressively large. Across both platforms, Apple pre-programmed a ton of text selection, editing, and related dictation tools, drag and drop controls, Force or 3D Touch selection (on supported devices), and sleep/wake up commands.

When the feature’s on, it does things pretty much automatically as you request them without any need for a button press or another vocal trigger. In keeping with the latest iOS/iPadOS UI paradigm, saying “3D Touch App Store” from the Home screen of an iPhone XS will make all the apps on screen jiggle, while saying “Long press App Store” will bring up the store’s former context-specific menu. On a text document where the word Geneva appears twice, saying “Select the word Geneva” will select the closest one, place a number next to each instance, and prompt you to “Say number of desired item,” responding immediately to your spoken response of “one” or “two.”

Many of the commands are the same between iOS, iPadOS, and macOS, but there are a few macOS-specific commands (Release mouse, triple click) and a lot of iOS/iPadOS-specific ones, including all of the features in the latter devices’ “device” and “accessibility” submenus. For now, only iOS and iPadOS devices get Voice Control’s classic music and phone control capabilities, which include dialing people at given devices, identifying songs, and playing by song, artist, or playlist. iOS and iPadOS users can also verbally call for the screen to be locked, Apple Pay to be opened, analytics data to be gathered, and for a device reboot. We’ll have to wait and see whether Apple brings some of these features to macOS Catalina this year.

At a time when Google is bringing more of its Assistant functionality directly onto the device without requiring an internet connection, it’s hard to understand why Apple wouldn’t just integrate these features directly into Siri and allow as many of them to be used offline as possible, by as many people as possible. Some would be incredibly handy for any user to have available regardless of an internet connection, and regardless of broader accessibility needs.

It’s possible that Voice Control is a testing ground for an offline Siri mode, and that the feature will become more widely available in the years to come. But it’s also possible that Apple really just intends for the new features to be there for users with special needs, at least for the time being. Regardless, I’d recommend peeking into the feature within the Accessibility settings when you get a new iOS, iPadOS, or macOS release installed; it’s impressive in scope and performance, with plenty of opportunity to improve the way everyone interacts with their favorite devices.

[“source=venturebeat”]