We just touched the surface of the problem area in today’s announcement, so we’d like to dive a little further with you in this post since this truly was an interesting problem.
How does Sway automatically crop images?
First a little background. When we built Sway’s “designer in a box” code, we created an algorithm that we call “Smart Cropping,” and this has been working silently in the background since Sway Preview started in October 2014. This algorithm evaluates pictures in order to understand what is interesting and what isn’t so Sway can create well-composed image crops.
When you drop a picture into your Sway, there’s a lot going on behind the scenes. Sway identifies things like faces, invariant regions (what’s uniform), horizons and other strong lines, and more, as well as how all of these elements play together. Then Sway looks at the context in which the author is using the picture. Is it in a heading? A title? A full-bleed (“intense”) image? Does it have to fit within an interactive element? Sway then makes a choice of what it will show from that image (and how) given the unique constraints for your device and the chosen layout and style. Sway could consider things like the “rule of thirds” or the overall composition in order to deliver beautiful content.
Smart Cropping can’t stand alone
Of course, Smart Cropping alone isn’t good enough. For starters, the algorithm itself can always be improved. (And we’ll continue to make it better!) In fact, if you ever notice that your image isn’t cropped to what you thought was important and you don’t understand why, we’d appreciate it if you left us some feedback on UserVoice or by tweeting us. This way we can understand what we need to do to make Sway’s Smart Cropping even smarter.
Another reason Smart Cropping can’t stand alone is the subjective nature of what you want to include in your Sway. We could have the best algorithm in the world, but Sway still wouldn’t know the difference between your child and a random child sitting next to each other (…at least not yet?). (In fact, one of our own team members was frustrated to see that Smart Cropping had excluded his daughter from an image he had taken of her playing soccer, focusing instead on a different player!)
Additionally, we made some stylistic decisions to include extreme horizontal crops for some of our heading cards. This means Sway may knowingly make a less than ideal cropping decision to achieve the stylistic effect of the design you’ve chosen. We have received a lot of feedback from exasperated users outlining this exact problem, asking why we wouldn’t give them a way to manually reposition the crop.
Unfortunately, just letting users manually reposition the crop in Sway isn’t as intuitive as it may seem. Let’s think about this a little bit… When you manually crop a picture (or even reposition it within the current constraint, like you might with a repositioning feature), you very specifically set length and width dimensions. But what does this mean when you hit Remix!, change the style or layout, move your picture to a different context (like from a title card to a heading card), or view your Sway on a different device? Yes, we understand that sometimes you just want to manually crop your image (and we are considering adding this as an override), but when you abstract the problem, what is the user truly trying to achieve in the majority of cases? From our user studies and from your feedback, we believe that the true issue was that what was important wasn’t in focus, not that the user wanted to fine-tune the crop (and worry about the myriad of devices and what their crop would look like on each one). And getting to the heart of what you want and intended, versus asking you to always specify the exact design details, is what Sway is all about.
So why Focus Points?
We considered a lot of different solutions, some that were very similar to manual cropping and others (like Focus Points) that were very different. We decided to move forward with the Focus Points approach because of its overall simplicity and because of how much more valuable it is for Sway to understand what’s important to you. Just imagine what else Sway could do with an understanding of what’s important to you, not just what settings you chose! By knowing, say, what two discrete portions of the image you care about, Sway could choose a better “unimportant” part of the image for text placement; or create different crops to induce parallax and bring them together; or animate between the two regions to make sure they’re both included. The potential is endless!
It’s tackling challenging problems like this, to bring you a product that you love that is unlike anything you’ve used before, which makes working on Sway so exciting. We’ll continue to listen to your feedback and refine the experience, so please get in touch! We’re listening on the other end whenever you have suggestions or feedback for us. Sway on!
—Sway team, @Sway
Source: Office Blog