Firefox Needs Content-Aware Image Resizing
The new feature called “content-aware image resizing” in Photoshop is amazing. There is a promotional video up on the Adobe site that’s really fun to watch. For example, they make a Volkswagen bus more “economical” (mind you, while keeping the wheels round):
The technology behind it is based on research from an Isreali research group. That group put a video up on youtube in 2007 already:
It’s a little more technical, but no less impressive, so all of you geeks who wonder how it actually works should watch this as well.
I can tell you one thing: I want this in Firefox’s page resizing code. Sadly, I assume it is strictly patented and Adobe will probably have made sure to have some sort of exclusive deal on it.
Update: A commenter points out, there is an open source plugin for the GIMP that does content-aware resizing–already since 2007. Thank you, Gandalf!
It shouldn’t be too hard as we already have Gimp plugins since 2007 (!!!) for that:
http://liquidrescale.wikidot.com/
http://arstechnica.com/journals/linux.ars/2007/09/10/content-aware-image-resizing-algorithm-implemented-as-gimp-plugin
I like it when commercial software has to catch up
How about starting with allowing websites to opt-out of bilinear filtering when upscaling. Pixel art doesn’t look so great blurry.
Also downscaling that wasn’t nearest neighbor would be awesome!
Heck, create an extension for it. See how useful it’d be…
I don’t know if this “feature” really is what we are looking for. Unlike simple scaling, you remove content and therefor remove the authenticy of the picture. For example look at the video above and see how they remove people from the beach…
Thank you guys for the comments!
@Pete: Obviously, you only remove content if you want to remove content. Did you notice they marked the people before they were removed? They make the assumtion that the content of a picture is, where “something happens”. So it removes the “boring” parts of the picture unless you tell it otherwise.
Ok. The beach is a bad example. But look at your picture. The user can also think: Hey, where can I buy this nice blue car. So, authenticy is not given anymore. So, my point is that the user should be aware that the content is modified. But I also agree it’s a great feature when it comes to boring pieces, like sand at the beach vs. people. Question is when are parts really boring?
Pretty sure there’s a bug already filed on this, by the way. But if not, you should consider doing it.
I get your point, Pete. It is not certain how much, if any, benefit this is going to give to the user. I am with commenter Havvy (#4) here: Making an extension could show how useful, or not, the feature would be in the browser world.
My first response is that there is no benefit to the end user.
But after a second thought, it could potentially be useful in situations where we know the web page is going to be resized/zoomed. i.e. mobile users
As long as the content-aware resizing is explicitly specified by the content provider.
So when you go to cnn.com, they could take a picture with multiple individuals and ‘protect’ one of them so they appear more close-upped when viewed on a mobile device.
But honestly, if I were cnn (or anyone else) I would want control over how my images were rendered (the potential for embarrassing images is high) so I’d rather just post alternate images that would be shown to mobile users at particular zoom levels.
Nope, on third thought I’m not seeing the benefit to users or providers.
Not saying there isn’t a scenario, just not seeing one yet.
I think this is not a good idea to have this feature in Firefox by default. It seem pretty useful on landscapes pictures, but as we can see in the second videos, the cube are loosing this original shape. The behavior is too unpredictable too use this feature automatically.
To enable it the author of the page should be able to specify the resize method he want with something like but i don’t think this is possible. An extension may be a good idea.
[…] a more intelligent tool to get excited about: content-aware image resizing in Photoshop 4 (hat tip: Frederic Wenzel). This feature lets you resize an image while respecting its subject and complexity by scaling the […]
[…] Fred and Borris both recently blogged about intelligent image resizing. This previously came up about a year ago, probably when this research video started making the rounds: […]
The algorithm is very simple to implement; we had to implement it for an undegraduate CS class, and it took about an hour in Matlab to write it and a few hours in C++. The real issue is that it selects lines of deletion based on “minimal energy”. Given a high-variance area of a web page, however, it may elect to start taking lines out through font glyphs. The name “content-aware” is somewhat of a misnomer, as the algorithm will have no problem making text illegible. It will also preferentially remove whitespace, which may again reduce readability.
The article[1] talk about “multi-size” images.
If some program (say, Gecko) supports multi-size images, the author is able to specify some images for some display devices or sizes of the screen.
1 – link to the article (22MB, English, technical) on the liquid rescale gimp plugin homepage.
Someone could write a specification for embedding the pixel seam removal order bitmaps (as seen in the second video) as a chunk into existing image formats. From what I know about PNG this would be fairly easy. Mozilla did something similar for the APNG specification. If this was done then compliant User Agents would know how to resize those images, and non-compliant UAs would show images like they do today.
I started an implementation in pure JavaScript using next Firefox 3.1 / 3.5 features. You can read it here : http://labs.pimsworld.org/2009/04/firefox-native-content-aware-image-resizing/
Stephane
Stephane: Wow, that is great, well done! You should make an add-on out of it