Search: The next five years.

[Mike Grehan]

Well, it’s been a very long time since I last visited these forums. 2005 I think!

So, for a guy who never does forums... What are you doing here, Mike?

Good question :-)

I’m kind of making a place holder for a thread I really want to start on Monday after my ClickZ column is published. Once people have had a chance to read it, then it’ll be quite clear why I wanted to open up a discussion over here.

Just a little advance about the column. I had to write a piece about the future of search over the next five years for something else last week. And that, tied to a couple of other projects (including the new book I’m writing) lead me to write my ClickZ column entirely about the future and how search will change.

All my usual topics, including the diminishing value of SEO and the switch to digital asset management and optimization. And the changing face of conventional public relations.

On page techniques from 1994 to 1998. The birth of Google and the switch to link building as opposed to keyword stuffing. Universal/blended search results and social media marketing... All included.

Just to be clear, what I’d like to do is invite as many readers and forum members to get involved in a lengthy dialogue not just a short SEO is dead debate, as that has had its day.

So, back here Monday to pick it up live, as it were.

Nice to be back I see the old place has a had a lick of paint...

[Kevin Newcomb]

You're a tease, Grehan. :D

You come in here and get us all excited about a deep and meaningful discussion, and then tell us we have to wait until Monday? No fair.

[Mel66]

Good to see you here, Mike. Can't wait to read the article!

Melissa

[AussieWebmaster]

Good prepublicity... are we testing how much of a spike in first day reads you can get by prepub on forums?

[Mike Grehan]

Excuse me Mr Aussie[NewYork}Webmaster...

Good prepublicity... are we testing how much of a spike in first day reads you can get by prepub on forums?

I haven't posted here for three years... You're one of about two people here who knows me! So, my open rate increases by at least one!

I had to have somewhere to link to in my column. So I needed a place holder - get it?

You owe me beer, mate. Next SEMPO New York gig... Damn! The beer's free there :confused:

[AussieWebmaster]

lol I see the light.... I will gladly buy you a beer any time.... if in NYC next week am having dinner with Matt and Kevin

[Kevin Newcomb]

Here's Mike's column: The Future of SEO (http://www.clickz.com/3629530). It's a look at search as it was yesterday, today, and tomorrow.

[Mike Grehan]

Okay, so I’m at a conference in London and pretty busy with the usual stuff. We actually just had a great conversation in the bar about the future of search and where it’s all going. It’s fascinating to get the views of people who aren’t into search all day long, like many in this forum.

Anyway, my ClickZ column is over here (http://www.clickz.com/showPage.html?page=3629530). As Kevin mentioned above.

If you can’t be bothered to read it all, no worries. Basically, I’m throwing out some of my thoughts, and I’m just interested if they make a connect with anyone in this forum.

I have a passion for trying to get a better understanding of information retrieval on the web. The classic approaches of Gerard Salton’s vector space model and the Kleinberg/Page adoption of social network analysis/citation analysis has been the mainstay of search for a while. But in a new era, an era where it’s not a bandwidth issue and connections are always on, search engines can now provide so much more meaningful and relevant data in their results (and a much richer experience).

Just analyzing text on a page to try and discover the “gist’ of a page limits the results to static, text based documents. And it’s difficult to determine true authority, even with weighting. As I’ve mentioned before, if a music student writes an article about Beethoven’s fifth symphony and a world class composer, such as Andre Previn writes an article: who’s is likely to be the most authoritative?

For me. I reckon it’s going to be Andre Previn. But if the keywords in the document are pretty much the same, how would a search engine, purely looking at text, know?

So we look to linkage data. This is important, but it’s a little elitist, in that, if a link counts as a vote, then only people with web pages can vote. What about the millions of end users who have a voice and opinion but don’t have web sites – how do they vote?

It’s only been with their clicks and their behavioral patterns. But now that social media sites provide an opportunity for end users to cast their votes via bookmarking and tagging and rating, all of a sudden there are some very strong signals going on well beyond the community with web sites and the ability to let Google know who they’re linking to.

And this is such an interesting advancement in social network analysis. Now people without web sites can upload photo sets to Flickr and tag and share them. They can do the same with videos at YouTube. They can go to Google Base and open an account and start using the local business center to promote their small business with maps and don’t need a web site.

Yahoo! and Google finance offers a way to let large businesses get a stock quote listing with their results. Popular video clips are popping up in all search engines. There are now myriad ways of sending signals to search engines. And they go well beyond some very dated keyword density analysis, basic submissions to vertical directory sites and starting a blog.

To recap. Should search engine ranking algorithms continue to be based only on the data they have about people who happen to have web sites and therefore have text pages and can link to others. Or is the voice and opinion of the end user now being heard much more clearly?

Do I really think that a number one result at Google for the term – blue widgets – is a fair result if only people who have web sites can vote for its top ranking position? Or should the millions of people who use blue widgets and don’t have a web site also be able to have some influence on that ranking?

I am sincerely interested in hearing what other people have to say and getting some genuine, constructive feedback.

And to the guy who just posted a silly comment at my blog about me doing this to sell my book (it’s six years old, so I may be a little late using this channel to promote it) go read the column again. This is not an SEO is dead thread. It’s a “what are your thoughts about the future of search over the next five years” thread.

[Heini van Bergen]

Hi Mike,

First of all a great initiative to start this discussion. I think it's keeping us Search marketeers busy for quite some time know. I must say that I don't have the answer (yet;)). But I believe that it will become key for us to convince our clients on the importance of having a full exposure on the web.

A lot of (large) companies still believe they can push the market with huge budgets, without listening to their (potential) clients.

[SEO Montreal]

Can you define what you mean by Digital Asset Management? Seems like a term you're looking to coin for doing SEO on non-traditional pages like PDFs and video. Still seems like SEO to me.

As to the future of SEO, considering the increasing personalization, there's going to be more work and efforts by people to get around it. So in order to rank well in more countries, for instance, SEOs will try to get more links from .ccTLD sites and especially those featuring an address in the targeted country.

There are likely also going to be increasing specializations, as some people focus on ranking videos and others focus on ranking PDFs etc. More important will be specialization in certain communities; as social media takes a greater role in SEO, more SEOs will use SMM. Agencies will feature interdisciplinary teams composed of specialists in these various forms of content and community.

[MrMark]

I’m very new to the world of SEO so please excuse my ignorance.

Firstly liked the article as a newbie it's good to get a quick understanding of past, present and future. Cheers.

One thing I’m not clear on is you talk about visitors casting votes via bookmarking, tagging and rating. I’m taking this to mean a page that is bookmarked allot be listed higher (it’s got to be good if people are bookmarking it).

Tagging and rating I’m not so clear on. I understand it as people interacting with the page? If this is so does this include performing searches, posting reviews, posting on a forum etc.

Thanks,
Mark

[jimbeetle]

Wow, five years out, huh? At the moment I find that a bit of a daunting timeline to comprehend. In my ten plus years in the business most of the improvements to search have been incremental, evolutionary if you will, the continued application of well-known and well-understood theory tweaked for the web.

Over the past couple of years I believe we've seen more user data being incorporated and this will continue. There will be more social network analysis of the, well, social networks. I think G's Orkut launch was a step in this direction that somehow got sidetracked. I also think there is a lot of noise in social network's where just about anybody can be "friends" with anybody else. Mining GMail, Hotmal, Yahoo mail and the various IM services would appear to provide a more reliable signal to noise ratio. Any type of user data is up for grabs, from click-through data in the SERPs to click-path data in analytics packages. <added>And, of course, user interaction with advertising.</added>

But these are all still somewhat evolutionary advances. I find the five-year timeline a bit daunting because I think there might be something revolutionary on the horizon. What form it might take I don't know. An IR theory that comes out of left field? Tech advances (organic computing, fiber optic circuitry) that make it possible for the SEs to digest the vast amount of information they've accumulated?

Who knows? I certainly don't, though I won't be surprised if and when it comes. Just hope I can roll with the punches.

[carlwimm]

The title misses the point.

Search has already reached a limit, beyond which there are few improvements possible.

The current Results List is a documents list. If Google gives you some list in the form of A, B and then C ... and Yahoo gives it to you as B, C and then A ...

does it really matter?

Further, what else can you do with that, that is not already in some fashion being done? A, B' and then C ????


Further .. where is an area of any improvement possible? As long as the final output is a document results list, my opinion is that Search has topped out.

Of course, there will be some small refinements. After all, they are building better vacuum tubes today than they built in 1948 when the transistor was developed.

Where will real improvement come?

How about BEYOND search?

Right now, the fact that some SE gives you 400,000 documents in response to a complex query, suggests that Discovery and Exploration is the next big thing.

How about systems that do what humans are required to do, with the 400,000 docs.

Read them.
Decompose them.
Recompose the fragments.
Present a final list of possible answers that are proved to exist (not merely suggested in some 400,000 deep list).

Start spanning across documents, instead of getting stuck at documents.

In other words, reverse the current post search state of affairs.

To paraphrase, instead of 59 minutes of chaff and one minute of wheat ...

1 min of chaff and 59 minutes of wheat.

You want to know where the next big thing lies....?

You have my answer.

Carl Wimmer

[sitetruth]

In practice, using the "customer voice" to affect search results doesn't really work. It's too easily spammed. User ratings work only when the number of raters is large relative to the number of things being rated. Movies and TV shows, yes. Hotels, maybe. Plumbers, no way. "Joe's Plumbing" will be rated by Joe, Joe's employees, and Joe's relatives.

Incidentally, IBM has user feedback into search ratings patented.

We as SiteTruth take the position that information about the real-world business behind the web site should be cranked into search rankings. As a minimum, a business has to have a verifiable real-world existence in a physical location. This is a legal requirement in Europe and is required when selling into California if the site accepts credit cards. If the business can't be identified, and the site is selling or promoting something, it probably shouldn't even appear in search results.

That's just the first phase. Once the business has been identified, there's an established infrastructure of business legitimacy one can draw upon, from corporate registrations to criminal records to Dun and Bradstreet records. A good credit rating should help search engine positioning. If a company is having problems, that should reduce its search engine positioning.

We sometimes call this "bottom-feeder filtering". This is the clear next step in improving search.

[LinkMoses]

Great stuff Mike! I applaud your willingness to look five years out. Most so called experts are afraid to take the chance. I envision the engines offering results the searcher controls via toggles or sliders, after the initial search takes place. It's been tried poorly before, so it wont happen fast, so I'd see it as more of a suggested refinement option. That way the searcher typing in a phrase like "margarita recipe" can either wade through the organic or PPC results, or choose to refine results according to whether they want to see a how-to video, a book from Amazon, a local bar, a definition, historical facts, or whatever. The engines need to be willing to gently prod the searcher to do some very basic refinements, rather than throwing all of it at them with a few blended options that could very well be meaningless. I'd use an Up\Down slider like this

/\
o We Trust This Completely
| Company stuff
| This is popular
| Audio and Video
| News
| They Said It
| From the Library Only
| From The Dark Side of Town
| Run for Your Life
\/

See the problem? It could be fun, it would work, but most searchers will have to be forced to do it.

Sort of like hybrid cars :)

-Eric

[LinkMoses]

sitetruth - if a physical location was required then most of the now famous web businesses would not have happened. I started at a kitchen table at an apartment complex. So did Sullivan. So did Bezos. So did Jobs.

[AussieWebmaster]

Welcome LinkMoses... please part the seo seas so we can find the promised land... couldn't resist.

Good to have another 'kitchen table' entrepreneur aboard

[Mike Grehan]

So after travelling so much and the fact that the email alerts for this thread have not been coming through, I’ve missed a little of the conversation.

In my ClickZ column today, I’m picking the subject up again. Two areas that I really want to get some feedback on include ranking reports and the development of the search engine crawler.

For me, the idea of ten blue links following a query is probably going to become sub optimal if those links only ever lead to static web pages. There’s so much rich content on line to provide the end user with a much better search experience.

And if the crawler really is only good at getting the gist of a page by the text it contains, what about the non text documents such as audio/video. And what about data that can be fed or submitted such as local results which may not necessarily depend on pages being crawled.

The technology which search engines use to index billions of web pages is remarkable. And then trying to get the ten most relevant results at the top is even more incredible. But what happens if some of the more relevant results are not text pages?
I’d really be interested in hearing thoughts on new signals to search engines. Text and links are strong still, but there must be so many more signals which can be folded in.

[AdrianB]

Hi Mike

I just read back through your 'Future of SEO' column having felt guilty about not posting here previously and your "sweet whatsitsname..." reference in today's newsletter. I guess I had shied away from this because, in all honesty, I don't know where search will be in 5 years and, being in the Industry, could imagine getting slated for that admission.

In your ClickZ column, you state:

SEO will give way to a new form of digital asset management and optimization. This new SEO will place a much larger emphasis on optimizing a range of file types, from PDFs to images to audio/visual.

Apart from introducing an interesting new TLA, I don't really see what is so different. Sure, the search engines are indexing more data, in different formats, and this has to somehow get found so there is going to be a form of optimisation (search engine optimisation?).

Anyway, thanks for starting this and it'll be interesting to see how it develops.

Adrian

[Mike Grehan]

MrMark said: One thing I’m not clear on is you talk about visitors casting votes via bookmarking, tagging and rating. I’m taking this to mean a page that is bookmarked allot be listed higher (it’s got to be good if people are bookmarking it).


What I’m trying to figure is how really important end user data will be folded into the ranking algorithm along with the linkage data search engines already have. There are a lot of ways, including social bookmarking, tag clouds and rating, (say restaurants), for instance. All of which provides a great indicator to the quality and popularity of a search result, text through to audio/video.

AdrianB said: Apart from introducing an interesting new TLA, I don't really see what is so different. Sure, the search engines are indexing more data, in different formats, and this has to somehow get found so there is going to be a form of optimisation (search engine optimisation?).

Working with a company called Acromym Media TLAs are an absolute requirement :)

Actually I don’t think I could claim that as something I coined. Around about the same time I started thinking about this much richer experience for the end user I started talking about optimizing digital assets. Certainly Lee Odden has used the term and I’ve seen it a few other places.

For me the term search engine optimization has always been a misnomer. I don’t know anyone who has optimized a search engine (other than the search engines). Just lie I don’t know anyone who has marketed a search engine (other than the search engines).

The reference to digital assets is closely aligned to Google’s universal search which has really made search very interesting. But side-by-side is the social media “thing.” I say thing, because we use the term in a singular manner. But I see a three tier connection: Social networks (such as FaceBook, MySpace etc.) at the core. Social media and bookmarking sites such as del.icio.us and Digg . And then social search itself which will be a by product of the first two. People being able to tap into a trusted network looking for information. And this human element then folded back into the algorithm somehow.

Link Moses and revered search marketer extraordinaire said: The engines need to be willing to gently prod the searcher to do some very basic refinements, rather than throwing all of it at them with a few blended options that could very well be meaningless. I'd use an Up\Down slider like this

I have a feeling that, looking at the way universal search is incorporating those verticals into the SERPs, certainly with a little more “personalization” involved Google and other search engines will be able to determine whether you prefer news results, or video or blog posts etc. I entirely agree with you that end user preferences will start to play a greater role in the way that results are presented.

[AussieWebmaster]

Got to love TLAs... well some of us do.

[khawe]

When looking at the future of search you need to look at how SE are treating video. Both Yahoo and MSFT factor in view #'s and content rating into their rankings for videos - along with KW / content relevance. Granted it is not comparing apples to apples but still bring up many interest SEO implications.

[beu]

Signals from end users who previously couldn't vote for content via links from Web pages are now able to vote for content with their clicks, bookmarks, tags, and ratings.
Thanks to Google's Googlebot for RSS feeds, Feedfetcher, "Voting" is possible not only via click, bookmarks, tags and ratings but then again by subscribing to feeds from bookmark tag and rating sites.


SEO will give way to a new form of digital asset management and optimization. This new SEO will place a much larger emphasis on optimizing a range of file types, from PDFs to images to audio/visual.
This was stressed during last weeks factory tour. In 5 years everyone here will know what "EXIF" means. GEO EXIF in particular, has a number of implications for Sketchup and as a result Sketchups integration with Google Maps, Earth and Android.


Personalization and digital asset optimization will end 1999-style ranking reports, as search engine results will be based on blended results from end-user specifics, such as geographic location, time of day, previous searching history, and peer group preference.
At Google I/O last week Marissa Mayer pointed out that perhaps the best signal is a users previous query.


Great article by the way Mike! I think "page load speed" will be the most critical issue for SEM, SEO and PPC during the next 5 years. Fast loading pages are critical for mobile not to mention, slow pages waste search engine resources. To support this idea, I would point out Larry Page's obsession with efficiency, Google's focus on reducing power usage (IE DCs using hydro, solar and wind power alternatives), landing page load time being a QS factor and the fact that Google picked off Steve Souders from Yahoo and seems to be paying him at least in some ways to write version 2 of his book.

Oh well, my two cents! Great thread guys... One thing is for certain, it will be interesting to look back at this thread in 5 years just to see how wrong we are!!!!

:)

[AdrianB]

From a user experience, I would imagine that 'search applications' will become more prominent. There are already tools I use today like Search Automator Pro to help me wade through the shear volume of different media types.

Further, applications like Copernic Agent will become more refined and will allow better management of search profiles and user preferences, e.g. only return pages without this particular type of contextual advertising included ;-)

A 'search aggregator' appeals to me personally as I like to think I'm in control of my searches!

Adrian

[adman1212]

(Adrian B - Welcome, I'm Adrian C! ;)

I figured the Paranoia over Analytics data showed that people do (or did) expect this to be used to some degree. There's plenty there for Google to pick through and do something with - bounce rates for example, are a pretty clear indication of a sites worth for a given query.

I think it somewhat depends on the search engines success at providing quality and democracy - if I all I ever find are pages from big companies or a limited number of sites then I'd probably start looking elsewhere (as I do for some searches). Maybe at vertical search sites, social or just known favoured sites - afterall Google is useless for some searches.

It's all time saving that underlies this - I know if I want football fixtures I'm best off with the BBC, as Google can entirely fail for this type of query. I buy from Amazon as a default choice, not due to price or service - just for ease, because I don't have to enter all my details into yet another form.

I didn't have ADHD before the internet came about.. but perhaps we're all going that way if we spend too long online! I think SE's have been catering to the diminishing attention span for some time.

[Mike Grehan]

beu said: Great article by the way Mike!

Thanks, I have a feeling that this is a good discussion that people can dip in and out of as new thoughts arise. Some excellent comparisons of your own there, by the way.

This is something which really caught my attention:

AdrianB said: A 'search aggregator' appeals to me personally as I like to think I'm in control of my searches!

I took a look over at the Copernic site. I’ve known about this tool forever...And then forgot about it.

But I was stunned to see that it now has over 30 million subscribers. And absolutely, Search Automator Pro seems to be getting a lot more traction with serious searchers by the look of it.

So your post raises this question: What happens if, in the future, we all have a search aggregator to manage our search results by file types and geography and business and personal and...

There is no doubt in my mind that search is predominantly the way that people will find things online. And a useful tool/resource such as a search aggregator is bound to make life easier when it comes to finding your stuff.

But that would mean you wouldn’t need to go to the Google interface or Yahoo! or MSN, or anywhere else for that matter. Your own search bot would be programmed to go out and find results from preferred (or recommended sources) and you also get to choose whether you want white papers, movies, images, blog posts, news etc. etc.

So here’s the big question: Who’s adverts would you see in that mix?

Then, I took a look at Google’s personalized search offering and also Yahoo! The types of services/tools which are being developed may at some point in the future easily be rolled into a Copernic/Search Automator type of widget/gadget/tool. It’s already very close.

Does that mean that, in the future there could be a situation where your chosen search service pulls in results from all over the web, Google pulling from Yahoo!, MSN and those doing the same. So now, you have results from all of the major search engines, but you only see the ads from your chosen search service.

Can anyone else see where I’m going with this? Have thoughts to add on the idea of a desktop search widget that rolls the whole thing together in one set of results that you can edit, save, delete?

And then how do we optimize around that? Just me thinking out loud, as usual.

[AdrianB]

There is no doubt in my mind that search is predominantly the way that people will find things online. And a useful tool/resource such as a search aggregator is bound to make life easier when it comes to finding your stuff...

I suspect the same thing will happen with this that has gone before with social networks, video distribution sites, analytic platforms, payment processors etc., The companies with the deepest pockets will either buy out the innovators or develop a killer app for free.

The difficultly is resisting the temptation, as a user, to wean myself off of Google search results and give some of the others a fair crack. If 'future search' is owned by one company, then that will be a very bad thing indeed for both searchers (manipulated results) and advertisers (ever increasing CPCs).

[bragadocchio]

Some possibilities for search advances over the next five years:

1. Regrouping of universal results by concept instead of content type, for rapid disambiguation of concepts related to a query and better targeted search advertisements.

2. Creator labeled content using something like openID, with metadata citing information such as editor, site owner, content creator, approved syndication locations, etc.

3. Object level ranking for named entities, with more answers shown directly within search results, and more final destination links (like Google's site links) listed under a search result, including results from other sites.

4. Better use of facial recognition and image recognition techniques so that statistical image models can be used to index pictures that don't have accompanying meta data.

5. Better use of user information in search sessions culled from query logs and tracking data from browsing history to aid in query expansion and refinement suggestions for informational and transactional queries, and in determining best final destination pages for navigational searches.

6. Increased use of geolocation information with mobile phones to identify the locational sources of queries, and to automatically tag mobile phone camera images and identify landmarks and locations, provide private shared maps, supply human friendly driving and walking and transit directions with ads for businesses and services along the way.

7. Search kiosks in shopping and resort areas, tied to billboard advertising, displaying business sales and available inventories and coupons and menus and wait times at restaurants, enabling the taking of reservations for those restaurants, and more.

8. Present day and future looking decision-making search tools for business and government and policy makers, based upon information taken from sources like news articles/announcements/press releases, that can enable those decision makers to base their actions on a more informed understanding of the activities that are, and will be taking place around them.

9. A deeper integration of desktop and Web based search, providing automated query results related to the content being used or recently used or viewed by people working upon desktop documents, reading emails, updating calendars, etc.

10. Targeted online advertising based upon actions taken offline, such as credit card purchase, television viewing, attending events, and others.

I could go on, but that's probably enough for now.

[AdrianB]

Wow bragadocchio, that's a very thorough list. I'd hate to be the Sales Executive pitching this to an SEO prospect in 2013!

[bragadocchio]

Hi Adrian,

I don't know if it's a thorough listing. I think my list is ust a small sampling of some possibilities, without taking too great a step away from what may be possible right now.

A number of them could have some interesting impacts on how SEO and marketing on the web is done, and I would look forward to talking with people about them. :)

[Mike Grehan]

bragadocchio said: I don't know if it's a thorough listing. I think my list is ust a small sampling of some possibilities, without taking too great a step away from what may be possible right now.

Hey Bill... You took your time :)

Excellent contribution and some great insight. The regrouping of universal results by concept as opposed to content type is something I touched on (here or in my ClickZ column).

My guess is that, as personalized search begins to get more traction, search engines will be able to get a much better picture of the end user search behavior. And then they will be able to tailor results according to the preferred vertical/content type.

Did you have any thoughts on the search aggregator type tools I mentioned. That’s an interesting area to explore. Not just a tool for maintaining and managing all your search stuff, but also your bookmarking, tagging and rating etc.

Thanks for pitching in. Looking forward to more!

(Why don’t you give that Li Evans at the desk next to you a nudge ;)

[jimbeetle]

and you also get to choose whether you want white papers, movies, images, blog posts, news etc. etc.
This past Sunday evening I watched "When We Left the Earth: The NASA Missions" on the Discovery network. Being a child of the 60s I was fascinated by the concept of man traveling into space and watched every launch and recovery (Yes, if something was happening in space, little Jimmy wasn't going to be in school that day.).

I'm still fascinated. On Monday I decided to do a little brushing up on the concepts tested during the three programs that led up to the moon landing. So, simple searches for [project mercury], [project gemini ] and [project apollo].

I was very, very disappointed. With all of the material generated about these three programs over the past 50 years -- much of it media rich -- I was mostly presented a list of links as results.

Google, for all its vaunted universal search, let's get everything in there that the user might be interested in approach, was the worst offender. In only one case (project gemini) did it even include an image onebox. It did include the related searches blurb at the bottom of the results, but by that time I was out of there. Yahoo and Ask did a bit better with related searches. Still no video results for either; Ask at least did have on-page image results. MSN Live? Not even in the game.

What do I want in search results? Well, I guess I want everything. I want all media formats returned. I want them easy to review. I want to select them, sort them. I want to save the sorts as an outline for a presentation or a paper (or maybe just to again relive the excitement of the 60s ten years from now).

One of the things that got me into this Internet stuff was Apple's introduction of HyperCard (http://en.wikipedia.org/wiki/HyperCard) back in the 80s. It was the first hyperlinking app available to the general public. So, when I saw the first surface computing vids a couple of years ago to me it was the perfect melding: search results returned in discrete stacks, maybe by format, maybe by concept. Clickable, sortable, savable -- usable.

That's what I want. I think. Until something better comes along.

[AdrianB]

What do I want in search results? Well, I guess I want everything. I want all media formats returned. I want them easy to review. I want to select them, sort them. I want to save the sorts as an outline for a presentation or a paper (or maybe just to again relive the excitement of the 60s ten years from now).

Then I think this where the search aggregator applications come in. As mentioned in my earlier post, Search Automator Pro gives me access to all of the major search engines and sort the results quickly in a way that suits me. It certainly isn't a perfect solution but definitely makes my search experience a lot easier.

[bragadocchio]

Hi Mike,

Thanks.

The major search engines have an incredible amount of data about user behavior. The bottleneck may not be so much in collecting that information as it may be in figuring out the best way to make use of it.

Did you have any thoughts on the search aggregator type tools I mentioned. That’s an interesting area to explore. Not just a tool for maintaining and managing all your search stuff, but also your bookmarking, tagging and rating etc.

I've been watching Google slowly release patent filings on something that might combine desktop/intranet/internet searches with alerts/annotations/tagging as well as software download management and many other tools and features only now hinted at in the titles to unpublished patent applications from them. One of the patent documents published at WIPO listed 51 related patent applications in this collection.

You're seeing some of the potential outlines to such an approach in the personalized services from Google, and from Yahoo.

Keep in mind also, that when someone searches at Yahoo, and they have a Google toolbar installed, that Google can see the results of their Yahoo searches, and the selections those searchers make. I believe that Yahoo explicitly mentioned in one of their patent filings that they might do that with their toolbar, looking at search results from other search engines and user behavior associated with those searches.

How do you optimize around that? There's the rub. :)

Liana's desktop next to mine is actually a virtual one. She's probably closer to you than to me. :)

[Webmaster T]

Microsoft Launches Beta Release of SearchTogether Plugin (http://blog.searchenginewatch.com/blog/080611-090602)Seems interesting and on topic. The social aspects of this are more interesting and usable then the academic examples provided in the release.

Ubiquitous/WiMax connectivity will be responsible for many changes to search going forward. Mobile will not be just phones but computing devices that have phone capabilities. This is a much different environment that search will approach in a different manner because the user expectations and reasons for searching are easier to determin.

Research by the US Gov't resulted in the ability to index voice in video. IMO, when that technology is improved (my discussion with them indicated that the voice recognition part of the technology had a long way to go) we'll still have 10 links in the results but many will point at video. IMO, video is coming of age with the increased penetration and affordability of broadband and the high regard videos have with many users.

The capability to index PDF has been around for a long time, but, you seldom see them in the results. Whether that is a result of the amount of content in that format, the degree to which it is SEOed, or a decision by the SE's to dampen the visibility has not to my knowledge been researched. Personally, I prefer HTML to PDF but... that's just me... or is it?

Personalized search is gaining traction and the social network sites do provide some indications of user preferences. Universal search is ever expanding as I see more and more Brand one boxes for sites that didn't have them 6 months ago. Going forward I see the SE's wanting these abilities to improve and drive more of the results.

[AussieWebmaster]

Mike's new article today (http://www.clickz.com/showPage.html?page=3630195) explores a new angle - Google who many now sees as the internet - or at least the way to get access to all of the internet/web - cannot meet the ever increasing additions to the total....

the crawler as it works today definitely has reached its limits.... as Mike told me last week over dinner - nearly 3 times as much new crawlable content is added to the web every day than is crawled. This growing gulf will be the biggest hurdle the web faces over the next couple of years...

What the new options are I think will be the topic of some future articles by Mr Grehan.

Your comment about the seeming non-existence of companies or people who do not have a web site or web presence is another area that will up for major discussion in the near future and beyond...

[Misscj]

Very interesting responses all round.

I work in A.I research, namely in natural language generation and natural language understanding, after a long stint in IR. We've been looking at machines interacting with data, "understanding" it (it has to be in inverted commas until someone defines what understanding is)and doing something with it. Some people, as well as myself are looking at question-answering systems, which would follow back to those days where you tried out those betas that kinda disapeared and not many people carried on using. Machines don't understand very much yet, and NLG is quite basic still, despite great efforts. It is however advancing. The pattern recognition and keyword retrieval based methods work to an extent but aren't scalable. This in my opinion goes for search enginges, and the work in digital libraries too. We're working on A.I techniques. So is everyone else :)

I guess what I'm saying is that I see machines playing a much greater part in our experience of the internet. Machines creating documents, creating summaries of groups of documents for you, can we we do away with ranking altogether please? Can we have data with answers that we're looking for rather a list of documents, it's hit and miss whether it's the right one and, I don't know about you but I have to go through a few until I have gathered all of the information I need. Can we bypass this? In time, yes I think so.

Accessing documents for research and information access without visiting a site, just through explaining what you want, in your own words, and getting a thorough response back, allowing for further refinemend through your own language. Sending out a machine to buy stuff for you? Maybe.

As we now shop from home, could we shop without window shopping for answers on the net? Maybe, maybe...maybe not soon, but maybe.

Machine translation and speech are big projects right now. Both need A.I techniques too, and when it comes down to it, all of the techniques in NLG, NLU, NLP, IR come together.

Another issue with developments here is gathering user data to allow for testing and learning. Privacy is an issue. I need to gather online conversations, and it is hard work getting people to let me access those.

btw, if any of you would like to donate your chats to me...don't hesitate to contact me :)

[Webmaster T]

Mike, great article and astute observations on crawlers ability and scope. Do you believe the dropping of supplemental results in the current Google SERPs are a step toward decreased crawling of that index? Do you believe the recent Google announcement of executing forms to uncover content was to find previously hidden content or just an excuse to look up a sites skirt?

My thinking has been that the supplemental index is being dropped to free up crawling resources for the new content they'll find behind the forms. I also believe that all Google crawling, including the media crawler, are contributing to the organic index.

IMO, if you don't refresh that supplemental index very often you have freed up a lot of crawling resources. I know I see large decreases in what's being crawled on large dynamic sites.

[Mike Grehan]

Misscj said: I guess what I'm saying is that I see machines playing a much greater part in our experience of the internet. Machines creating documents, creating summaries of groups of documents for you, can we we do away with ranking altogether please? Can we have data with answers that we're looking for rather a list of documents, it's hit and miss whether it's the right one and, I don't know about you but I have to go through a few until I have gathered all of the information I need. Can we bypass this? In time, yes I think so.

Misscj - Nice to have someone else new in the thread.

I must have written about this so many times. You know, when you had a dial-up connection and limited bandwidth, ten blue links (which as you say, some of which may be relevant and some won't) was a decent enough experience. But in this day and age it's just a suboptimal return.

For those who were present at Andrew Tomkins (VP, Search Research, Yahoo!) keynote at SES, New York, you'll have heard him voicing similar concerns. At one point, he stated something along the lines of how "scarily similar" the SERPs looked - ten blue links - ten years ago.

And apart from anything else, here we are still talking about the only way a document can rank is if people have links to point to it. No wonder spammers don't want the ten blue links to go away. They'll be out of business.

Hey search engines – did you hear that?

Anyway, the crawler thing... The bare basics, that I wrote about in my recent column are based around this. If (and these figures are out of the blue no research, purely as an example) a search engine such as Google downloads 30 million new pages a day and content creators (and end user generated content) hits 90 million pages a day, the math is not too hard to figure.

Let's take this new search medium and measure it against an established medium such as TV. A catastrophe happens somewhere. User generated content appears on news channels world-wide within hours.

In search? A day? Days? Weeks? It all depends on where it appears on the web first.

Maybe tag-clouds and bookmarking and rating are also slow to the mark (although I do believe they carry a little weight for certain types of query).

So, anyone else have thoughts about the crawler and the tightening limitations on what it can bring to the surface in the SERPs?

Webmaster T said: My thinking has been that the supplemental index is being dropped to free up crawling resources for the new content they'll find behind the forms. I also believe that all Google crawling, including the media crawler, are contributing to the organic index.

Personally, I think the supplemental index hasn't changed. It's just not being flagged anymore. And as we know, Google is under represented (the crawling thing) for certain queries, so the supplemental creates a neat little back fill. Rubbish, but a back fill.

[AdrianB]

I mentioned in a previous post about search aggregators but a colleague has just flagged up rollyo.com to me. Apart from being deeply embarrassed about something else that seems to have passed me by, could this fall into the 'search in the next 5 years' category?

My first impression of this was 'wow' followed by a 'oh' as is 'oh, what will the major search engines do when their ad revenue goes down because the searchers are pulling in organic results without 'their' paid ads?'.

I'd be interested to hear of other's experiences of Rollyo and opinions for where it fits in the future of personal search.

[Mike Grehan]

AdrianB said: My first impression of this was 'wow' followed by a 'oh' as is 'oh, what will the major search engines do when their ad revenue goes down because the searchers are pulling in organic results without 'their' paid ads?'.


Adrian, it's a very interesting observation. Rollyo had so much exposure within the industry and technology press as a whole – but only a little dent into the consumer/end user side.

I think, certainly in the future, the Google standard of your 2.8 word query returned 14,200,000 results… Of which you'll only look at the top three, of course...

Being swamped in a deluge of results, which, generally speaking, after the first couple of pages the relevancy drops like a stone, doesn't have a novelty value anymore. People actually expect a much richer experience on the web.

Part of the problem is that, in our industry we look at the search engine as a black box - we have no real idea exactly what's in it. And yet, it's the same the other way around. The search engine looks at you the end user, and you're the black box. The more you let Google (or any search engine) know about you, the more granular and refined the results can be.

Given that you could take something like Rollyo and up its performance to more of a learning machine, the more personalized the results could be. So, all search engines want you to get involved in personalizing your results. Log in, tell us about yourself, let us track your search history, that sort of thing.

The big question though is, who's ad results would you see in a search aggregator?

Well, I certainly see traces of Google's personalized search moving slowly towards a search aggregator type of service. And if they did, you'd be able to get the best of Google's results and Yahoo and MSN and any verticals and news and blog sites etc. etc. all in one place in front of you. With AdWords next to them, of course.

So the next question is... Who is going to develop the search aggregator version of Tivo? And that way I'll just skip all of the ads :)

In my ClickZ column (http://www.clickz.com/showPage.html?page=3630337) today, I'm talking about the voice of the end user and how much of a say they should have in ranking results. I'd be very keen to hear anyone else's thoughts on the subject.

[pedro5000]

Hi Mike,

I've always enjoyed your thought provoking columns.

Your comment at the end of the last one...
And then it occurred to me that there is, in fact, a huge, global search engine with results powered by the end users.

It's called AdWords!

...got me thinking. One thing that may happen in the "future of search" might be that Google and the rest may be forced by legislation to split the paid and organic parts of their businesses.

By examining the data available to it in Adwords, Google can see which retailer sells a lot of a particular product. I can't think of a better "explicit signal" of a sites relevance than the fact that it sells more of a particular product than anyone else.

So, Google should use that data to bump that site to the top of the organic search results for a term related to that product shouldn't they? But, of course that would conflict with their paid search business...

Whilst we all trust Google not to be evil. Is it really fair to let them regulate themselves?

[Webmaster T]

Name a single regulated industry that wasn't ruined or become more costly by said regulation and you have a point... otherwise, better the "potential evil" we know then the evil regulation causes for everything it touches! Gov't and regulators F everything up! Google, IMO, don't mess with that they know what drives revenue and searches. You need look no further than Yahoo! for that lesson!

[NewKidOnTheBlock]

I don't have enough time to follow the whole discussion, but hope I can still chip in as I'd like to comment something on one of your first posts Mike:

I'm not sure if allowing more people voting (by allowing people w/o websites to vote) would mean the results would be better for a certain reason:

I've often thought of search and links and voting as an analogy to political voting. I'm 25 and didn't vote the last two times. The reason is, I really don't know a lot about politics, actually I know almost nothing about politics.

Democracy sounds great, but in reality how many people who vote really know what's going on? I bet a really tiny percentage of them do. I see this in all areas of life. People know just a tad bit about a certain topic, but feel like they should have a strong opinion on everything they know a little bit about.

But often if you actually know the topic you realize how bad and flawed their decision-making (which is based on superficial knowledge) is. Example: Listen to a few people who've heard a bit about SEO without really learning about the topic. They'll say stuff such as "ah the search engines..just stuff your meta tags with keywords and you'll be doing fine ;)" or "SEOs should all be locked up. Only because of those SEOs search engines have to spend millions of dollars in order to keep on finding new ways to rank webpages". I have seriously heard those two (and A LOT of other such reasoning coming from people who have just a little knowledge on a certain subject, but feel like they have the right to have a strong opinion on it).

I think in English they say "A little knowledge is a dangerous thing" and in politics the majority of people in a democracy make their decisions based on "a little knowledge", imho.

To be quite honest, I think I (and a ton of people I know - maybe that has to do with my age, but I think many more mature people would have to be included) should NOT be allowed to vote, because my knowledge of politics is close to zero (what's even more absurd is that most people would tell me it's wrong that I haven't gone voting - despite the fact that I dont know crap about politics...). To drive a car you need a driver's license, but to vote a president (who can make decisions for the whole country and start wars, etc.) you just need to be 18 (or 16) hehe ;).

Sorry for brabbling, but I think a lot of times having more people 'vote' to make a decision is worse than having a select few people vote. In politics I'd like the idea, that everybody who wants to vote should pass a test on 'politics' before they're allowed to do so (but the problem with this would obviously be that this would give an edge to extreme parties (including strong right/racist parties)).

I've thought about this quite a bit and to be honest, if I had to make a decision and a handful of people who are very educated on a certain topic and who I consider to be brillant would give me one piece of advice, and another 100,000 people who are not very educated on the topic and are less intelligent than those handful of people.....I would go with the advice of the handful of brillant, educated people (almost) everytime. Extreme example: Ask 100,000 people from the general population what to do to get good rankings in the search engines and ask 5 people who are doing SEO for a living, who's opinion will you go with? easy decision.

Sorry for brabbling........but because of this (I've thought about decision making and voting, etc. a ton) I think that allowing everyone to vote might be worse than only allowing people who have taken the time to create a website on their topic to vote, because they will usually make the better decision.

I think there's a place for usage data (if u can find a way to leverage that to create better rankings as a search engine), and maybe there's a good use for social tagging, too. But I think allowing more people to vote might not necessarily increase the quality of rankings - if those people know less on average.

Maybe search engines should allow people to vote by tagging if they have a rich internet history in their field (which would show that they are probably educated about the topic) - power accounts or something? The higher the reputation in a certain field the more their 'vote' should count.

P.S.: I'm just not voting at the moment, I'll probably go voting in the future when I learn more about politics.