Search Engine Watch
SEO News

Go Back   Search Engine Watch Forums > Search Engine Marketing Strategies > Search Engine Optimization > Dynamic Website and Technical Issues
FAQ Members List Calendar Forum Search Today's Posts Mark Forums Read

Reply
 
Thread Tools
Old 06-02-2004   #1
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
Welcome to the Dynamic Web Site Issues forum

I am looking very much forward to see what this dynamic forum will bring. As anybody that knows me know, I am a dynamic-geek but there is always room to learn and I hope to do so from all the great people that will soon populate this forum.

I will personally do my best to share usefull information about this important subject. My personal experience, working with search engine marketing and dynamic websites is that it's not really a problem but rather an option. It's an option to do better than any static site will ever do.

Lets get some real postings roling
Mikkel deMib Svendsen is offline   Reply With Quote
Old 06-02-2004   #2
rustybrick
 
rustybrick's Avatar
 
Join Date: Jun 2004
Location: New York, USA
Posts: 2,810
rustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud of
I am sure the spiders will get better at indexing dynamic content, so I do anticipate this forum to get a lot of use. I've heard you speak at SES and I look forward to your posts in this forum.

Welcome!
rustybrick is offline   Reply With Quote
Old 06-02-2004   #3
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
> I am sure the spiders will get better at indexing dynamic content

At SES in Toronto Chris Sherman talked about a conversation he had with some of the leading engineers at Google, working with the issues on crawling dynamic content. Google said, according to Chris, that for them to develop a solution that would uncover the entire "hidden web" it would (conservatively) take 50 years! I bet they are right.

So, for the next 50 years or so we sure need this forum
Mikkel deMib Svendsen is offline   Reply With Quote
Old 06-02-2004   #4
rustybrick
 
rustybrick's Avatar
 
Join Date: Jun 2004
Location: New York, USA
Posts: 2,810
rustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud of
Toronto was a long time ago.

Uncovering all of the 'hidden web', in my opinion, will never be done.

I agree.
rustybrick is offline   Reply With Quote
Old 06-02-2004   #5
AdamH
Member
 
Join Date: Jun 2004
Posts: 5
AdamH is on a distinguished road
Do most people use mod_rewrites to create hard links and get dynamic content indexed, or other tactics?

I currently have a client whose site is in a proprietary content management system and am working to get this site more search engine friendly.

Any advice from those with prior experience would be greatly appreciated.

[Removed signature per guidelines]

Last edited by David Wallace : 06-09-2004 at 04:06 PM.
AdamH is offline   Reply With Quote
Old 06-03-2004   #6
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
The reason I put The Hidden Web in "" is that I do not think you can ever define how many pages it represent. How many pages do Google have? One for every search made, right. There is an endless number of pages. And the same goes for most other query based dynamic ("hidden") web sites and content.

So, if the hidden web is infinite how can you ever index it all?
Mikkel deMib Svendsen is offline   Reply With Quote
Old 06-03-2004   #7
rustybrick
 
rustybrick's Avatar
 
Join Date: Jun 2004
Location: New York, USA
Posts: 2,810
rustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud ofrustybrick has much to be proud of
Mikkel, lol, very true.

AdamH, I think most people use mod_rewrite if they are running their dynamic site on a Unix box. I use mod-rewrite all the time, not just for SEO reasons.
rustybrick is offline   Reply With Quote
Old 06-03-2004   #8
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
Yes, mod_rewrite is only for Unix but with ISAPI filters you can do the same for NT - and, as you say, there are many interesting things you can use it for beside just SEO
Mikkel deMib Svendsen is offline   Reply With Quote
Old 06-03-2004   #9
Ron Carnell
Member
 
Join Date: Jun 2004
Location: Michigan
Posts: 12
Ron Carnell is on a distinguished road
Quote:
Do most people use mod_rewrite to create hard links and get dynamic content indexed, or other tactics?
I'm going to go against the grain a bit here, Adam, because IMO mod_rewrite should be the last tool a developer reaches for, not the first.

If your CMS isn't SE-friendly, that's a problem with the CMS, not with the web server, and the best solution will be implemented at the correct level. Realistically, I know it's not always possible to solve the problem at the source, but I think that should always be the goal. At the end of the day, mod_rewrite is typically a Band-Aid used to cover up sloppy programming. Heal the wound, and the Band-Aid can be removed.
Ron Carnell is offline   Reply With Quote
Old 06-03-2004   #10
mcanerin
 
mcanerin's Avatar
 
Join Date: Jun 2004
Location: Calgary, Alberta, Canada
Posts: 1,564
mcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond reputemcanerin has a reputation beyond repute
I agree with Ron entirely.

I also think the general direction of SEO in general will change in the future due to this happening. CMS systems will be coming "out of the box" SE friendly, HTML editors with SEO (and accessability) tools built in and easier to use, web design courses including an SEO/SEM component and so forth.

I suspect that a lot of the simple (and not so simple, in the case of getting some CMS systems to an SE friendly stage) "grunt work" will stop being in the SEO realm and SEO's will begin to be able to focus on other issues that can't be handled as easily as upgrading to properly designed software or launching a wizard that checks for Alt, Meta and Title tags and other simple issues.

Ian
__________________
International SEO
mcanerin is offline   Reply With Quote
Old 06-07-2004   #11
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
"Theoretical" speaking I agree with you, Ron, but in my personal experience it is just very often easier in the long run to have the URL-rewrite middle layer, so engineers (as good or bad as they come) can have it their way and marketers and SEOs can have it their way. It sort of bridge two groups of people in large corporations that do not always communicate very well

On top of that comes all the other cool stuff you can do once you start utilizing mod_rewrite ...
Mikkel deMib Svendsen is offline   Reply With Quote
Old 06-07-2004   #12
Ron Carnell
Member
 
Join Date: Jun 2004
Location: Michigan
Posts: 12
Ron Carnell is on a distinguished road
Hey, Mikkel! What's it been, five or six years since we had not dissimilar discussions at SEF? Except, I think that was before there was an ISAPI solution and all the threads revolved around 401 traps. Glad to see you've kept busy.

You're absolutely right that it's very often easier, but I'm not so sure that adding unnecessary layers is necessarily better. Those engineers (as good or bad as they come) aren't stupid people, and I think education can sometimes be a better, more long-term solution. If an old sea dog like me can learn ...

While mod_rewrite falls in the very cool category, nothing comes without cost and maintaining multiple API's in anything except the most static environment can be a high price to pay, especially when you have to pay and pay and continue paying. Not to mention the server overhead when the monthly uniques start to push into the high seven figures.

There are times when the underlying software simply can't be changed, for various reasons, and mod_rewrite is the only answer available. Other than that, I think it's just a short-term solution. My rule of thumb is, if you expect to live with the application for more than six months, pay the piper and fix it right. A year down the road, you'll be glad you did.
Ron Carnell is offline   Reply With Quote
Old 06-07-2004   #13
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
It's good to see different point of views on the issue. There are many valid arguments on both sides (though, I am not even sure we are on different sides)
Mikkel deMib Svendsen is offline   Reply With Quote
Old 06-08-2004   #14
rcjordan
There are a lot of truths out there. Just choose one that suits you. -Wes Allison
 
Join Date: Jun 2004
Posts: 279
rcjordan is a name known to allrcjordan is a name known to allrcjordan is a name known to allrcjordan is a name known to allrcjordan is a name known to allrcjordan is a name known to all
>Not to mention the server overhead

Yeah, there's the sticking point for me. Even if I'm not hitting high-7 uniques, I tend to have a lot of other draws on the server and I've found out the hard way that sites crack under load on their best $$-performance days.
rcjordan is offline   Reply With Quote
Old 06-08-2004   #15
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
I have used URL'rewrite on even sites with heavy traffic without ever running into server problems because of that. In my experience, and from talking to other engineers I know use rewrite on NT, .NET and Apache most say the same: Rewriting use very little recourses. Off course, if you are running on the limit it may be the determining factor but then you might benifit from a bit more headroom anyway

Every process you run on a webserver "cost" but if the benifits are good enough it may well be woth it. I do not always find it the most important goal to save a few percentages of CPU if that power is used for something useful.

If I really want to save resources I'd stop server monitor logging that can take up as much as 25% of actual resources (on NT) and move data collection to network level (packet sniffing) or client side tracking - but thats another discussion
Mikkel deMib Svendsen is offline   Reply With Quote
Old 06-08-2004   #16
JBL
Member
 
Join Date: Jun 2004
Posts: 9
JBL is on a distinguished road
Yahoo and Dynamic Sites

Just under a year ago, I used Spider Safe URL to re-write my dynamic urls. Old urls looked like www.my-web-site.com/?city=10. Now they look like www.my-web-site.com/default.asp/city/10. Google has it figured out and ranks the new urls properly. But, Yahoo has both sets of urls in their index and seems to be penalizing me for duplicate content. At least, that's what Sit Match tells me.

Site Match seems to think I should go back to the old urls. But, I'm not willing to risk the good positions I have at Google. Will Yahoo ever figure out that this is not really duplicate content, but really just old urls that shouldn't be in the index at all?

Thanks for any advice or comments.
JBL is offline   Reply With Quote
Old 06-08-2004   #17
Dodger
Honorary Member
 
Dodger's Avatar
 
Join Date: Jun 2004
Location: Central US
Posts: 349
Dodger has a spectacular aura aboutDodger has a spectacular aura aboutDodger has a spectacular aura about
Yahoo also does not appear to be indexing pages past the first query in the URL either. Anything after that first ampersand is not being picked up. You can do a site: query on Yahoo for any of your popular forums and see that compound queries in the Url are not there.

Another thing they are not picking up is Blogger pages. But that is for another subject altogether. ;0)


As for mod_rewrite ... yep that is sloppy, I agree. But someone mentioned it was for Unix only. My impression it was for Apache Server, which runs on Unix, Linux, and Windows boxes.
__________________
I am Ronnie
Dodger is offline   Reply With Quote
Old 06-08-2004   #18
seomike
Md_Rewrite Guru
 
Join Date: Jun 2004
Location: Dallas, Texas but forever a Floridian!
Posts: 627
seomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to beholdseomike is a splendid one to behold
There are many mod rewrite programs available for IIS. I think there is a thread in here that has a pretty good list.

Last edited by seomike : 06-08-2004 at 10:16 PM.
seomike is offline   Reply With Quote
Old 06-09-2004   #19
Ron Carnell
Member
 
Join Date: Jun 2004
Location: Michigan
Posts: 12
Ron Carnell is on a distinguished road
Quote:
Off course, if you are running on the limit it may be the determining factor but then you might benifit from a bit more headroom anyway
Server overhead covers a lot of ground and, unfortunately, isn't always a black and white issue. For example, one of my sites is a poetry site, which tends to be very seasonal. During most of the year, our uniques hang around 2M a month, while the summer months will see a dip of almost fifty percent and major holidays can see a short-term rise of several hundred percent. On February 13 and 14, it's not unusual to see over 25K uniques an hour for several hours at a time. Ideally, the server would have sufficient headroom to handle that kind of load without a hiccup, but realistically, it's hard to justify paying for that much server 365 days a year. There's not quite enough money in poetry, I'm afraid, to warrant unnecessary expenses.

My situation may be extreme, but seasonal fluctuations are certainly not unusual. I've seen too many developers implement mod_rewrite solutions that "work" just fine during average loads, but result in unacceptably sluggish response times during peak loads. This can be especially dangerous on large sites if a high-traffic user period coincides with a spider crawl, since the whole goal of mod_rewrite is to fool the spiders into thinking they are getting static pages (which WILL be requested faster than dynamic ones).

I still think mod_rewrite is a great Swiss-army knife every developer should have in their took kit. But it's a knife with very sharp edges and should be handled accordingly.
Ron Carnell is offline   Reply With Quote
Old 06-09-2004   #20
Mikkel deMib Svendsen
 
Mikkel deMib Svendsen's Avatar
 
Join Date: Jun 2004
Location: Copenhagen, Denmark
Posts: 1,576
Mikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud ofMikkel deMib Svendsen has much to be proud of
You got a very valid point, Ron. You case is probably extreme, as you say, but I guess there are many sites with similar problems (alltough maybe not as dramatic).

Do you have any estimate on how much CPU (%) the overloaded servers used on rewrite? As far as I have been able to find out it's usually less than 2% and it that case you should be able to squeze it in on most systems.

But, off course only if the code is decent and as effecient as possible
Mikkel deMib Svendsen is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off