Building Large Scale e-Commerce Systems?

Fross asks: "I am Technical Architect at a reasonably large e-commerce firm, which naturally has a very large and active Website, with tens of thousands of registered users, and millions of page hits a month. The time has come to reimplement the old architecture from the ground up, and I'm looking for other (preferably Open Source, or Open Source-friendly) solutions that will give a faster, more scalable, more reliable, easily maintainable solution than the current implementation (which is running under ASP, on IIS of course). I have a fair amount of experience with Perl, some with PHP, but have never used them for something of this size. Does anyone have experience of deploying something this large with these technologies, and can give some valued information on how these solutions cope?"

Check out WebObjects

[Chris Hanson]

I strongly suggest checking out WebObjcts (at www.webobjects.com [webobjects.com]). It's what Dell used to create their online store before Microsoft developed a replacement in ASP as a showcase for their mediocre technology. There are thousands of other companies large and small using WebObjects to build web applications for both internal and external use.

The big advantage of WebObjects over systems like the ArsDigita Community System, PHP, ASP, and JSP is that it provides nice layers of abstraction. You never have to write SQL (if you don't want to) to get database-backed persistence for your business objects. Your application logic is separate from your business logic which is separate from your presentation logic. It also has nice graphical tools for doing entity-relationship modeling and laying out your dynamic pages and so on.

There are also some great third-party add-ons available for doing things like electronic comerce, wireless access (e.g. WAP), PDF report generation, etc. There is even a large, dynamic developer community.

Of course, WebObjects is a commercial solution, not an Open Source one, but it's a good solution and it's relatively inexpensive considering its capabilities.

Email me if you'd like more information; my company does web application development with WebObjects.

--
Christopher M. Hanson
President

A link for you to check out.

[jonfromspace]

I am not sure that this will suit your purpose, but check it out anyway.

OpenSales [opensales.org]

Hope this helps

Building Large Scale e-Commerce Systems

[mami]

I recommend Akopia Interchange, a merger between the development team of Tallyman (formerly Shopsite) and Minivend.

Completely based on Open Source software, extremely flexible and scalable.

Visit www.akopia.com and www.minivend.com and contact the authors of the software for more information.

why not

[AbbyNormal]

out source it? If it is beyond your grasp and you need help you could always get consultants to help you out. It works for Dilbert.

Try J2EE on IBM's WebSphere with Oracle or DB2

[Anonymous Coward]

Although my employee contract explicitly forbids me from mentioning any details about our arquitecture, I can give you a few pointers if what you're after is high-transaction, low-latency, 24x7, and high-availability. As surprising as it may sound, the most flexible solution we've found is Java-related. We had 4 teams each with 5 people trying out 2 different approaches per team (total of 10 different options). The main technologies (in no particular grouping or order) we looked at were Apache/PHP, Microsoft's ASP, Alaire's ColdFussion, Oracle, IBM, and Microsoft databases, iPlanet and WebSphere's J2EE (Java 2 Enterprise Edition) servers, XML, Linux, Solaris, Windows 2000. The approach I'd recommend is a cluster of one of these: Linux, Solaris, IBM Mainframes (you'd be surprise at how their latest generations handle web traffic and transactions), both for the application servers and the database servers. The beauty of J2EE and one of the strongest reasons to select it was that we could trully mix and match different OSs, databases, hardware, etc with virtually no problems, and our internal benchmarks show that JSP (Java Server Pages) are superior to Microsoft's ASP or Alaire's ColdFussion. Also, under heavy load for high-transaction computations we found out (and this we double-checked for it was a complete surprise) that Java code performed just about the same as C++ compiled code. We assume this has to do with the the latest HotSpot dynamic compilation technology from SUN. If you can afford it, I'd say put 2-4 people testing this stuff out. If you're organization is sufficiently large you can get full versions from all of these products for free for a decent trial period. Above anything, I recommend you keep and open mind and be objective by doing your own tests. This simple vision is what's causing us to transition to a Java-based arquitecture. It's light years ahead of when we first tried it a couple of years ago (when we dismissed it for mission-critical operations), and it will certainly shield you from the constant stream of OS versions, DB engines, vendors, etc. Saile

One option

[djweis]

My employer is using the ars Digita software. We haven't had any problems with it, and if you plan on doing more than just ecommerce, it's got a huge amount of functionality, all integrated together. You can save a ton of time.

Try Contacting a Few Companies

[scotpurl]

I recently noticed that http://www.insight.com started using PHP, which seemed downright forward-thinking for a vendor to do.

Most folks like to blab about what they've done, so try emailing a few folks, or see what old classmates or fishing buddies the VP's in your company know elsewhere. That'll open doors quickly.

The only thing of real substance I can offer is obvious, but I'll offer it anyway (just to get the ball rolling). Try to modularize everything, and make it generic. Perl, mod-Perl, and PHP all work on NT or *nix, which helps divorce you from a single-vendor situation. The servers can then have any OS or any web server. You could even do a mix in a server farm.

You'll likely need to do something tricky for to bypass some performance bottleneck, but try to keep those tricks simple, with the intent of eventually finding some non-tricky method of getting around that bottleneck.

Re:Large scale systems

[PhiRatE]

My apologies, our current system has scaled up to a million hits/day, not a month :)

Many thanks

[Fross]

...for all your suggestions. got a few ideas to chase up now. :)

fross

Large scale systems

[PhiRatE]

Yep, been there, done that. Our particular solution revolved around a cluster of linux servers fronted by a linux/iptables layer-4 router and various custom stuff to perform rapid failover. The cluster nodes themselves used Apache with PHP4, with a database backend and rsync to distribute content to the cluster nodes from a distribution box.

In order to reduce the work, you could use a cisco localdirector or equivalent layer-4 switch instead of the iptables stuff, the rest should basically fit together almost out of the box.

This solution scales very well and handles millions of hits/month. Recommended additions are squid running as an accelerator, since it can absorb a significant number of hits that apache would otherwise have to deal with. However the addition of the Tux in-kernel web stuff on the cluster nodes would probably be an even more preferable solution.

Also recommend dual-cpu machines, they do really well with apache.