Balancing Performance and Convention 171
markmcb writes "My development team was recently brainstorming over finding a practical solution to the problem that's haunted anyone who's ever used a framework: convention vs. customization. We specifically use Rails, and like most frameworks, it's great for 95% of our situations, but it's creating big bottlenecks for the other 5%. Our biggest worry isn't necessarily that we don't know how to customize, but rather that we won't have the resources to maintain customized code going forward; it's quite simple to update Rails as it matures versus the alternative. What have your experiences been with this problem? Have you found any best practices to avoid digging custom holes you can't climb out of?"
Build vs. Buy (Score:5, Interesting)
I agree with the above poster. (Score:5, Interesting)
Without knowing much more detail about your problem, it is difficult to give advice. Having created large, complex websites, and maintained them for long periods of time, we have yet to run into intractible problems as a result of the language or framework themselves. The fact that there are many large, complex Ruby on Rails websites being created and maintained by Fortune 500 companies also contradicts the idea that Ruby and Rails have serious, show-stopping shortcomings.
I strongly suspect that part of the problem is that you do not know your platform (in particular, Ruby) as well as you perhaps could. Ruby is very extensible via Gems that are freely available or that you can write yourself, or even just code modules. Rails is similarly, wonderfully extensible via plugins. Plugins are also freely available for just about every situation you can imagine, or again you can write your own.
Ruby is one of the most flexible languages in existence. Rails is hardly perfect, but any shortcomings are easy to code around by adding your own extensions. But expecting any framework to be complete for your purposes, before you begin, is pretty much wishful thinking.
Once again, I do not know enough about your situation. But if you expect any framework to be "plug and play" for very complex tasks, like an erector set, you will be disappointed. There is no substitute for knowledge of your platform, and programming ability.
Write Tests, Open Source, & use Rack (Score:2, Interesting)
[test!]
A good test suite == "best practice to avoid digging custom holes you can't climb out of"
I work at a Rails shop too and, when I/we need to do something highly custom, we create it as a gem (or a Rails plugin) and post it somewhere incase someone else finds it useful. None of the plugins/gems I've released have required any maintenance to speak of, unless I've wanted to add additional features.
Be sure to write tests for your customizations (gem/plugin)! This will make it really easy to discover if your plugin no longer works for the next version of Rails/ActiveRecord/whatever it is you're extending.
[open source!]
If your changes might help other developers (they're not very, very specific to your product), open source them as a gem and let people know how to use it.
Not only can others benefit from your changes, but they can commit back too! Put the gem up on github[1], as it's the current de facto standard home for such things.
[rack it up!]
If you really need crazy performance out of Rails, look into using Rack[2]. Rails 2.3 (currently Rails Edge, will be released this month) *finally* uses Rack. Something like Rails Metal[3] makes it easy to return directly from Rack, letting you *highly* optimize certain requests. This is like rewriting some of your Ruby as C extensions to speed it up - Rack is really easy to use.
Good luck!
[1]: http://github.com/ [github.com]
[2]: http://rack.rubyforge.org/ [rubyforge.org]
[3]: http://weblog.rubyonrails.org/2008/12/17/introducing-rails-metal [rubyonrails.org]
Contribute (Score:3, Interesting)
``Our biggest worry isn't necessarily that we don't know how to customize, but rather that we won't have the resources to maintain customized code going forward; it's quite simple to update Rails as it matures versus the alternative.''
As Alan Kay said, the best way to predict the future is to invent it. If you are worried about your useful code being broken by a future version of Rails, contribute it to Rails so that that future version will include it. Or, if you can't get it into the framework, make it available as a separate plugin.
Assuming that it's ok to share the code and that the code is useful outside your project, this will allow open source to work for you and ease your maintenance burden. If it's not ok to share the code or it isn't useful outside your project, then it's part of the project and the project will have to carry the cost.
Re:Switch to django and python for starters (Score:3, Interesting)
The best option is to not use one single framework for all situations unless it's appropriate. Too many developers put blinders on, picking one framework and using it for every project.
My company has a flexible custom framework that we use for the majority of projects. But before we start on any project we discuss the options to see if another framework or open source app would be a better fit. Along with picking the best language for a task, the same should be applied to frameworks.
No need to use Rails or any other particular framework if it doesn't fit the project.
JEE tech is too complex and fragile (Score:4, Interesting)
We are well into a project that has JBoss SEAM as its basis, but required significant mods to give us multi-database capability and a FLEX front end. So instead of a community maintained opinionated meta-framework, we now have our own super complex, fragile framework cluster(****).
It is also effectively a dense, opaque yarnball. The stack traces of exceptions are so long, due to the interceptor architecture, that the Java stack trace displaying algorithm gives up and prints ...
So my advice would be, at the risk of trolling, but I intend it as a serious debate position, don't start with something as excessively complex as JEE technology.
Because JEE technology is barely manageable, maybe, if you stick EXACTLY to the most popular opinionated meta-framework for it, but if you deviate off the straight and narrow path, you are on your own in a dense jungle with a dull machete.
Good frameworks provide flexibility (Score:2, Interesting)
The true differences between frameworks are inevitably revealed as soon as you try to build a product of significant complexity. A truly flexible framework makes it easy to override, overload, or otherwise replace functionality WITHOUT having to modify the framework itself. This has long been my issue with a number of web frameworks implemented in a variety of frameworks. There are many frameworks that will make a fairly simple form based web application come together very quickly. If you've got 5-10 forms, a simple data model, and low enough load to run on a single host, there are lots of frameworks that will let you churn out an app in the blink of an eye. But as soon as you have a sophisticated and complex data model, loads that require scaling not just the web layer, but also the persistence layer, across multiple hosts and even multiple data centers, and you have to support a variety of views (html, web services, json for ajax, iphone, pdf, etc), a lot of frameworks fall completely flat and are unable to help and, in many cases, become an active obstruction.
It has been a number of years since I seriously investigate RoR for a web project because the vast majority of my work involves very high performance and high volume sites and when I looked into RoR, it offered no easy mechanism to work cooperatively in a cluster with any more sophistication than sticky load balancing and a single shared db. I'm sure it is much improved since then, but every time I take a quick look-see, it still seems to be very lacking - not just in scalability, but in extensibility in general, and I just can't tolerate that.
There aren't many frameworks that I've encountered that offer the kind of flexibility to really allow you to take our app any direction you want to go. The one that does requires a fair bit more work up front, but boy does it pay off as soon as you have to get sophisticated. Of course, it is native to the language everyone round here loves to hate, Java, but I highly recommend you look into Spring and SpringMVC. Plug in any persistence mechanism you like, including a mixed persistence layer. Plug in any view technology you like. Use AOP to handle cross cutting concerns. You have the flexibility to replace any built-in component with an implementation of your own, so you are never required to modify the source code of the framework itself.
At every turn, SPring has provided the flexibility I've need to build very sophisticated applications for a number of years now. It will take quite a bit to move me to another platform for anything but the simplest of apps. I recommend you check it out. The fact that it runs in a language that is 25x faster than Ruby doesn't hurt any, either. And the ecosystem of useful code is larger for Java than for any other language I'm familiar with. JEE is needlessly complex, but you can do just about everything JEE does via Spring with a tiny fraction of the complexity, and for the very few things spring won't do, Spring will happily expose JEE directly to you. Oh, and I gather it runs in .NET, too, though I have no idea if it is quite as functional, since it is developed from a java-centric paradigm.
Re:Switch to django and python for starters (Score:4, Interesting)
Rails has a lot of problems mixed in with a lot of greatness. For a great many projects, its greatness overcomes its probems.
Its biggest problem, IMHO, is that it does not grow well with the needs of a developer over time. Some things:
* Doesn't scale well
* Not easy to do anything outside the Rails world
I had a conversation with Ezra back when he was starting Merb, and I remember the large resistance to the idea that there were problems with Rails at all. As Merb got released and then popular, it was clear that there are holes in Rails (yes I know they are merging).
At about the same time, I started my own framework (the Caffeine framework now the Go framework) which has not been made open yet (and is not complete yet), but there are several issues that it addresses which could help (a) improve Rails or (b) create other frameworks that can fill holes. I'll mention some of the ideas below.
Note that the original goal was to use Rails for our "very cool application" (tm). Unfortunately we have some pretty lofty goals and Rails turned out to be the wrong tool. I am the CEO of a successful Internet company CityMax (about 30 employees) so I do have experience with the problems that happen after launch and success. They are not the same problems as during the startup phase.
Here are some things that an ideal framework has that I believe are missing or lacking in Rails. Again, I do want to stress that Rails has greatness and that I really respect its creator. That said, I think frameworks can grow to another level.
# Built in scalability and clustering. Should require zero changes (except some configuration) to go from single database to a cluster of databases.
# Reusable code. If you build a feature (like a message board) for one application, that same code should be reusable in another app with zero code changes. It should also be reusable multiple times in the same application. For example, a blog app might use the message board in each blog but there might also be a support message board for the whole system.
# Built in scalable file system. The file should be storable in local, Amazon S3, MogileFS, network share or whatever. Directories can be mapped to any file store (e.g. temp directories might be local, permanent storage might be S3 and archives might be a network share). Should be easy to remap directories and migrate from one data store to the other.
# No magic. Hate it. Makes it hard to figure out and debug the code.
# Custom field types for database, particularly the ability to insert files and images into a database record. The actual files and images are stored in the scalable file system and automatically managed (basically automatic garbage collection for files).
# An image field type where you can change the image sizes (e.g. thumbnail, preview, full size) at will and images are automatically regenerated in those sizes the first time they are needed. This is one of those problems that comes out as you get bigger. Say your shopping cart app stores images in 3 sizes. Now you have 10 million products and you want to add another size. Without this, you are basically stuck with the decision you made at the start.
# Make easy things easy and make hard things easy. In Rails, the solution is usually to memorize the option that allows you to do what you want. With the right design, the problems should be solvable using just Ruby and the framework. This helps keep the framework light and easy to learn over time.
# Discoverable. The framework needs to be easy to figure out on ones own. Merb was more like this. Rails was less.
# Multi-threading. This was one of the primary reasons I didn't want to use Rails originally. I understand that threading is in Rails 2 now but I'm not sure how good it is.
# CouchDB. Okay, this feature is a little cutting edge but the biggest issue after launch is migrating the database. Add a new feature that requires a new column on a table with a bajillion records and you're doing many long midnight runs
Re:Hard to say, but... (Score:2, Interesting)
Re:Switch to django and python for starters (Score:3, Interesting)
Re:That's rich! (Score:3, Interesting)
As I stated before, it may be the fastest in its class, but it still ain't fast.
What do you use when you need something better performing than Ruby? You aren't going to say "hand-tuned assembly language", are you?
I've been using Java for a long time (this may be obvious from my userid & screen name), though it's not the only language I use nowadays.
Startup is slow (though finally they have addressed this for client apps in a recent release, and it's irrelevant for stable server apps); memory consumption is still not optimal (this hasn't seen massive improvements), but performance is excellent nowadays, assuming your code isn't crap (or using some library that's doing a billion extra things at each call).
The stats that show up in various performance shootouts, which place it just a tad behind straight C/C++ and miles ahead of languages like PHP, Perl, Ruby, Python, etc., match up with my experiences pretty well ...though of course actual webapp hotspots are usually in data access or remote communication -- and have nothing to do with the development language.
There are also now more situations where the hotspot compiler can result in *faster* execution than C/C++ because it operates using live runtime performance data, which is obviously unavailable to a static compiler.
Do you have any specifics to point to, or are you just talking?